Vector Memory Hack

**Quick judgement**: Solid, pragmatic skill for fast, lightweight "semantic-ish" search over MEMORY.md and similar docs. Excellent fit for constrained environments and agent workflows that need quick context lookup without heavy vector infra. Not a replacement for modern embedding-based semantic search when accuracy is critical. **What it does well** - Uses TF-IDF + cosine similarity with SQLite to provide very fast (<10ms) retrieval of the most relevant sections from large markdown files. - Entirely standard library + SQLite: easy to install, good for locked-down servers, edge devices, or minimal containers. - Supports rebuild, incremental update, search, and stats via a simple CLI (`vsearch` / `vector_search.py`). - Designed specifically for agent workflows (e.g., "always run vsearch before a task"), which makes it easy to integrate into OpenClaw agent chains. **Key limitations / risks** - **Not true semantic embeddings**: TF-IDF works well for keyword and near-keyword similarity, but may miss deeper paraphrases or conceptual matches; results will be weaker than modern embedding models on complex queries. - **Markdown structure dependency**: Assumes MEMORY.md and similar files are reasonably structured with `##`/`###` headers; poorly structured docs may yield odd sections and weaker retrieval. - **SQLite locking & concurrency**: Under heavy concurrent access or frequent reindexing, SQLite database locks could occur; the skill mentions this but doesn’t provide robust concurrency handling. - **Language coverage is basic**: The tokenizer + stopwords are customizable but not as robust as dedicated multilingual NLP pipelines; non‑English or mixed-language content may see reduced quality without tuning. **Recommended scenarios** - You want **fast, cheap, and simple** context retrieval for an OpenClaw agent using a MEMORY.md or similar markdown doc. - You’re deploying to a **resource-constrained VPS / edge device** or a **locked-down environment** where installing heavy ML or vector DB stacks is painful or impossible. - You’re building a **prototype or internal tooling** where approximate semantic search is acceptable and the priority is low overhead and speed. **When to consider alternatives** - You need **high recall and nuanced semantic understanding** across large document sets, especially with complex paraphrasing or cross-language queries → use embeddings + a vector DB or an embedding API. - You operate at **larger scales (10k+ docs)**, need **concurrent indexing and querying**, or require **advanced features** like metadata filters and hybrid search → use a dedicated vector database or retrieval system.