When RAG and vector databases are the wrong tool

Common error in app-layer LLM products is that people use RAG and Vector DBs incorrectly. To process a single document or small amount of input information, people first chunk it, throw it in a vectorDB then retrieve it and put in the context. That makes no sense if the original document is small and could literally be fully packaged in the context window.

Remember - you need RAG if during inference, you need to select a subset of information from thousands of documents (100s of millions or billions of tokens), which of course is impossible to include in the context window of a single inference.