This project has been created as part of the 42 curriculum by qcorsius

Description

RAG Against The Machine is a project made to understand Retrieval-Augmented Generation processes. Our objective here is to be able to slice multiple files of a code base in order to get an LLM answer our questions about how to use the tool given to it.
Notions learnt during this project were:

• Chunking
• Indexing
• Retrieving
• Embedding
• Caching
• HTTP local API

Instructions

To install the project, you must run

make install

or

uv sync

To launch the HTTP API, run

make run

or

uv run fastapi run

To communicate with the API, run

uv run rag-cli <command> [arguments]

System Architecture

HTTP API

I built an HTTP API that runs all the processes, and put in cache long loading components, such as the retriever, the LLM model used to generate answers, or the previous search results. Then each HTTP route executes a step on signal:

• /index → Slices in chunks the codebase build the RAG retriever around them
• /search → Finds in the retriever the k most relevant sources (chunks) to answer a question
• /answer → Uses the k most relevant sources (chunks) to generate an answer for a question
• /search/dataset -> Does the same thing as /search, but for a list of questions, then writes a file containing the search results
• /answer/dataset -> Does the same thing as /answer, but for a list of questions, then writes a file containing the answers
• /evaluate -> Compares the sources found by the /search/dataset route and compares it with a gold dataset containing the awaited results

RAG CLI

To communicate easily with the API, I created a CLI that send queries to the API and awaits a response. Each response is formatted to be human-readable.
Here is the list of available commands:

• rag-cli index [--max_chunk_size]
• rag-cli search --query --k
• rag-cli answer --query --k
• rag-cli search_dataset --dataset_path --k --save_directory
• rag-cli answer_dataset --student_search_results_path --save_directory
• rag-cli evaluate --student_search_results_path --dataset_path

Environment variables

• CORPUS_DIR: Corpus location (default data/raw)
• RETRIEVER_SAVE_DIR: Retriever location (default data/processed)
• ENDPOINT: HTTP API's endpoint for communication (default http://localhost:8000)
• ANSWERING_MODEL: Model to use for answer generation (default Qwen/Qwen3-0.6B)
• EMBEDDING_MODEL: Model used for embedding generation, resolved via chonkie's embeddings registry (default minishlab/potion-retrieval-32M, a Model2Vec static-embedding model that runs fast on CPU with no GPU needed; a sentence-transformers/... name can be used instead for higher-quality but slower embeddings)

Chunking strategy

I did some research about chunking methods, and each language or file format should be sliced differently to be optimal. For example python files must be separated on function/class definitions to keep as much data as possible, Markdown files must be separated on titles , etc... Instead of reimplementing everything, I searched for a Python module that handles chunking, and I found the chonkie lib. I use a chonkie implementation called chonkie[code] that is able to select the right chunking method for the file type/programming language detected. It supports multiple languages such as Python, JSON, YML, C, etc...
Any unsupported language is sliced as a plain text file.

Retrieval method

BM25 algorithm

BM25 mesures term frequency and document relevance accurately. It also accounts for document length normalization, giving fair weight to all documents. It is robust and efficient, but alone it is only lexical.That's why I use embeddings in this project to tune the results towards word meanings instead of just searching blindly for the term.

Ranking mechanisms

My ranking works by combining the BM25 algorithm and semantic embeddings. It ranks files based on content relevance if terms or same meaning words are found in the file. The embedding model (minishlab/potion-retrieval-32M by default) creates semantic embeddings that defines similarities even if the term is not present in the file

Performance analysis

The answers are more precise the more you give sources. You can see it in the rag-cli evaluate command: for 1 source (Recall@1) it gives less than 30% accuracy, 3 sources is better with more than 40% accuracy, but 5 sources goes over 55% accuracy. Those data were calculated based on code question, for docs, accuracy is way more precise: 80% accuracy for 5 given sources and almost 90% for 10 sources.

Design Decision

Crucial data such as corpus path and http endpoint are possible to change via environment variables, so the project can be moved.
I chose BM25 over TF-IDF because I was told that it was more performant. I also chose to implement windowed encoding to increase embedding accuracy.

Challenges faced

I had to learn how works fastapi and to configure proper HTTP routes and responses with corresponding data. Understanding of the embedding concept was also a bit tough. Another point is the overall understanding of the BM25 algorithm, not just using the lib without knowing what it does.

Example Usage

First run the API:

(rag-against-the-machine) ╭─qcorsius@z1r11p1 ~/Documents/Projects/RAG-against-the-machine ‹master●› 
╰─$ uv run fastapi run                                                                                                                                                                                                                                                                                          2 ↵

 ⚡️ Starting FastAPI in production mode
 
 🐍 Using import string: api:main
 
 🌐 Server started at http://0.0.0.0:8000
    Documentation at http://0.0.0.0:8000/docs
 
  Logs:
 
[trimmed: HF logs]

Successfully reloaded cached retriever!
Successfully reloaded cached vector index!
 ▕  ASGI app factory detected. Using it, but please consider setting the --factory flag explicitly.
 ▕  Started server process [66370]
 ▕  Waiting for application startup.
 ▕  Application startup complete.
 ▕  Uvicorn running on http://0.0.0.0:8000 (Press CTRL+C to quit)

Then just run a cli command and see the result:

(rag-against-the-machine) ╭─qcorsius@z1r11p1 ~/Documents/Projects/RAG-against-the-machine ‹master●› 
╰─$ uv run rag-cli search --query "How to setup an OpenAI compliant server?" --k 5
data/raw/vllm-0.10.1/docs/deployment/frameworks/bentoml.md [0:445]
data/raw/vllm-0.10.1/docs/deployment/frameworks/chatbox.md [0:896]
data/raw/vllm-0.10.1/examples/online_serving/openai_chat_completion_client_with_tools_required.py [0:1993]
data/raw/vllm-0.10.1/docs/deployment/frameworks/litellm.md [0:1996]
data/raw/vllm-0.10.1/examples/online_serving/openai_chat_completion_with_reasoning.py [0:1926]