BUILDING A PRODUCTION-READY GITHUB RAG SYSTEM
Build Chapter 1: What Are We Building, and Why GitHub RAG?
Part 1: Foundation and Repository Understanding | For engineers learning RAG | About 8 minutes to read
Software repositories hold far more than source code. They contain documentation, SQL, configuration, deployment steps, business rules, tests, and years of design decisions. The problem is that this knowledge is hard to find, especially in a project you did not write.
Over this series we will build a Retrieval-Augmented Generation (RAG) system that answers natural-language questions about a GitHub repository. Every answer will be grounded in the repository's actual content and traceable to file paths and line numbers. This first chapter is the map: what we are building, why the simpler alternatives fall short, what it costs, and how the pieces fit together. There is no code to run yet.
What you will take away
• The problem a repository RAG system solves, and where keyword search and plain LLM prompting fall short.
• A worked cost comparison between sending the whole repository to an LLM and retrieving only what matters.
• The two-pipeline architecture and the ten-phase build plan we follow for the rest of the series.
Prerequisites: comfortable reading Python and SQL. No RAG experience needed. From Chapter 2 you will need Python 3.11 and uv. Series code: [to be shared shortly]
For technical leaders: the 60-second version • The decision. Once a repository is too big to paste into a prompt, or many people will query it, build retrieval over an index with citations instead of sending the whole repository to an LLM every time. • The cost. In the worked example in section 4, 1,000 questions cost about $16 in input tokens with retrieval, against roughly $100 to $1,000 when the whole repository is sent each time, depending on prompt caching. Answer generation, not embedding or storage, is the main ongoing cost. • The risk. The design answers the questions a security review will ask. Answers are verifiable because they carry citations. The system says so when evidence is missing instead of guessing. Authorization happens before retrieval, and repository content is treated as untrusted input. • When not to use it. A small repository queried a handful of times does not need RAG. Paste it into the prompt. • Questions to ask your team. How do we measure answer quality? Does retrieval respect repository permissions? What happens when the evidence is missing? How is the index kept fresh? What does one question cost at our expected volume? |
1. The objective
We want users to ask questions like these and get a useful answer:
• What does this repository do?
• Where is a particular function implemented?
• Which configuration controls this feature?
• How does data move through this project?
• Which SQL query produces this metric?
• What depends on this class, and where is this table referenced?
• How do I run or deploy the application?
A good answer is not just plausible text. The system must:
1. Search only the repository the user asked about.
2. Find the most relevant files and code sections.
3. Generate the answer from that retrieved content.
4. Cite file paths and line ranges, and link back to the source on GitHub.
5. Refuse to guess when the repository does not contain enough evidence.
Here is the shape of answer we are aiming for. It is an illustration of the target format, not output from the finished system:
Q: How is customer churn calculated, and where is it implemented?
A: Churn is the share of customers who were active in the previous
period and are no longer active in the current one. The calculation
is in the churn_rate model, which reads from customer_activity.
Sources:
[1] models/churn_rate.sql, lines 12-48
[2] docs/metrics.md, lines 30-41
If the repository does not contain the answer:
A: I could not find where this is calculated in the indexed files.
The goal is not a chatbot. It is a searchable, trustworthy knowledge layer over GitHub repositories.
2. The problem we are solving
A repository can hold thousands of files across many directories, and the knowledge you need is usually spread over several of them: Python modules, SQL, READMEs, YAML and JSON configuration, infrastructure definitions, shell scripts, tests, and docstrings.
Keyword search works when you already know the filename, function, or term. It breaks down on conceptual questions such as "How is customer churn calculated in this project?" You may not know which file holds the logic, whether it is written in Python or SQL, which upstream tables are involved, or whether a README explains it. A keyword search finds every occurrence of the word churn but does not assemble the explanation.
A RAG system retrieves related content from several files and builds one coherent answer from it. In practice that gives you four things:
• Faster onboarding. New team members can ask for the entry point, the data flow, and how to run the project locally.
• Preserved knowledge. Reasoning that lives in a few people's heads becomes easier to recover when they move on.
• Higher productivity. Start from a focused answer, then open the cited files only when you need depth.
• Wider access. Analysts, product owners, and support teams can ask specific questions, and citations let them verify the result [5].
This does not replace good documentation. It makes what you already have easier to reach.
3. Why not ask an LLM directly, or send it the whole repository?
A general LLM has never seen your repository
A general-purpose model has no access to a private or recently changed repository. Without the code it may give a generic explanation, assume a common pattern, rely on outdated information, mix up similarly named components, or produce an answer that sounds right and is unsupported. RAG fixes this by retrieving from an external knowledge source before generating, so the model can reason over repository-specific content without retraining [1].
The repository stays the source of truth. The LLM interprets and explains retrieved evidence. It does not invent the implementation.
Pasting the whole repository is the obvious next idea
Read every file, join them into one prompt, and ask the question. Modern context windows are large enough that this can technically work for mid-sized repositories. It still has real costs:
Concern | Whole repository in the prompt | RAG |
Context limit | Large repositories do not fit. Even when they do, less room is left for instructions, history, and the answer. | Only the relevant chunks are sent. |
Cost | You pay for the full repository on every question. | You pay once to index, then send a small context per question. |
Latency | A large prompt on every request. | A small prompt on every request. |
Answer quality | Unrelated files add noise: duplicates, stale code, generated files, other environments. | Retrieval focuses the model on relevant evidence. |
Access control | Hard to keep restricted files out of the prompt. | Filters and authorization apply before retrieval. |
Freshness | Any change to the repository can invalidate cached prompt content. | Only changed files are re-parsed and re-embedded. |
RAG changes the pattern from "process everything for every question" to "index once, retrieve the relevant parts, query many times."
4. What it costs: a worked comparison
Exact savings depend on repository size, chunking, question volume, embedding and LLM prices, retrieved context size, and any vector database or reranking costs. A simple example shows the shape of the difference.
Assumptions
• A repository of 500,000 tokens. A typical question needs 8 chunks, 8,000 tokens in total. There are 1,000 questions.
• Input price of $2.00 per million tokens, and $0.20 per million for cached input. These are the published rates for OpenAI's GPT-6 Sol in October 2026 [3]. Check current rates [2] and substitute your own model's prices.
• Output cost is similar in both approaches, so we compare input only.
Approach | Input tokens billed | Input cost |
Whole repository, no caching | 500,000,000 | $1,000 |
Whole repository, prompt caching (best case) | 500,000,000 at the cached rate | $100 |
RAG | 8,000,000 | $16 |
With retrieval, the system sends 1.6% of the repository input volume, which is 98.4% fewer repository tokens.
What this example does not show
• Caching narrows the gap but does not close it. The best case assumes every request hits a warm cache, and it does nothing for noise, access control, or the cost of the first load. This model also charges more for prompts above 272,000 tokens, which would raise both whole-repository figures [3].
• RAG has its own costs. You pay once to embed the repository, plus vector storage, retrieval, optional reranking, and the prompt around the retrieved chunks. Embedding is usually a small one-time cost, while answer generation dominates ongoing spend [4]. That is why chunk quality, retrieval quality, and context size matter for cost.
• Small repositories are different. For a very small repository queried once or twice, sending the content directly is simpler. RAG pays off when the repository is large, changes over time, serves many users, needs citations or access control, or must give verifiable answers.
The two cost models, side by side:
Direct cost = questions x repository tokens x input rate + output cost
RAG cost = one-time embedding + vector storage
+ retrieval and optional reranking
+ questions x retrieved tokens x input rate
+ output cost
5. Design principles we will follow
These ten principles shape the design decisions in the rest of the series.
1. GitHub remains the source of truth.
2. Answers must be grounded in retrieved repository content.
3. Citations are preserved from ingestion to response.
4. Repository content is treated as untrusted input.
5. Authorization happens before retrieval.
6. Chunking preserves code structure.
7. Configuration stays external instead of hardcoded.
8. Unchanged content is not reprocessed.
9. Retrieval is validated before agent behavior is added.
10. Every component is independently testable and replaceable.
6. High-level architecture
The system has two pipelines that meet at the index. The indexing pipeline turns repository files into searchable records. The retrieval pipeline starts when a user asks a question: it checks what the user may see, retrieves and reranks evidence, and generates a cited answer.
Figure 1. Indexing runs once and again whenever files change. The retrieval pipeline runs for every question.
The agent and cloud deployment come last on purpose. Retrieval must work reliably before an agent is introduced.
7. How we will build it, step by step
We build incrementally so that every component can be understood and tested on its own.
Phase | What we build | Why it matters |
1. Project foundations | Python project structure, dependencies, configuration, logging, domain models, custom exceptions. | A maintainable base before any feature. |
2. Discovery and file policies | Local repository connector, safe traversal, file classification, YAML-driven include and exclude rules, content hashes, metadata inventory. | Know what is in the repository and keep unsafe or useless files out. |
3. Extraction and parsing | Read eligible files, normalize documents, parsers for Markdown, configuration, Python, and SQL. | Preserve structure and source metadata. |
4. Code-aware chunking | Split on logical units (Markdown sections, Python functions and classes, SQL statements and CTEs, configuration sections), not arbitrary character counts. | Chunks that make sense on their own retrieve better. |
5. Embeddings and vector storage | Embed chunks, store vectors with metadata, support insert, update, and delete, skip unchanged content. | Index once, update cheaply. |
6. Hybrid retrieval | Combine semantic search, keyword search, symbol search, and metadata filters. | Semantic search understands concepts but can miss exact identifiers. Keyword search finds identifiers but misses concepts. |
7. Reranking and grounded generation | Rerank candidates, build a concise context, generate cited answers, say so when evidence is insufficient. | Answers come from evidence only. |
8. Security and evaluation | Authorization before retrieval, path-level controls, prompt-injection protection, sensitive-file policies, retrieval evaluation, groundedness and citation checks, regression tests. | Trust and safety are measured, not assumed. |
9. GitHub synchronization | Repository and branch sync, commit SHA tracking, detection of added, changed, and deleted files, incremental reprocessing. | The index stays current. |
10. Agent and deployment | Service layer, Google ADK LlmAgent, deployment on Vertex AI, monitoring, scaling. | Added only after retrieval is stable. |
Managed services such as Vertex AI RAG Engine also cover ingestion, chunking, embedding, indexing, and retrieval [6]. This is a learning project, so we first build and understand each component ourselves.
Conclusion and next chapter
GitHub repositories hold valuable organizational knowledge that is hard to navigate. Sending the whole repository to an LLM for every question is costly, repetitive, noisy, and hard to scale or secure. A well-designed GitHub RAG system indexes the repository once, retrieves only the relevant content, and has an LLM explain it with citations.
A successful system is not the one that writes the longest answer. It is the one that returns the most useful, traceable, secure, and repository-grounded answer.
In Chapter 2 we set up a maintainable Python 3.11 project with uv, so everything that follows has a clean foundation instead of growing into one large, tightly coupled script.
About SNPaul Infotech We help engineering and data teams design, evaluate, and secure retrieval and GenAI systems like this one. If you are weighing RAG for your own repositories or documents, get in touch: connect@snpaulinfotech.com |
References
[2] OpenAI. API pricing
[4] AICostCheck. RAG cost guide 2026
[5] Databricks. Retrieval-augmented generation
[6] Google Cloud. RAG Engine overview
Comments