When you development environments (IDEs) paired with coding agents, you’ve likely seen code suggestions and edits which are surprisingly accurate and relevant.
This level of quality and precision comes from the agents being grounded in a deep understanding of your codebase.
Take Cursor for example. Within the Index & Docs tab, you possibly can see a piece showing that Cursor has already “ingested” and indexed your project’s codebase:
So how can we construct a comprehensive understanding of a codebase in the primary place?
At its core, the reply is retrieval-augmented generation (RAG), an idea many readers may already be conversant in. Like most RAG-based systems, these tools depend on semantic search as a key capability.
Slightly than organizing knowledge purely by raw text, the codebase is indexed and retrieved based on meaning.
This enables natural-language queries to fetch essentially the most relevant codes, which coding agents can then use to reason, modify, and generate responses more effectively.
In this text, we explore the RAG pipeline in Cursor that allows coding agents to do its work using contextual awareness of the codebase.
Contents
(1) Exploring the Codebase RAG Pipeline
Let’s explore the steps in Cursor’s RAG pipeline for indexing and contextualizing codebases:
Step 1 — Chunking
In most RAG pipelines, we first need to manage data loading, text preprocessing, and document parsing from multiple sources.
Nevertheless, when working with a codebase, much of this effort could be avoided. Source code is already well structured and cleanly organized inside a project repo, allowing us to skip the customary document parsing and move straight into chunking.
On this context, the goal of chunking is to interrupt code into meaningful, semantically coherent units (e.g., functions, classes, and logical code blocks) slightly than splitting code text arbitrarily.
Semantic code chunking ensures that every chunk captures the essence of a specific code section, resulting in more accurate retrieval and useful generation downstream.
To make this more concrete, let’s have a look at how code chunking works. Consider the next example Python script (don’t worry about what the code does; the main focus here is on its structure):
After applying code chunking, the script is cleanly divided into 4 structurally meaningful and coherent chunks:
As you possibly can see, the chunks are meaningful and contextually relevant because they respect code semantics. In other words, chunking avoids splitting code in the course of a logical block unless required by size constraints.
In practice, it means chunk splits are likely to be created between functions slightly than inside them, and between statements slightly than mid-line.
For the instance above, I used Chonkie, a light-weight open-source framework designed specifically for code chunking. It provides an easy and practical strategy to implement code chunking, amongst many other chunking techniques available.
[Optional Reading] Under the Hood of Code Chunking
The code chunking above just isn’t accidental, neither is it achieved by naively splitting code using character counts or regular expressions.
It begins with an understanding of the code’s syntax. The method typically starts through the use of a source code parser (comparable to tree-sitter) to convert the raw code into an abstract syntax tree (AST).
An abstract syntax tree is actually a tree-shaped representation of code that captures its structure, and never the actual text. As an alternative of seeing code as a string, the system now sees it as logical units of code like functions, classes, methods, and blocks.
Consider the next line of Python code:
x = a + bSlightly than being treated as plain text, the code is converted right into a conceptual structure like this:
Task
├── Variable(x)
└── BinaryExpression(+)
├── Variable(a)
└── Variable(b)This structural understanding is what enables effective code chunking.
Each meaningful code construct, comparable to a function, block, or statement, is represented as a node within the syntax tree.
As an alternative of operating on raw text, the chunking works directly on the syntax tree.
The chunker will traverse these nodes and groups adjoining ones together until a token limit is reached, producing chunks which are semantically coherent and size-bounded.
Here is an example of a rather more complicated code and the corresponding abstract syntax tree:
while b != 0:
if a > b:
a := a - b
else:
b := b - a
return 
Step 2 — Generating Embeddings and Metadata
Once the chunks are prepared, an embedding model is applied to generate a vector representation (aka embeddings) for every code chunk.
These embeddings capture the semantic meaning of the code, enabling retrieval for user queries and generation prompts to be matched with semantically related code, even when exact keywords don’t overlap.
This significantly improves retrieval quality for tasks comparable to code understanding, refactoring, and debugging.
Beyond generating embeddings, one other critical step is enriching each chunk with relevant metadata.
For instance, metadata comparable to the file path and the corresponding code line range for every chunk is stored alongside its embedding vector.
This metadata not only provides vital context about where a bit comes from, but in addition enables metadata-based keyword filtering during retrieval.
Step 3 — Enhancing Data Privacy
As with every RAG-based system, data privacy is a primary concern. This naturally raises the query of whether file paths themselves may contain sensitive information.
In practice, file and directory names often reveal greater than expected, comparable to internal project structures, product codenames, client identifiers, or ownership boundaries inside a codebase.
Consequently, file paths are treated as sensitive metadata and require careful handling.
To handle this, Cursor applies file path obfuscation (aka path masking) on the client side before any data is transmitted. Each component of the trail, split by / and ., is masked using a secret key and a small fixed nonce.
This approach hides the actual file and folder names while preserving enough directory structure to support effective retrieval and filtering.
For instance, src/payments/invoice_processor.py could also be transformed into a9f3/x72k/qp1m8d.f4.
Note: Users can control which parts of their codebase are shared with Cursor by utilizing a
.cursorignorefile. Cursor makes a best effort to forestall the listed content from being transmitted or referenced in LLM requests.
Step 4— Storing Embeddings
Once generated, the chunk embeddings (with the corresponding metadata) are stored in a vector database using Turbopuffer, which is optimized for fast semantic search across hundreds of thousands of code chunks.
Turbopuffer is a serverless, high-performance search engine that mixes vector and full-text search and is backed by low-cost object storage.
To hurry up re-indexing, embeddings are also cached in AWS and keyed by the hash of every chunk, allowing unchanged code to be reused across subsequent indexing execution.
From an information privacy perspective, it will be significant to notice that only embeddings and metadata are stored within the cloud. It implies that our original source code stays on our local machine and is never stored on Cursor servers or in Turbopuffer.
Step 5 — Running Semantic Search
After we submit a question in Cursor, it’s first converted right into a vector using the identical embedding model for the chunk embeddings generation. It ensures that each queries and code chunks live in the identical semantic space.
From the angle of semantic search, the method unfolds as follows:
- Cursor compares the query embedding against code embeddings within the vector database to discover essentially the most semantically similar code chunks.
- These candidate chunks are returned by Turbopuffer in ranked order based on their similarity scores.
- Since raw source code is rarely stored within the cloud or the vector database, the search results consist only of metadata, specifically the masked file paths and corresponding code line ranges.
- By resolving the metadata of decrypted file paths and line ranges, the local client is then capable of retrieve the actual code chunks from the local codebase.
- The retrieved code chunks, in its original text form, are then provided as context alongside the query to the LLM to generate a context-aware response.
As a part of a hybrid search (semantic + keyword) strategy, the coding agent may also use tools comparable to grep and ripgrep to locate code snippets based on exact string matches.
OpenCode is a well-liked open-source coding agent framework available within the terminal, IDEs, and desktop environments.
Unlike Cursor, it really works directly on the codebase using text search, file matching, and LSP-based navigation slightly than embedding-based semantic search.
Consequently, OpenCode provides strong structural awareness but lacks the deeper semantic retrieval capabilities present in Cursor.
As a reminder, our original source code is not stored on Cursor servers or in Turbopuffer.
Nevertheless, when answering a question, Cursor still must temporarily pass the relevant original code chunks to the coding agent so it may possibly produce an accurate response.
It’s because the chunk embeddings can’t be used to directly reconstruct the unique code.
Plain text code is retrieved only at inference time and just for the precise files and contours needed. Outside of this short-lived inference runtime, the codebase just isn’t stored or continued remotely.
(2) Keeping Codebase Index As much as Date
Overview
Our codebase evolves quickly as we either accept the agent-generated edits or as we make manual code changes.
To maintain semantic retrieval accurate, Cursor mechanically synchronizes the code index through periodic checks, typically every five minutes.
During each sync, the system securely detects changes and refreshes only the affected files by removing outdated embeddings and generating latest ones.
As well as, files are processed in batches to optimize performance and minimize disruption to our development workflow.
Using Merkle Trees
So how does Cursor make this work so seamlessly? It scans the opened folder and computes a Merkle tree of file hashes, which allows the system to efficiently detect and track changes across the codebase.
Alright, so what’s a Merkle tree?
It’s an information structure that works like a system of digital cryptographic fingerprints, allowing changes across a big set of files to be tracked efficiently.
Each code file is converted right into a short fingerprint, and these fingerprints are combined hierarchically right into a single top-level fingerprint that represents your entire folder.
When a file changes, only its fingerprint and a small variety of related fingerprints should be updated.
The Merkle tree of the codebase is synced to the Cursor server, which periodically checks for fingerprint mismatches to discover what has modified.
Consequently, it may possibly pinpoint which files were modified and update only those files during index synchronization, keeping the method fast and efficient.
Handling Different File Types
Here is how Cursor efficiently handles different file types as a part of the indexing process:
- Latest files: Mechanically added to index
- Modified files: Old embeddings removed, fresh ones created
- Deleted files: Promptly faraway from index
- Large/complex files: Could also be skipped for performance
Note: Cursor’s codebase indexing begins mechanically each time you open a workspace.
(3) Wrapping It Up
In this text, we looked beyond LLM generation to explore the pipeline behind tools like Cursor that builds the appropriate context through RAG.
By chunking code along meaningful boundaries, indexing it efficiently, and constantly refreshing that context because the codebase evolves, coding agents are capable of deliver much more relevant and reliable suggestions.
