geniusOS
AI Transformation

RAG Done Right: Building a Retrieval-Augmented Assistant in .NET

Your LLM knows a lot, but it doesn’t know your business. Ask it about your product catalog, your internal policies, or the contents of your contracts, and it will confidently invent answers from its training data. Retrieval-augmented generation (RAG) fixes that by giving the model access to your documents at query time — retrieve the relevant content, stuff it into the prompt, and let the model answer from what it can actually see.

RAG sounds simple: index your documents, embed them, search them, generate. The demo works in an afternoon. The production system takes months. The difference is almost never the model — it’s the retrieval. In this guide, I’ll walk through building a RAG assistant in .NET that actually works: chunking that preserves meaning, embeddings that match your content, vector storage that scales, and retrieval that returns the right context the first time.

When RAG Beats Fine-Tuning (and When It Doesn’t)

Before writing any code, decide whether RAG is even the right tool. Fine-tuning changes the model’s behavior; RAG changes what the model can see. They solve different problems.

Choose RAG when:

  • Your content changes — policies, pricing, product data, contracts
  • You need citations and verifiable answers
  • You have lots of documents and limited GPU budget
  • Different users should get different answers based on permissions

Choose fine-tuning when:

  • You need to change tone, format, or domain vocabulary consistently
  • The knowledge is static and small enough to bake in
  • Latency and token cost of retrieval are unacceptable

For most enterprise content — knowledge bases, support docs, legal repositories, operational manuals — RAG is the right answer. It costs less to maintain, updates instantly when documents change, and can point to the source of every claim.

Chunking: The Most Underrated Step

Retrieval quality starts and ends with how you split your documents. Get chunking wrong and no embedding model or reranker will save you. There are three mistakes I see constantly: chunks too small (lose context), chunks too large (dilute relevance), and split points that butcher meaning.

Fixed-size chunking — split every N characters with overlap — is the baseline. It’s simple and predictable, but it cuts sentences and tables in half. For .NET you can write it in a few lines, but I don’t recommend it as your final approach.

Recursive or structural chunking — split by headings, paragraphs, and then sentences, in that order — respects document structure. Markdown and HTML have natural boundaries: split on ## headings first, then paragraphs, then sentences. This is what most production pipelines use, and it works with any content type.

Semantic chunking — use an embedding model to detect topic shifts and cut there — gives the best retrieval quality but costs more at index time. It pays off on long, unstructured documents where paragraph boundaries don’t track meaning.

Practical rules for enterprise content:

  1. Target 300–800 tokens per chunk for general knowledge; go smaller (150–300) for legal or technical content where precision matters
  2. Overlap by 10–20% so context spanning a boundary isn’t lost
  3. Keep tables, lists, and code blocks intact — split around them, never through them
  4. Attach metadata to every chunk: source document, section heading, page number, last-updated date, and any permission tags

That last point matters more than people expect. Metadata lets you filter before semantic search, and it gives the model context to produce better answers and honest citations.

Embeddings: Picking the Right Model

The embedding model determines what “similar” means in your search. For .NET, you have solid options without leaving the ecosystem:

  • Azure OpenAI embeddings (text-embedding-3-*) — strong quality, easy integration with Azure.AI.OpenAI, dimensions from 256 to 3072
  • OpenAI embeddings — same models via the OpenAI .NET SDK
  • Local models (e.g. via ONNX or ML.NET integrations) — for air-gapped deployments or when data can’t leave the network

Three practical tips:

  1. Match the embedding model to your content. Code, legal text, and conversational support tickets embed differently. Test on a sample before committing.
  2. Normalize and batch. Embed in batches of 64–256 and normalize vectors; it keeps similarity scores consistent.
  3. Don’t re-embed everything on every update. Store the model version with each vector and only re-embed changed documents.

If you’re on .NET 8+, Microsoft.Extensions.AI gives you a clean, dependency-injected abstraction over embedding providers — worth using so you can swap models without rewriting the pipeline.

Vector Storage for .NET Applications

Your chunks and embeddings need a home. The right choice depends on where you already store data:

  • PostgreSQL + pgvector — the default choice if you already run Postgres. No new infrastructure, SQL <=> operators for cosine distance, and you can join vectors with relational data (permissions, tenants, metadata). Excellent fit for most .NET shops.
  • Azure AI Search — hybrid keyword + vector search out of the box, with semantic ranking. Strong when you’re already on Azure and need full-text search alongside vectors.
  • Qdrant, Weaviate, or Milvus — purpose-built vector databases with advanced filtering and scaling. Worth it at very large scale or with heavy metadata filtering.
  • SQL Server — limited native vector support; generally use it only for small workloads.

For most business applications, my recommendation is pgvector. It eliminates a whole class of operational problems: one database, one backup, one connection string. You can filter by tenant or permission in the same query that does the similarity search.

Retrieval: Where RAG Succeeds or Fails

The generation step is table stakes — any decent LLM will write a plausible answer. The retrieval step is where quality is won or lost. A naive “embed the query, take the top 5 neighbors” approach fails on real content because:

  • Lexical matches matter. “Refund policy” as a phrase won’t surface chunks about “returned payments” unless the embedding captures it — often it doesn’t, especially for jargon-heavy internal documents.
  • The top 5 by similarity are often 5 versions of the same paragraph. You need diversity, not just similarity.
  • A single generic query is a weak probe. Users ask vague questions; retrieval needs multiple angles.

The fix is hybrid search: run keyword search (BM25-style) and vector search in parallel, then fuse the results. This catches both exact terms and semantic matches. On top of that, add a reranker — a cross-encoder model that scores query-chunk pairs — to reorder the fused results. Reranking is the single highest-leverage improvement you can make to RAG quality, often worth more than switching embedding models.

In practice, a production .NET retrieval pipeline looks like:

  1. Expand the user query into 2–3 search variants (original, keyword-focused, question-form)
  2. Run vector search and full-text search in parallel against pgvector
  3. Merge and dedupe candidates
  4. Rerank the top ~50 with a cross-encoder
  5. Filter by permission and tenant metadata
  6. Take the top 5–8 chunks, plus their source metadata, into the prompt

Building It in .NET

Here’s the shape of a minimal production-ready pipeline in modern .NET:

Indexing:

// 1. Load documents, split into chunks with metadata
var chunks = Chunker.Split(document, new ChunkOptions
{
    Strategy = ChunkStrategy.Structural,  // headings -> paragraphs -> sentences
    MaxTokens = 600,
    OverlapTokens = 80
});

// 2. Embed in batches
var embeddings = await embeddingGenerator.GenerateAsync(chunks.Select(c => c.Text));

// 3. Store in Postgres with pgvector
await using var cmd = db.CreateCommand();
cmd.CommandText = """
    INSERT INTO document_chunks (doc_id, heading, content, embedding, meta)
    VALUES ($1, $2, $3, $4::vector, $5)
    """;

Retrieval:

// Hybrid: vector similarity + full-text search, fused and reranked
var vectorHits = await SearchVectorAsync(question, tenantId);
var lexicalHits = await SearchFullTextAsync(question, tenantId);
var fused = Fuse(vectorHits, lexicalHits);
var reranked = await reranker.RerankAsync(question, fused.Take(50));

Generation:

var context = string.Join("\n\n", reranked.Take(6).Select(c => c.Content));
var answer = await chatClient.CompleteAsync($$"""
    Answer the question using ONLY the context below.
    Cite the source document for each claim.
    If the context doesn't contain the answer, say so.

    CONTEXT:
    {{context}}

    QUESTION: {{question}}
    """);

For the plumbing — Microsoft.Extensions.AI covers the chat and embedding abstractions, and Microsoft.SemanticKernel adds orchestration if you want plugins and function calling. You don’t need either to build a working system, but they keep the code maintainable as the assistant grows.

Evaluation: You Can’t Improve What You Can’t Measure

RAG systems look great on two cherry-picked examples and fail on the hundredth. Before you ship, build a small evaluation set: 50–100 real questions from your actual users, each with the correct source chunk(s) and a model answer. Then measure:

  • Retrieval recall@k — did the right chunk make it into the top k?
  • Answer faithfulness — does the answer stay within the retrieved context (no hallucination)?
  • Answer completeness — does it cover what the question asks?
  • Latency and cost — p50/p95 time per query, tokens per query

Keep the eval set in the repo and run it on every change to chunking, embeddings, or retrieval. It’s the only way to know whether that new reranker actually helped or just moved the failure elsewhere.

Common Failure Modes (and Fixes)

  • “The assistant doesn’t know X” — the chunk isn’t being retrieved. Check recall first: is the content indexed? Is the chunking cutting the relevant section in half? Is metadata filtering dropping it?
  • “The assistant gives wrong answers confidently” — the model is ignoring the context or the context is ambiguous. Add a strict instruction to answer only from context, and include source metadata with each chunk.
  • “Answers are outdated” — the index is stale. Track document versions and re-index on change; include last_updated in the prompt so the model can flag age.
  • “Slow queries” — index your vector columns (HNSW/IVFFlat in pgvector), cap candidates before reranking, and cache frequent questions.

What This Means for Your Business

A RAG assistant is one of the highest-ROI AI projects a company can build — support deflection, internal knowledge search, contract Q&A, onboarding copilots — because it turns the documents you already own into an answerable system, with citations and without retraining. The difference between a demo and a deployable product is boring engineering: chunking that preserves meaning, embeddings that match your content, hybrid retrieval with reranking, and an eval set that tells you when it’s actually working.

At geniusOS, we build these systems for businesses on the .NET stack — from a single internal knowledge assistant to full AI-powered customer-facing copilots, integrated with your existing data and permission model. If you’re tired of chatbots that make things up, let’s build one that reads your actual documents.

👉 Talk to us about building a RAG assistant — or explore our AI business transformation services to see how retrieval-augmented AI fits into your broader AI roadmap.