Framework-Based Knowledge Retrieval Pipeline (LangChain Reference Build)
A retrieval-augmented Q&A pipeline built deliberately on LangChain, with a SOLID code structure, a dedicated prompt layer and LangSmith tracing. It is our framework-based reference build, used to compare development speed, control and observability against the first-principles SDLC artifact pipeline.
Overview
Most of our retrieval systems are built without agent frameworks. To make that choice on evidence rather than preference, we also built the same kind of system the framework way. This pipeline answers plain-language questions from a single, structured reference book with an index, used as a test corpus because its facts are easy to check. It is built on LangChain's standard components, traced in LangSmith, and organised with SOLID principles so it can be compared fairly with the custom pipeline.
Problem
General chatbots answer from what they learned in training, so for a specific business's information they can be vague, out of date or simply wrong. A client needs answers grounded in its own material, and a way to see why the bot answered the way it did.
Context
The point of this build is the comparison. The SDLC artifact pipeline solves a retrieval problem with hand-written ingestion, a custom keyword index, rank fusion and its own provider layer. This one solves a similar problem with LangChain's building blocks and LangSmith's tracing. Both follow the same development approach (clear layers, prompts kept separate, observability from day one), so the differences come from the framework, not from how carefully each was built.
Solution
Retrieval-augmented generation (RAG). Instead of trusting the model's memory, the bot first finds the passages of the book that relate to the question, then asks the model to answer using only those passages. The code follows SOLID principles, with each responsibility in its own place. Prompts live in a dedicated Prompts class rather than being scattered through the code, so they can be read, changed and versioned on their own. LangSmith tracing records each run, so we can see which passages were retrieved and what the model was given for any answer.
RAG, explained
THE CONCEPT
Retrieval-augmented generation
THINK OF IT LIKE
A librarian who looks it up before answering
Retrieval-augmented generation is like A librarian who looks it up before answering
- Indexing the source bookis like Cataloguing the library
- Retrieving relevant passagesis like Pulling the right pages off the shelf
- The model answering from those passagesis like Answering by reading those pages, not guessing
- Grounded answersis like Being able to point to the page
My Role
I designed and built the POC: the RAG pipeline on LangChain, the SOLID code structure and Prompts class, and the LangSmith tracing setup.
Architecture
Two phases, as in every RAG system. Preparing the knowledge: the book is loaded, split into passages and indexed so passages can be found by meaning, not just keywords. Answering: the user's question is matched against the index, the most relevant passages are retrieved, and the model receives the question plus those passages through a prompt from the Prompts class, which tells it to answer from the provided text. Observability: every step of each run is traced in LangSmith, which makes wrong or weak answers easy to diagnose: was the right passage retrieved, or did the model ignore it?
Preparing the knowledge base
- 1. Input
Source book (text, with index)
- 2. Action
Split into passages
- 3. Data store
Index by meaning
Workflow
1. Ingest: load the book, split it into passages, index them. 2. Ask: the user types a question. 3. Retrieve: the closest passages are fetched. 4. Answer: the model writes an answer from those passages. 5. Trace: the whole run is recorded in LangSmith for review.
How a question gets an answer
- 1. Input
User asks a question
- 2. Data store
Find matching passages
From the indexed book
- 3. Action
Prompt from the Prompts class
- 4. AI
LLM answers from the passages
- 5. Output
Answer + LangSmith trace
AI Models
Challenges
Keeping answers grounded. The core risk in any RAG bot is the model filling gaps from its own memory. Keeping the prompt focused on the retrieved text, and checking traces, is how we spot and correct that. Keeping prompts maintainable. Prompts change more often than code. Putting them in one dedicated class keeps every change in one place and makes them easy to version later. Seeing inside the pipeline. Without tracing, a wrong answer is a black box. LangSmith shows exactly which passages were retrieved for each question.
Results
The pipeline answers questions from the test corpus through a traceable LangChain pipeline, and serves as the framework-based side of a documented comparison with the first-principles pipeline. It hasn't been measured with a formal evaluation yet, so there are no accuracy figures to report.
Learnings
Ground first, generate second. Retrieval is what turns a general model into a business's own assistant. Treat prompts as first-class code. A dedicated Prompts class makes them easy to review, change and version. Trace from day one. Being able to see retrieval and generation for every answer is what makes a RAG bot improvable.
Future Scope
Planned next: LangSmith prompt versioning, an evaluation set to score answers against the corpus, and A/B testing of prompts. The same evaluation set, run against both this build and the first-principles pipeline, is what will turn the approach comparison from qualitative into measured.
Confidentiality note: A reference build on a single test corpus, kept to compare a framework-based approach with our first-principles pipelines. Configuration and security details are left out.