Experiment

Framework-Based vs First-Principles Retrieval Pipelines: Same Problem, Two Build Approaches

Compare two ways of building a retrieval-augmented system, one on LangChain with LangSmith tracing and one in plain Python with direct API calls, to show what an agentic or LLM framework actually changes in development and execution.

DESIGNED · NOT RUN YET

Hypothesis

Both approaches are valid ways to build the same kind of system. The framework build should be faster to stand up and easier to trace out of the box; the first-principles build should give tighter control over latency, token use, retries and failure handling, with fewer dependencies to maintain.

ANALOGY

Framework vs first principles, explained

THE CONCEPT

Framework-based vs first-principles builds

THINK OF IT LIKE

A meal kit vs cooking from scratch

Framework-based vs first-principles builds is like A meal kit vs cooking from scratch

Ready-made loaders, retrievers and chainsis like Pre-portioned ingredients and a recipe card
Own retrieval, retries and context assemblyis like Choosing and measuring every ingredient
Built-in tracingis like The recipe card shows each step
Token and latency controlis like Adjusting portion and cooking time yourself

A framework is like a meal kit: quicker to start and hard to get badly wrong, but you cook what the kit decided. Cooking from scratch takes longer but you control every ingredient. The analogy ignores that frameworks keep improving underneath you.

Problem

Teams often pick a framework by default, or avoid one by habit. Neither is a decision. We have one system of each kind, so we can compare them on the same criteria instead of arguing from preference.

Setup

The two systems compared: Framework-based reference build: LangChain's standard components for loading, splitting, retrieval and generation, a dedicated prompt layer, and LangSmith tracing. First-principles pipeline (the SDLC artifact pipeline): hand-written ingestion and speaker-aware chunking, a FAISS vector index plus a custom BM25 keyword index fused with Reciprocal Rank Fusion, its own provider layer with retries and a local fallback, and per-step timing middleware. What we compare today, qualitatively, from the two codebases: how retrieval is composed, where state lives, how prompts are managed, how failures are retried or fall back, how runs are observed, and how many third-party layers sit between the app and the model. The measured round (not run yet): the same corpus and the same set of real questions through both, recording answer quality against expected answers, end-to-end latency, tokens per answer, retries and failures, lines of pipeline code and dependency count.

DIAGRAM

Two builds, one problem

flowchart TD
  Q([Same question]) --> A
  Q --> B
  subgraph A[Framework-based build]
    A1[Framework loader and splitter] --> A2[Framework retriever]
    A2 --> A3[Prompt layer + chain]
    A3 --> A4[Framework tracing]
  end
  subgraph B[First-principles pipeline]
    B1[Custom ingestion and chunking] --> B2[Vector + keyword search, fused]
    B2 --> B3[Own context assembly + provider layer]
    B3 --> B4[Retries, fallback, step timings]
  end
  A4 --> C[Same evaluation set]
  B4 --> C
  C --> R([Side-by-side comparison])

The same question flows through both pipelines. In one, framework components do the loading, retrieval and chaining; in the other, our own code does each step and owns the retries, fallback and timing.

Prompt / Workflow

Both builds follow the same development approach, so the comparison isolates the framework: 1. Clear layers: ingestion, retrieval, generation and presentation kept separate. 2. Prompts kept out of business logic: a prompt class in one build, a prompt library in the other. 3. Observability from the first version: LangSmith traces in one, step timings in the other. 4. Grounding rule: answer only from retrieved passages.

Input

One shared test corpus and one evaluation set of real questions with expected answers, run through both pipelines.

Output

A side-by-side table per criterion, plus a recommendation on when each approach fits.

Decision

Status: designed. The qualitative comparison is drawn from the two codebases; the measured round has not been run, so there are no figures yet.

Next Step

Build the shared evaluation set, run both pipelines on it, and publish the measured comparison here.

Use Cases

▸Choosing whether to adopt a framework for a new retrieval system
▸Explaining framework trade-offs to engineering teams