Citation-grounded retrieval over 19.4M judgments: how CourtMesh refuses to fabricate case law
An invented case in a filed brief is a career-ending failure mode. This is the architecture that refuses to produce one, and what it costs to enforce that refusal.
There is a category of software failure where the system crashes, and everybody knows something went wrong. Then there is the category I lose sleep over: the system produces output that looks exactly right and is quietly false. In legal research, that second category is not an inconvenience. A fabricated citation (a case name that sounds plausible, a paragraph number that fits the format, a holding that reads like something a High Court might have said) can travel from a chat window into a lawyer’s brief and from there into a courtroom. Large language models are unusually good at producing this kind of fluent falsehood, because the surface features of Indian case law (party names, neutral citations, the cadence of a judgment) are precisely the patterns a language model learns to imitate.
CourtMesh is my answer to that problem: a retrieval and chat system over 19.4 million Indian Supreme Court and High Court judgments, 49,487 Acts, and 213,540 statutory sections, built around a single non-negotiable rule. Every claim the model makes must carry a retrieved citation. If retrieval cannot support a statement, the model is not permitted to make it. What follows is how that rule shapes everything else in the system, because once you take it seriously, it stops being a feature and starts being an architecture.
Why “trust the model” was never on the table §
The tempting shortcut is to prompt a capable model to “only cite real cases” and hope. This fails for a structural reason, not a tuning one: a language model has no internal distinction between a case it has memorised and a case it has synthesised. Both are just high-probability token sequences. You cannot prompt your way out of that, because the model has no signal to attend to; asking it to be honest about its sources is asking it to introspect on machinery it does not have.
The only honest position is to move the burden of truth out of the model entirely. In CourtMesh, the model is a narrator, not a witness. The witnesses are the retrieved documents, and the model’s job is to summarise what they actually say, with a pointer back to each one. That inversion (retrieval as the source of truth, generation as presentation) is the design decision from which everything below follows.
The corpus decides the architecture §
A corpus of 19.4 million judgments is large enough that naive approaches stop working, and Indian case law has properties that punish generic RAG pipelines specifically.
First, judgments are long, internally structured documents. A single judgment can contain a factual narrative, submissions from both sides, discussion of precedent, and, crucially, the holding, which is the part that actually binds. A fixed-window chunker slices through all of this indifferently, and the damage is worse than it sounds: a chunk that straddles the boundary between counsel’s argument and the court’s conclusion can make a rejected submission look like the ratio of the case. So the chunking stage is tuned to judgment structure. The chunker splits on the document’s own units (paragraphs and holdings) rather than fixed token windows. A chunk should be something a lawyer could quote; if it isn’t, the citation attached to it is not worth much.
Second, legal relevance is not the same as semantic similarity. Two judgments can discuss nearly identical facts and be worlds apart in authority: one from the Supreme Court, one from a single-judge bench of a High Court; one good law, one implicitly superseded by a later constitutional bench. Pure vector search is blind to all of this. It will happily surface the most semantically resonant paragraph regardless of which court said it, when, or in what strength of bench. That is why retrieval in CourtMesh is hybrid: OpenSearch combines vector similarity with structured filters over court, year, bench, and citation metadata. The embedding answers “what is this about?”; the metadata answers “does this actually carry weight for the question being asked?”. In legal research, the second question is frequently the more important one, and it is the one embeddings cannot answer.
Ingestion at corpus scale §
The pipeline itself is deliberately boring in shape (parse, chunk, embed, index) because the interesting engineering is in making each stage survive the corpus. Parsing millions of judgments means confronting decades of inconsistent formatting; the parser has to produce structure reliable enough that the chunker downstream can trust paragraph boundaries, because every parsing error becomes a chunking error becomes a citation that points at the wrong text.
Embedding runs on autoscaled workers, and there are really two workloads wearing one uniform. The batch backfill is a throughput problem: churn through the historical corpus, tolerate restarts, never lose track of what has been processed. The streaming path for newly published judgments is a freshness problem: when a court hands down a decision, the window before it is retrievable is a window in which the system’s answers are silently incomplete. The two paths share the embedding workers but have opposite scaling profiles, which is exactly the situation Kubernetes autoscaling is for: the backfill can saturate whatever capacity exists, while the streaming path stays responsive because it never queues behind a bulk job.
Rerank: because recall and precision want different retrievers §
Hybrid retrieval gets you a good candidate set. It does not get you a good top-of-list, and in a citation-grounded system the top of the list is everything; those are the passages the model will be permitted to speak from. So there is a reranking stage between retrieval and generation.
The reasoning is a classic two-stage argument. First-stage retrieval must be cheap enough to run against an index built from 19.4 million documents, which forces it to score query and document independently. A reranker gets to be expensive precisely because it only sees the shortlist, and that budget buys it the thing the first stage cannot afford: joint attention over the query and each candidate together. It can notice that a paragraph mentions the right section of the right Act but in the context of distinguishing it rather than applying it: the kind of judgement call that separates a supporting citation from a misleading one. Skipping rerank does not save you this cost; it just moves it into the generation stage, where the model wastes context on weak passages and the grounding constraint forces refusals it needn’t have made.
The grounding contract §
Here is the part I consider the actual product. In CourtMesh’s chat layer, the constraint is not “please cite sources”. It is: every claim must be attached to a retrieved citation, and a claim that cannot be attached to one does not get made. The model’s permissible output is bounded by what retrieval returned.
The system is allowed to say “I could not find authority for that”, and it says so. It is not allowed to invent authority to avoid saying it.
This produces behaviour that feels unusual if you’ve grown accustomed to chatbots that always have an answer. Ask about a proposition the corpus doesn’t support and you get an honest gap, not a confident paragraph. I regard this as the system working, not failing. A refusal costs a user a few seconds of disappointment. A fabricated case costs them their credibility in front of a judge. Those are not comparable harms, and the asymmetry should be built into the architecture rather than left to the model’s mood.
How the contract is enforced §
The rule is only as real as its enforcement point. In CourtMesh that point sits between generation and the user: a gate that walks the drafted answer claim by claim and demands a supporting passage for each one before anything is shown. The shape of it, as pseudocode (the production version adds structured-output constraints, retries, and logging), but the logic is exactly this:
kept, kept_ids = [], set()
for claim in draft_answer.claims:
support = retrieved_passages.get(claim.named_passage_id) # only ids it was handed
if support is not None: # unsupported claims do not survive
kept.append(attach(claim, support.citation))
kept_ids.add(claim.id)
gaps = [c for c in draft_answer.claims if c.id not in kept_ids]
if not kept:
return refusal("No supporting authority found for this proposition.")
return answer(kept, gaps) # pruned claims are reported as explicit gaps, never silently dropped
What decides support? Support is structural, not a similarity judgement: the answer is generated under a structured-output constraint that forces each claim to name the passage it draws on, and the gate then verifies that the named passage actually exists in the retrieved set and that the citation printed to the user is the one attached to that passage. The model cannot cite what retrieval did not return, because the only citation identifiers it can emit are the ones handed to it.
Worth being precise about what that buys. The gate guarantees every citation is real and points into the retrieved set. It does not by itself guarantee the passage entails the claim as strongly as the sentence implies. That residual risk is why the passage travels with the citation: the reader can check the claim against its source in one click, which keeps the failure mode visible instead of buried.
When only some claims survive, the answer says so: the surviving, cited material is presented as what the corpus supports, and the pruned ground is surfaced as an explicit gap rather than silently papered over; a partial answer that admits it is partial beats a complete-looking one that isn’t.
Two properties matter more than the code. The gate runs outside the model, so a more persuasive model cannot argue its way past it. And failure degrades toward silence, not toward invention: the worst outcome of a gate bug is an over-cautious refusal, which is an annoyance, not a fabricated authority in a brief.
There is a subtler benefit: grounding makes the system auditable. When every sentence carries a pointer into the corpus, a sceptical user (and lawyers are professionally sceptical) can check the work. The trust the system earns is inspectable trust, which is the only kind worth having in this domain.
Caching, cost, and the shape of latency §
In front of the LLM sits a response cache, and repeat queries skip inference entirely. The economics are straightforward (inference is the expensive step, and legal research queries cluster heavily around recurring questions), but the latency argument matters just as much as the cost one. LLM inference dominates tail latency; everything before it in the pipeline is comparatively predictable. Serving repeats from cache doesn’t just shave the average, it removes the slowest component from the path altogether for a meaningful slice of traffic.
Operating it §
The whole system runs on Kubernetes, with autoscaling applied to both embedding and inference workers: the two workloads whose demand is spiky and whose idle cost is worst. Observability concentrates on the two signals that actually describe whether CourtMesh is doing its job: query latency, and retrieval quality. That second one deserves emphasis. In a grounded system, retrieval quality is answer quality: if the right passages don’t come back, the model is constrained to either a weaker answer or a refusal. Watching generation while ignoring retrieval is monitoring the narrator and ignoring the witnesses.
How retrieval quality is judged needs a sentence, because “quality” hides the method. The yardstick is a fixed set of research questions with expected authorities, and the comparison is positional: after any change to chunking, embeddings, or reranking, does the controlling judgment still surface at the top, and do the metadata filters still keep the wrong courts out? No single score is the point. The point is that a pipeline change cannot silently degrade what the model is allowed to see. It is the same regression discipline you would apply to a test suite, applied to relevance.
The stack splits along the grain of each problem: Python for ingestion, embeddings, and retrieval, where the ML tooling lives; Node.js for the API and orchestration layer, which is fundamentally an I/O coordination problem; React with TypeScript on the front end. Nothing exotic: the novelty budget was spent on the grounding contract, deliberately, because a system whose selling point is trustworthiness should not also be an adventure in infrastructure.
What I’d want a CTO to take from this §
The lesson generalises well beyond legal tech. Wherever an LLM’s output can be actioned by someone who trusts it (medicine, finance, compliance, law), the central engineering question is not “how good is the model?” but “what is the model permitted to assert, and on what evidence?”. CourtMesh’s answer is strict: nothing without a citation, and the citation must come from retrieval over a corpus the system controls, chunked to respect the documents’ own structure, retrieved with the metadata that encodes authority, and reranked so the model speaks only from the strongest evidence available.
The model never gets to be the source of truth. It gets to be articulate about the truth the system hands it. In a domain where a single invented case can end a career, I consider that constraint the most important line in the system, and the easiest one to defend to anyone who has watched a fluent model be confidently wrong.