LangChain Mastery

Course Content

LangChain Mastery

7 sections · 109 lessons

Write a function to validate LangChain LLM outputs.

Three checks before an invoice reaches the ledgerStructuredoutput:Invoice schemaShape —types, GSTIN,no negative sumsRules —subtotal plusGST equals totalGrounding —invoice numberis in the PDFPass: save.Fail: humanreview queueAbout 3% go to review, each with the rule that failed.
A schema proves the shape, not the facts — the totals rule and the grounding check are what catch a valid-looking wrong number.

What you need to know

An LLM can return valid-looking JSON with wrong values, or text that is not JSON at all. Validation happens at three levels.

LevelQuestion it answersTool
ShapeIs it the right structure and types?Pydantic model, structured output
RulesDo the values make sense together?Pydantic validators, your own checks
GroundingAre the values actually in the source?String or number match against the input
Python
from datetime import datefrom pydantic import BaseModel, Field, model_validatorclass Invoice(BaseModel):    vendor: str    gstin: str = Field(pattern=r"^\d{2}[A-Z]{5}\d{4}[A-Z][1-9A-Z]Z[0-9A-Z]$")    invoice_number: str    invoice_date: date    subtotal: float = Field(ge=0)    gst: float = Field(ge=0)    total: float = Field(gt=0)    @model_validator(mode="after")    def totals_add_up(self):        if abs(self.subtotal + self.gst - self.total) > 1:            raise ValueError("subtotal + gst does not equal total")        return selfextractor = prompt | llm.with_structured_output(Invoice, include_raw=True)def extract_invoice(text: str) -> Invoice | None:    result = extractor.invoke({"text": text})    if result["parsing_error"] is not None:        return None                              # send to review queue    inv = result["parsed"]    if inv.invoice_number not in text:           # grounding check        return None    return inv

The Field constraints check each value (GSTIN format, no negative amounts). The model_validator checks values together. With include_raw=True a failure does not raise; you get parsing_error and the raw message, which you can log. The last check makes sure the invoice number was copied from the document, not invented.

When structured output is not available

Use PydanticOutputParser: put parser.get_format_instructions() in the prompt, then prompt | llm | parser. It raises OutputParserException on bad output. OutputFixingParser, which sends bad output back to the model for repair, now lives in langchain_classic.output_parsers; a simple with_retry or one manual retry with the error message usually does the same job.

A real-life example

An invoice extraction chain at a Chennai logistics firm handles 900 invoices a week. Before validation, about 1 in 30 records had a total that did not match subtotal plus GST, often because the model read a line-item amount as the total. After adding the Invoice model with the totals rule and the GSTIN pattern, those records were caught before reaching the accounting system. About 3% of invoices go to a human queue, and each one arrives with the raw model output and the exact rule that failed, so a clerk fixes it in under a minute.

Follow-up questions to expect

  • "with_structured_output or an output parser?" — Structured output when the provider supports it, because generation itself is constrained; a parser for models that don't.
  • "What does include_raw=True give you?" — A dict with raw (the AIMessage), parsed (the object or None) and parsing_error, so failures can be logged instead of crashing.
  • "Should you retry on validation failure?" — Once, with the error message in the prompt. If it fails twice, send it to a person; more retries rarely fix a real misreading.