Aditya Parab

Open to senior / staff AI engineering roles & select consulting

Agents that work in production. Evals that prove it.

I’m Aditya — an AI engineer who takes retrieval and agentic systems from whiteboard to production: framing the problem, designing the architecture, building it, and measuring it until the numbers hold.

See the work Get in touch GitHub (opens in a new tab)
trace · run_7f3a2c

“What changed in our refund policy for EU customers after the March update?”

  1. agent.plan180ms
  2. query.rewrite240ms
  3. retrieve.hybrid k=40190ms
  4. rerank 40→8180ms
  5. tool.policy_db260ms
  6. agent.reflect190ms
  7. generate.cited480ms
  8. eval.faithfulness80ms

EU customers now have a 30-day refund window (was 14) for annual plans, with pro-rated refunds after day 30 12.

latency
1.80s
tokens
3,412
cost
$0.004
grounded
8/8 claims
eval
PASS 0.96
LLM Retrieval Tool Eval Illustrative agent trace: planning, query rewriting, hybrid retrieval, reranking, a tool call, self-reflection, cited generation and an automatic faithfulness check, completing in 1.8 seconds.
Years shipping ML & LLM systems
6+
Production GenAI systems launched
12
Agent runs served per month
2M+
Curated eval cases authored
4,000+

01 / Selected work

Case studies, scored before and after.

Every system below shipped to real users. Each one is reported the way I’d report it internally: the problem, the system, how it was evaluated, and what moved.

Insurance · Lead AI engineer · 2025

Enterprise knowledge assistant

Hybrid RAG over 2.1M policy and operations documents.

Problem
Support agents didn’t trust the existing chatbot — it paraphrased policies loosely and invented clauses, so they went back to searching PDFs by hand.
System
Query rewriting, BM25 + dense hybrid retrieval, cross-encoder reranking, and citation-enforced generation with an explicit “I don’t know” path when evidence is thin.
Evaluation
1,200-question golden set built with SMEs; LLM judge calibrated against human graders (κ = 0.81) and wired into CI as a release gate.
  1. Query rewrite
  2. BM25 + dense
  3. Rerank
  4. Cited generation
  5. Abstain gate

Baseline → shipped

Answer accuracy58% to 87%

Faithfulness0.71 to 0.95

p95 latency lower is better7.4s to 2.1s

  • Python
  • OpenSearch
  • pgvector
  • Cohere Rerank
  • Claude
  • FastAPI
  • Langfuse

Claims operations · Architect & builder · 2025

Multi-agent claims triage

Supervisor + specialist agents with tool use and human hand-off.

Problem
Every incoming claim was read and routed by hand. Queues backed up for days and roughly one in eight claims landed with the wrong team.
System
A supervisor agent delegating to document-extraction, coverage-check and fraud-signal agents over typed tools, with confidence-based escalation to a human reviewer.
Evaluation
Trajectory-level evals on 900 replayed claims: tool-call correctness, routing accuracy and escalation precision, tracked per agent and per release.
  1. Intake
  2. Supervisor
  3. Specialist agents
  4. Confidence gate
  5. Human review

Baseline → shipped

Claims auto-resolved0% to 41%

Misroute rate lower is better12% to 3.5%

Handling time lower is better100% to 54%

  • LangGraph
  • MCP
  • OpenAI
  • Temporal
  • Postgres
  • Kubernetes

Platform · Creator & owner · 2024

LLM evaluation & regression platform

Versioned datasets, calibrated judges, and a CI gate on every prompt or model change.

Problem
Prompt and model changes shipped on vibes. Regressions surfaced as customer tickets, and a model migration took a quarter of manual spot-checking.
System
Dataset versioning, rubric-based LLM judges with human-agreement tracking, pairwise comparisons, and a pull-request check that blocks merges on metric regressions.
Evaluation
The platform evaluates itself: judge-vs-human agreement is monitored per rubric, and drifting judges are re-calibrated before they can gate releases.
  1. Golden sets
  2. Run candidates
  3. Judge + metrics
  4. Diff vs. main
  5. CI gate

Baseline → shipped

Eval cycle time lower is better2 days to 25 min

Regressions caught pre-release0 to 23

Model migration lower is better12 wks to 1 wk

  • Python
  • DuckDB
  • GitHub Actions
  • Ragas
  • Weights & Biases

Financial research · Solo build · 2024

Agentic research copilot

Plans, searches, cross-checks sources and drafts analyst briefs.

Problem
Analysts spent most of a day assembling each company brief from filings, transcripts and internal notes before any real analysis began.
System
A planning agent that decomposes the brief, runs parallel searches over web and internal corpora, triangulates claims across sources, and drafts with inline citations.
Evaluation
Citation precision and claim-level fact checks on 150 historical briefs, plus blind side-by-side preference tests with analysts.
  1. Plan
  2. Parallel search
  3. Triangulate
  4. Draft
  5. Self-critique

Baseline → shipped

Time per brief lower is better6 h to 45 min

Citation precision— to 0.93

Analyst preference— to 78%

  • Claude
  • Tavily
  • Qdrant
  • Pydantic AI
  • Redis

02 / Approach

Evaluation isn’t the last step. It’s the spine.

I own the whole lifecycle. Measurement runs through every phase, so “it works” always means “it scores” — on data that looks like production.

  1. 01

    Ideate

    Frame the job, not the model. Map the workflow, the failure costs, and whether an LLM is even the right tool.

    Leaves behindSuccess metrics & risk map

  2. 02

    Design

    Choose the simplest architecture that clears the bar: retrieval strategy, agent topology, tool contracts, guardrails.

    Leaves behindArchitecture & eval plan

  3. 03

    Build

    Typed tools, versioned prompts, structured outputs, and tracing from day one — production code, not notebook code.

    Leaves behindTested, traced services

  4. 04

    Evaluate

    Golden sets from real traffic, judges calibrated against humans, and regression gates that block bad merges.

    Leaves behindEval suite in CI

  5. 05

    Operate

    Watch quality, latency and cost in production; feed failures back into the eval set; tighten the loop.

    Leaves behindDashboards & feedback loop

03 / Capabilities

What I bring to the team.

Retrieval

  • Hybrid search (lexical + dense)
  • Chunking & document parsing strategy
  • Query rewriting & decomposition
  • Cross-encoder reranking
  • GraphRAG & structured retrieval
  • Citation & grounding enforcement

Agents

  • Tool design & MCP servers
  • Planning & reflection loops
  • Multi-agent orchestration
  • Human-in-the-loop escalation
  • Guardrails & policy enforcement
  • Durable, resumable workflows

Evaluation

  • Golden-set design from real traffic
  • LLM-as-judge & human calibration
  • Retrieval metrics (recall@k, nDCG)
  • Trajectory & tool-call evals
  • Regression gates in CI
  • Online A/B & feedback signals

Production

  • Tracing & observability
  • Latency & cost budgets
  • Semantic & prompt caching
  • Prompt / model versioning
  • Model migration playbooks
  • Safety, PII & access control

Toolbelt

  • Python
  • TypeScript
  • Claude
  • OpenAI
  • LangGraph
  • LlamaIndex
  • Pydantic AI
  • MCP
  • pgvector
  • Qdrant
  • OpenSearch
  • Ragas
  • Langfuse
  • FastAPI
  • Temporal
  • Docker
  • Kubernetes
  • AWS

04 / Contact

Have an agent that demos well but breaks in production?

I’m happy to talk roles, consulting, or just compare notes on evals. DMs are open on X.