Learn how LLM agents classify, route, and resolve wholesale order exceptions — cutting manual review from hours to minutes with RAG, tool-calling, and eval harnesses.
An LLM agent workflow for distributor order exceptions uses a tool-calling large language model — typically GPT-4o or Claude 3.5 Sonnet — to ingest unstructured order data from NetSuite, Microsoft Dynamics 365 Business Central, or EDI feeds, classify exception types (price mismatch, inventory short, shipping hold, compliance flag), retrieve relevant policy via RAG over your SOP docs, propose or execute resolutions through ERP APIs, and log every decision for audit. The agent runs on AWS Lambda or GCP Cloud Run, orchestrated by LangGraph or a custom state machine, with eval harnesses (Promptfoo, LangSmith) measuring accuracy against a golden set of 200+ historical exceptions before production deployment.
Regional wholesale distributors processing 5,000–50,000 orders monthly typically route 8–15% of lines to exception queues. Each exception — price override requests, backorder splits, ship-complete violations, tax-exempt certificate gaps, EDI 850/855 mapping errors — lands in a shared inbox where a $22/hr coordinator reads emails, opens NetSuite or Business Central, cross-references PDF price lists, and emails the sales rep or customer. Median touch time: 27 minutes. Annual cost for a mid-market distributor: $180K–$420K in pure labor, not counting delayed shipments, lost margin on unauthorized discounts, or chargebacks from retail partners like Walmart or Kroger for ASN failures.
The problem compounds because exceptions are semi-structured: the ERP gives you fields, but the reason lives in email threads, scanned PDFs, or a rep’s Slack message. Traditional RPA (UiPath, Power Automate) breaks when the PDF layout shifts or the customer replies in Spanish. Rules engines (Drools, custom SQL) require a developer sprint for every new exception type. Distributors need a system that reads like a human, acts like software, and improves without code changes.
The agent uses in-context few-shot classification with a structured output schema (Pydantic or Zod) instead of fine-tuning. You feed it 15–20 labeled examples per exception type — price mismatch, qty short, ship-hold, compliance, duplicate, address validation — inside the system prompt. The model outputs JSON: {"exception_type": "price_mismatch", "confidence": 0.94, "line_items": [{"sku": "WH-4402", "ordered_price": 14.50, "contract_price": 13.75}], "required_action": "approve_override", "policy_ref": "PRC-042"}. Confidence thresholds route high-confidence (>0.9) to auto-resolution, medium (0.7–0.9) to a human-in-the-loop Slack approval, low (<0.7) to the exception queue with a suggested label. No training run, no GPU, no labeled dataset beyond your 200 historical cases.
Key entities in this stack: OpenAI GPT-4o / Anthropic Claude 3.5 Sonnet, LangGraph for stateful multi-step reasoning, Pydantic for output validation, NetSuite SuiteScript 2.x or Business Central AL for ERP writes, AWS Lambda / GCP Cloud Run for serverless execution, LangSmith / Promptfoo for eval, Slack Bolt for human-in-the-loop, PostgreSQL / pgvector for RAG index.
Distributor policy lives in three places: a Confluence wiki (PDF export), a shared Google Drive folder of vendor price lists (Excel, updated weekly), and tribal knowledge in the VP of Operations’ head. The RAG pipeline ingests all three: PDFs parsed with Unstructured.io, Excel price lists normalized to CSV and embedded row-by-row with text-embedding-3-large, Confluence pages chunked by heading hierarchy. Metadata tags (doc_type: "price_list", vendor: "acme_fasteners", effective_date: "2024-01-15") let the retriever filter by recency and authority. At inference, the agent queries: “What is the current contract price for SKU WH-4402 for customer CUST-8822?” — the retriever returns the exact row from the latest price list, the agent compares it to the order line, and cites the source in its resolution note. Re-index nightly via GitHub Actions; no vector DB migration needed — pgvector on your existing PostgreSQL (RDS or Cloud SQL) keeps infra simple.
The agent never writes directly to production ERP. It proposes a structured resolution object — {"action": "create_credit_memo", "params": {"customer_id": "CUST-8822", "lines": [{"item": "WH-4402", "qty": 12, "rate": 13.75}], "memo": "Auto-resolved: price mismatch per PRC-042"}} — which lands in a review queue table (PostgreSQL). A human approver in Slack clicks “Approve” → a NetSuite Suitelet or Business Central OData endpoint executes the write with the approver’s credentials, logs the transaction ID back to the queue, and posts a confirmation to the original email thread via Microsoft Graph API or Gmail API. Rollback is a single “Void” click that triggers the inverse transaction. Every step is idempotent (deduplication key = exception_id + action_hash). This pattern satisfies SOC 2 auditors and your VP of Finance.
Before go-live, you build a golden set: 200 real exceptions from the last 6 months, labeled by your best coordinator with the correct type, resolution, and policy citation. The eval pipeline (Promptfoo + LangSmith) runs the agent against every case, measuring: classification accuracy (target >92%), policy citation precision (target >95%), resolution correctness (human-rated, target >88%), hallucination rate (zero tolerance — any invented policy = fail), latency p95 (<3.5s end-to-end). You also run adversarial tests: mismatched SKUs, expired price lists, contradictory SOPs, multilingual notes. The agent must either resolve correctly or escalate — never guess. Weekly regression runs on new exceptions catch drift. This eval artifact becomes your SOC 2 evidence and your sales deck for the next fixed-scope engagement.
Our AI & LLM builds start from $14k; the final fixed price depends on how many exception types, ERP objects and approval paths are in scope, and is agreed in the SOW before any code is written. A typical 6–8 week build splits its effort roughly like this:
| Workstream | Share of effort | What you get |
|---|---|---|
| Discovery and golden-set creation | ~15–20% | 200 labelled real exceptions, exception taxonomy, policy sources |
| RAG pipeline and eval harness | ~20–25% | Indexed SOPs and price rules, automated eval run in CI |
| Agent core and ERP adapters | ~25–30% | Classify → retrieve → propose → execute flow, NetSuite or Business Central reads/writes |
| Human-in-the-loop approvals and audit log | ~15% | Slack or Teams approval cards, full decision trail |
| UAT and handoff | ~10% | Acceptance run against the eval thresholds, runbooks, repo handover |
Running costs are third-party bills you pay directly: LLM API usage (scales with exception volume and turns per case), a small Postgres instance with pgvector, and serverless compute. No GPU, no fine-tuning, no vector DB licence. We scope this as a fixed-price SOW with acceptance criteria tied to the eval harness thresholds above.
Example scenario (illustrative, not a client project): a multi-branch electrical/industrial distributor on NetSuite with a small exception team. Their exception mix is dominated by vendor price mismatches, backorder split decisions, ship-complete violations, missing tax-exempt certificates and EDI mapping errors. The build uses NetSuite SuiteScript 2.1 for reads and writes, a LangGraph state machine with four nodes (classify → retrieve → propose → execute), pgvector on PostgreSQL, Slack Bolt for approvals and Promptfoo in CI. Phase one covers the two or three highest-volume exception types; the rest stay in the human queue until the eval harness shows the agent clears its thresholds on them. Purchase-order acknowledgment (EDI 855) exceptions are a natural phase two on the same agent.
It depends on exception volume and how long each one takes today. Multiply monthly exceptions by current touch time and loaded hourly cost, then compare that with the fixed build price and the monthly API bill. The eval harness gives you the auto-resolution rate to plug in before go-live, so the payback estimate is based on your data rather than a vendor promise.
Yes. The agent only needs read/write API access (REST, OData, or stored-proc wrapper). Epicor Kinetic REST, Infor CloudSuite OData and custom .NET APIs all fit the same adapter pattern. The RAG and agent logic stay identical; only the ERP adapter layer changes — typically 1–2 weeks of additional scope.
The eval harness exposes this immediately. During golden-set creation, we flag every case where the agent retrieves conflicting policies. You resolve the conflict once, update the source doc, re-index, and the agent follows the new rule. The system forces SOP hygiene — a feature, not a bug.
No. We strip PII (customer name, address, contact) before the LLM call using a Presidio-based redaction layer. Only SKU, qty, price, date, exception code, and policy text leave your VPC. For zero-SaaS requirements, we can run Llama 3.1 70B or Nemotron 3 Ultra on your AWS Bedrock or GCP Vertex AI tenancy — same agent logic, no external API calls.
You own the repo, the eval suite, and the golden set. We deliver a 2-hour runbook walkthrough: how to add exception types (add 15 examples to the prompt, run eval, deploy), how to update price lists (drop new Excel in Drive, nightly re-index picks it up), how to read LangSmith dashboards. Most teams manage it with 0.5 FTE of a developer or a technical ops analyst. We offer a quarterly eval audit retainer if you want external eyes.
Need this built? Our team does fixed-scope rag and agentic workflows — scoped, priced and delivered in weeks.
Mehedi Hasan · Senior Software Engineer