Module 9: Human-in-the-Loop
Not every decision should be automated. Human-in-the-loop is how you build the pause button into your agent workflow.
When You Need It
- High-stakes decisions - Contract approvals, financial transactions, medical recommendations
- Regulatory requirements - Some industries require human sign-off by law
- Edge cases - When the agent's confidence is low or the input is unusual
- Early deployment - While you're building trust in the system, keep a human in the loop for everything. Gradually reduce as confidence grows.
How It Works
The agent processes the input through all automated steps, packages the results into a structured summary, and pauses. A human reviews the summary, makes a decision (approve, reject, modify), and the agent resumes.
Agent processes contract
↓
Agent packages analysis:
- Key terms extracted
- 3 compliance risks flagged
- Financial exposure: $2.1M
- Recommendation: approve with amendments
↓
⏸️ PAUSE - Waiting for legal review
↓
Attorney reviews and approves
↓
Agent generates decision letter and distributes
Implementation Patterns
Callback Pattern (Lambda Durable Functions)
The agent workflow pauses at a callback step. No compute charges while waiting. When the human responds, the workflow resumes from where it stopped.
This is the approach we covered in our Lambda Durable Functions article. The callback can wait minutes or weeks without cost.
Queue Pattern (SQS + DynamoDB)
The agent writes its analysis to a review queue. A separate application (dashboard, email, Slack notification) alerts the reviewer. The reviewer's decision triggers the next step via an API call.
Approval Tiers
Not all reviews need the same level of scrutiny.
| Risk Level | Action | Reviewer |
|---|---|---|
| Low (standard terms, known vendor) | Auto-approve | None |
| Medium (minor deviations) | Notify + auto-approve after 24h | Team lead |
| High (significant deviations) | Block until approved | Legal counsel |
| Critical (above $ threshold) | Block until approved | VP + Legal |
The agent classifies the risk level. The orchestrator routes to the appropriate approval tier.
Designing the Review Experience
The human reviewer's time is expensive. Make their job easy:
- Executive summary first - One paragraph that says what the agent found and what it recommends
- Highlight deviations - Show exactly what's different from the standard, not the entire contract
- Provide context - Link to the source document, the policy it was compared against, and the agent's reasoning
- One-click decisions - Approve, reject, or request changes with a single action
- Audit trail - Log who reviewed, when, and what decision they made
Common Mistakes
- Reviewing everything - If the agent is accurate 95% of the time and you review 100% of outputs, you're paying for a human to confirm what the AI already got right. Use risk-based routing.
- No feedback loop - When a human overrides the agent's recommendation, that should feed back into improving the agent. Track override rates and reasons.
- Blocking on slow reviewers - Set SLAs. If a review isn't completed within the timeframe, escalate automatically.
What's Next
Human-in-the-loop happens once per workflow run. But the agent also needs to remember things across runs. In Module 10: Memory, we cover how agents retain and retrieve information over time.
Human-in-the-Loop Lab
Build a complete approval workflow with Lambda Durable Functions callbacks, risk-based routing, reviewer dashboards, and audit logging.