Human-in-the-Loop AI Document Review System
A prototype for an AI-powered document review tool where users review, edit, approve, or reject AI suggestions.
Try the live demo- Role
- Technical Product Designer and AI Prototype Developer
- Project type
- Product & Prototype Case Study
- Duration
- 6 weeks
- Tools
- Next.js, TypeScript, Tailwind CSS, Supabase, Figma, LLM APIs
01
Executive Summary
The project delivers a review workspace where an AI model proposes a summary, risk points, and recommended actions for a business document, and a human decides what is accepted. Every suggestion carries its source passage and a confidence score, and every human decision is logged for audit and model evaluation.
02
Business Problem
AI systems can generate useful outputs, but users often do not trust them because they cannot explain why a decision was made. In business environments, incorrect AI outputs can create legal, financial, or operational risks.
03
Stakeholders
- Reviewers and analysts
- Legal and compliance
- Team leaders
- Data protection officer
- IT and security
- Management
04
Discovery and Research
- How reviewers read documents today
- Which mistakes are most costly
- What evidence a reviewer needs to trust a suggestion
- Document formats and quality
- Audit and retention requirements
- Acceptable latency and cost per document
05
Objectives
- Reduce document review time by 25%
- Make every AI statement traceable to a source passage
- Keep the human as the approving decision maker
- Capture structured feedback on every suggestion
- Produce an audit trail suitable for compliance review
06
Proposed Solution
The user uploads a document or pastes text. The system extracts a summary, key risks, missing information, recommended actions, and a confidence score, then highlights the passage that supports each statement. The reviewer can accept, edit, reject, request an explanation, or flag an issue, and the audit log records what the AI suggested, what the user changed, who approved it, and when.
V
Visual analysis (D3)
Every figure below is generated with D3 from this case study’s own data and is annotated using Tamara Munzner’s what / why / how framework: the data abstraction, the abstract task it supports, and the visual idiom with its marks and channels.
V1Risk exposure by impact
- High
- Medium
- Low
What — dataItems are identified project risks; one ordered attribute (impact) and one categorical attribute (mitigation, on hover).
Why — taskRank the risks and identify the extremes that need mitigation first.
How — idiomDot plot: point marks, position on a common ordered scale as the primary channel, colour hue as a redundant encoding.
V3Implementation sequence
What — dataItems are roadmap phases with an ordered key (sequence) and a quantitative attribute (workstreams per phase).
Why — taskSummarise the delivery order and see where the workload concentrates.
How — idiomGantt-style ranged bars: line marks with length encoding magnitude, aligned on a shared horizontal scale, colour separating phases.
V4Solution architecture as a network
What — dataA node-link network: nodes are architecture components, links are data flow, and layer is a categorical attribute.
Why — taskExplore topology — trace a path from data source to governed output and locate dependencies.
How — idiomLayered node-link layout: rectangle marks positioned by layer, connection marks for flow, greyscale luminance separating adjacent layers.
V6Engagement depth profile — radar
- Documented items
What — dataOne item (this project) with seven quantitative attributes: the number of documented artefacts per workstream.
Why — taskSummarise the shape of the engagement and compare workstreams — which parts are deep and which are thin.
How — idiomRadar/star plot: line and point marks, angle channel for the attribute key, radial distance for magnitude.
V7Roadmap workload — circular barplot
- Workstream items
What — dataItems are roadmap phases with one ordered key (sequence) and one quantitative attribute (workstream items).
Why — taskCompare how much work each phase carries and see the cycle of delivery as a whole.
How — idiomCircular barplot: arc marks, angle for the ordered phase key, radial length for magnitude.
V8Architecture composition — treemap
- Component
- Layer
What — dataA hierarchy: architecture layers containing components, each component counting as one unit.
Why — taskSee where the system's weight sits — which layers hold the most moving parts.
How — idiomTreemap: containment for the hierarchy, area marks sized by component count, nested rectangles for layers.
07
System Architecture
Upload / paste text
|
v
+----------------------+
| Ingestion |
| parse, chunk, store |
+----------------------+
|
v
+----------------------+ +---------------------+
| Analysis service | --> | LLM provider |
| prompt template | | structured JSON |
| schema validation | +---------------------+
+----------------------+
|
v
+----------------------+
| Review workspace |
| citations, scores |
| accept/edit/reject |
+----------------------+
|
v
+----------------------+
| Postgres + audit |
| documents, analyses |
| user_feedback |
+----------------------+Frontend
- Next.js
- TypeScript
- Tailwind CSS
- Shadcn/UI
- React Query
Backend
- API routes
- Node.js
- LLM provider API
- Supabase
AI behaviour
- Structured JSON output
- Schema validation before display
- Confidence score per finding
- Refusal when no supporting passage exists
Audit
- Suggestion snapshot
- User edit diff
- Approver identity
- Timestamp and document version
08
UX Design
The review screen puts the document and the AI findings side by side. Nothing leaves the screen as approved without a human action.
Screens
- Dashboard: recent documents, review status, AI confidence, user actions
- Document review: original document left, AI summary and risks right, approve/edit/reject/export in the top bar
- Explanation panel: plain-language reason for each finding
- Feedback screen: usefulness, clarity, trust, and improvement notes
- Admin analytics: acceptance rate, edit rate, rejection rate, average confidence, most flagged issues
Key UX elements
- Source highlighting in the original text
- Confidence indicator with guidance on what to check
- Inline editing of AI text
- Reject with reason
- Request explanation
- Immutable audit trail
10
Core data model
| Table | Key columns |
|---|---|
| documents | id uuid, title text, content text, status text, created_at timestamp |
| ai_analyses | id uuid, document_id uuid, summary text, risk_points jsonb, confidence_score numeric, created_at timestamp |
| user_feedback | id uuid, analysis_id uuid, accepted boolean, edited boolean, rejected boolean, comment text, created_at timestamp |
11
Evaluation framework
| Signal | How it is measured | Target |
|---|---|---|
| Acceptance rate | Accepted findings / total findings | Above 70% |
| Edit rate | Edited findings / accepted findings | Below 30% |
| Citation accuracy | Manual sample review | Above 95% |
| Trust score | Post-review survey, 1–5 | Above 4.0 |
| Review time | Median minutes per document | 25% lower than baseline |
R1
Risk Analysis
| Risk | Impact | Mitigation |
|---|---|---|
| Confident but wrong findings | High | Mandatory citations, schema validation, reviewer approval |
| Sensitive documents sent to a model provider | High | Data classification, regional hosting, redaction before analysis |
| Reviewers rubber-stamping suggestions | Medium | Sampling audits, edit-rate monitoring, no bulk approve |
| Prompt changes degrading quality silently | Medium | Versioned prompts and regression evaluation set |
| Cost growth with document volume | Medium | Chunk limits, caching, model tiering |
R2
Implementation Roadmap
Week 1
Phase 1: Concept
- User interviews
- Task analysis
- Trust requirements
- Success metrics
Weeks 2–3
Phase 2: Prototype
- Wireframes
- Prompt and output schema
- Clickable review flow
Weeks 4–5
Phase 3: Build
- Working web application
- Audit log
- Feedback capture
Week 6
Phase 4: Evaluate
- Usability test with reviewers
- Evaluation report
- Roadmap for pilot
R3
Change Management
- Reviewer co-design sessions
- Training on system limits, not only features
- Published changelog of improvements from user feedback
- Named product owner for AI behaviour
R4
Success Metrics
Business Metrics
- Review time reduction
- Time saved per document
- Error reduction
- Throughput per reviewer
AI Quality Metrics
- Acceptance rate
- Edit rate
- Rejection rate
- Citation accuracy
Trust Metrics
- User trust score
- Explanation requests
- Flagged issues
R5
Deliverables
- Product concept
- User flow
- UI prototype
- Live web application
- Prompt design
- Evaluation framework
- Case study
- Demo video
“Human-in-the-loop design is essential for building trust in AI systems.”
Next project
AI ROI Calculator & Business Case Builder