proofrag/
RAG Eval Scorecard
76/100
3 of 8 cases need attention
Overall generation quality: mean of groundedness, correctness, completeness, citation quality. Weakest metric: Precision@5 (55).
Generation · proofrag
Retrieval @5 · rank-aware
Cases · worst first, click to expand
GrndCorrCompCiteNDCG@5
How long are audit logs retained on the enterprise plan?
single_doc
Wrong (retention is 2 years) and no relevant context was retrieved.
201530200
Wrong (retention is 2 years) and no relevant context was retrieved.
- Answer
- Forever.
- Retrieval
- Recall@5 0 · Precision@5 0 · NDCG@5 0 · MRR 0
Does the free plan include webhook retries and a dead-letter queue?
multi_doc
Hallucinated: only retries are free; DLQ is paid. Retriever ranked the pricing doc low.
3020402539
Hallucinated: only retries are free; DLQ is paid. Retriever ranked the pricing doc low.
- Answer
- Yes, the free plan includes both.
- Retrieval
- Recall@5 50 · Precision@5 20 · NDCG@5 39 · MRR 33
How does SSO group mapping interact with custom roles?
multi_doc
Partially right; the precedence rule is stated backwards.
6055504563
Partially right; the precedence rule is stated backwards.
- Answer
- Groups map to roles automatically; custom roles override defaults.
- Retrieval
- Recall@5 50 · Precision@5 40 · NDCG@5 63 · MRR 50
What regions support the EU data residency tier?
single_doc
Correct but omits the Paris region the reference lists.
9085607092
Correct but omits the Paris region the reference lists.
- Answer
- Frankfurt and Dublin.
- Retrieval
- Recall@5 100 · Precision@5 40 · NDCG@5 92 · MRR 100
What is the maximum payload size for the batch endpoint?
single_doc
Accurate and grounded.
88908075100
Accurate and grounded.
- Answer
- 10 MB per request.
- Retrieval
- Recall@5 100 · Precision@5 80 · NDCG@5 100 · MRR 100
Which auth methods does the CLI support?
single_doc
Grounded and complete.
92888580100
Grounded and complete.
- Answer
- API key and OAuth device flow.
- Retrieval
- Recall@5 100 · Precision@5 80 · NDCG@5 100 · MRR 100
How do I rotate an API key without downtime?
single_doc
Fully grounded and matches the reference.
95928885100
Fully grounded and matches the reference.
- Answer
- Create a new key, deploy it, then revoke the old one.
- Retrieval
- Recall@5 100 · Precision@5 60 · NDCG@5 100 · MRR 100
What is the CEO's home address?
unanswerable
Correctly refused an unanswerable question.
10010010090—
Correctly refused an unanswerable question.
- Answer
- I don't have that information in the provided context.
- Retrieval
- Recall@5 — · Precision@5 — · NDCG@5 — · MRR —
85+ good65–84 fair
under 65 needs attention