In June, Redgrave LLP published a study comparing aiR Review with a 24-person managed review team on the same 45,004 documents (Robert Keeling and Ray Mangum summarized it here). Their results showed that aiR Review found more of the responsive documents, with far less attorney time.
This post takes a second look at those results: how the study ran; what Jessica Maurer, a second expert, made of the same documents; and the documents where the reviewers disagreed.
At the end of this post is Close Calls, a short game: 10 documents from the study, most of them close calls where the reviewers did not all agree, and a timer. Make your decisions and find out whose calls best match yours. Skip ahead and play.
The study
In this simulated matter, reviewers worked through documents that Mallinckrodt produced in an opioid litigation, which are publicly available in the Opioid Industry Documents Archive at the University of California San Francisco. According to the study's responsiveness protocol, a document was responsive if it contained evidence of Mallinckrodt's or Covidien's compliance with, or violations of, federal rules on how drug companies may market their products, including the rules for controlled substances. A document was not responsive merely because it discussed sales or drug promotion generally.
Three parties applied the same protocol:
- Kevin Reiss, counsel at Redgrave LLP, was the study's expert. He labeled a control set of 250 randomly selected documents, which trained the contract reviewers, seeded the active-learning classifier, and provided examples for Ray's instructions to aiR Review. He then reviewed a separate random sample of 1,000 documents blind, without seeing anyone else's calls. Kevin's calls on that sample are the ground truth for everything that follows.
- A review team of 24 people: 1 review manager, 2 team leads, and 21 contract attorneys. They used Relativity Review Center's Prioritized Review Queue functionality (formerly known as Relativity Active Learning) to decide which documents to review first. The study calls this first-pass workflow "RAL"; here, it is the First-Pass Review Team.
- aiR Review, running instructions that Ray Mangum, a partner at Redgrave, wrote from the protocol in about 18 hours. aiR then reviewed all 45,004 documents in a single run of about an hour. Here, aiR running Ray's instructions is called Ray's aiR Review.
aiR Review and Review Center are solutions available in Relativity aiR, the AI platform for legal data intelligence, and provided by Relativity, who also publishes this blog.
How the study ran, and what Kevin's sample shows
Of the 73 responsive documents in Kevin's sample, the First-Pass Review Team found 47 (64 percent recall) and Ray's aiR Review found 64 (88 percent recall). Ray's aiR Review marked more documents to get there: 224 of the 1,000, for a precision of 29 percent, against 120 and 39 percent for the review team. Attorney time was about 18 hours for Ray's aiR Review and about 1,123 hours for the review team's active-learning phase. Extrapolating to the full document populations, the chance that the review team found more of the responsive documents than Ray's aiR Review is about 1 in 7,000.
Jessica's review
Every recall and precision figure in the study rests on Kevin's calls. To see how much the results depend on him, Jessica Maurer of Relativity's applied legal intelligence team reviewed Kevin's sample blind, without seeing his calls or anyone else's. Luke Duttweiler of our applied science team then compared Jessica's calls with the study's.
Kevin and Jessica made the same call on 96.1 percent of the documents (961 of 1,000), which is substantial agreement even after adjusting for chance (Cohen's kappa of 0.73). Their disagreements ran both ways: 15 documents only Kevin called responsive, and 24 only Jessica did. Jessica found the sample slightly richer than Kevin did: 8.2 percent responsive, against his 7.3 percent.
How often Kevin and Jessica agreed with each other, and with the First-Pass Review Team.
Kevin and Jessica disagreed on 39 documents. Each of them disagreed with the First-Pass Review Team on more than twice as many: 99 for Kevin and 100 for Jessica, mostly documents the team marked responsive and Kevin or Jessica did not.
Measuring against Jessica's calls instead of Kevin's improves the recall and precision of Ray's aiR Review:
| Reviewer | Measured against Kevin | Measured against Jessica |
|---|---|---|
| Ray's aiR Review | Recall 88%, precision 29% | Recall 91%, precision 33% |
| First-Pass Review Team | Recall 64%, precision 39% | Recall 62%, precision 42% |
On the 1,000 documents Kevin and Jessica both reviewed. Scores of 2 (borderline) or higher from Ray's aiR Review count as responsive, as in the study.
Against Kevin's calls or Jessica's, Ray's aiR Review found about nine in 10 of the responsive documents and the First-Pass Review Team about six in 10.
At a stricter cutoff, counting only scores of 3 or 4, Ray's aiR Review agreed with Kevin and with Jessica on about 94 percent of documents each, nearly as often as they agreed with each other.
Close calls
Recall and precision summarize thousands of decisions; they don't show what a disagreement looks like. Across Kevin's full sample of 1,000 documents, the First-Pass Review Team and Kevin made different calls on 99 documents. On 26 of the 99, Kevin called the document responsive and the review team did not. Ray's aiR Review agreed with Kevin on 20 of those 26. On the other 73, the review team called the document responsive and Kevin did not. Ray's aiR Review agreed with Kevin on 38 of those 73 and with the review team on the remaining 35.
aiR Review also shows its work. For each document it gives a short rationale, the considerations that cut against its call, and the passages it relied on, so a reviewer can check its reasoning and agree or disagree with it.
We picked 10 documents from the sample, most of them close calls where the reviewers did not all agree, and turned them into a short game. Read each document, apply the study's responsiveness protocol, make your call, and see how Kevin, Jessica, Ray's aiR Review, and the First-Pass Review Team called it. You will be timed. At the end you will see whose team you are on: Kevin, Jessica, Ray's aiR Review, or the First-Pass Review Team.
Test yourself: play Close Calls
How I did
I reviewed the 10 documents blind, before seeing anyone else's calls. I landed on Team Jessica: I agreed with Jessica on 7 of the 10, and with Kevin, Ray's aiR Review, and the First-Pass Review Team on 6 each. (Disclaimer: I am Jessica's supervisor, but we did not discuss our classifications until after we had both completed our reviews.) Kevin's 6 includes half a point on each of the two documents he changed after seeing the call from Ray's aiR Review. The 10 documents took me nine minutes to review. In that time, Ray's aiR Review would have reviewed about 6,750 documents.
My Close Calls card.
What surprised me was how unsure I was on a couple of the calls. I think they could have reasonably gone either way, depending on whether a reviewer followed the exact letter of the protocol or its spirit.
About the data
- The figures in "How the study ran" come from Kevin's full 1,000-document random sample and match the published paper.
- aiR Review scores each document from 0 to 4. As in the paper, a score of 2 (borderline) or higher counts as responsive, and documents that Ray's aiR Review could not process are counted with its responsive calls: 12 of the 1,000, three of them responsive per Kevin.
- Kevin's calls are from his blind first pass. In the Close Calls game, the documents he changed on his second look, after seeing the call from Ray's aiR Review, are marked.
- Counting the 18 hours Ray spent writing and refining his instructions, the effective pace of Ray's aiR Review was about 2,400 documents an hour.
- Source: Keeling, Mangum, Nelson & Reiss, Generative AI for Complex Document Review, Redgrave LLP Working Paper 2026-01 (June 2026).
