Close Calls Jump to reviewing ↓AnalysisClose Calls
A recent study led by Redgrave LLP compared aiR Review with a 24-person managed review team on the same 45,004 documents. The study showed that aiR Review found more responsive documents than the human review team, with far less attorney time. This activity gives you an opportunity to test yourself against aiR Review and two expert reviewers, Kevin Reiss of Redgrave and Jessica Maurer of Relativity’s applied legal intelligence team. Want to see if you can win in a head-to-head review challenge?
Keep scrolling to gather context on what responsiveness means in this matter. Then, code 10 documents from Redgrave’s study. Most are close calls, where the reviewers did not all agree.
You will be timed. The clock starts when the first document comes into view and stops at your last call. At the end, you will see how many documents aiR Review and the managed review team would have reviewed within the same amount of time.
Begin ↓The investigation
For the purposes of this matter, you will be reviewing documents that Mallinckrodt produced in an opioid litigation. They are publicly available in the Opioid Industry Documents Archive at the University of California San Francisco.
Every reviewer works from the same protocol. Its responsiveness standard reads:
A document is responsive if it contains evidence of Mallinckrodt's or Covidien's compliance with, violations of, or reckless disregard of federal laws and regulations regarding how pharmaceutical companies may market or promote their products, including any applicable statutory or regulatory requirements for controlled substances as appropriate. … A document is not responsive merely because it discusses or relates to sales activities or drug promotion generally.
The standard covers both compliance and violations. A document showing a sales team following the rules is responsive, and so is one showing it breaking them. The protocol sets out five areas: promotional statements, drug samples, diversion, suspicious orders, and interactions with healthcare professionals. Most of the collection is sales correspondence, so the work is separating evidence about compliance from ordinary sales talk.
, condensed to what decides responsiveness and the issue tags.
Test yourself against four reviewers
Here are 10 documents from the study's random sample. For each one, read the document and decide whether it is responsive under the protocol.
When you decide, you will see how four reviewers called it:
- Kevin Reiss, counsel at Redgrave LLP and the study's expert, who reviewed the sample blind.
- Jessica Maurer of Relativity’s applied legal intelligence team, who also reviewed it blind.
- Ray's aiR Review: aiR Review running the instructions that Ray Mangum of Redgrave wrote for the study.
- First-Pass Review Team: the study's 24-person team, which included 1 review manager, 2 team leads, and 21 contract attorneys. They used Relativity Review Center's Prioritized Review Queue functionality (formerly known as Relativity Active Learning) to decide which documents to review first. The study calls this first-pass workflow "RAL".
The passages that Ray's aiR Review cited are highlighted in the document.
The clock starts when the first document comes into view. It has not started yet, so consider reading the protocol first. Your time shows in the top bar, and you can open the protocol at any point.