How did they do it?
- Leveraged prior work product to train Assisted Review
- Manually reviewed less than 15 percent of the total documents
The Case at Hand
The legal department of a large financial services company—a long-standing client of CDS—reached out to the company for help with a seemingly straightforward project: e-discovery related to an ongoing investigation. The federal government was demanding documents be turned over quickly, meaning the client had a short time frame to produce relevant information.
The client initially identified 100,000 documents belonging to three custodians, and a team of contract attorneys set to work. After reviewers started coding the original data set, however, the government agency’s lawyers requested documents from a fourth custodian, who they believed was crucial to the case.
“Nearly all of the original 100,000 documents had been reviewed, and suddenly a fourth custodian shows up with 200,000 documents,” said Chris O’Connor, Director of eDiscovery Strategy at CDS. “Bringing in this single custodian tripled the size of the case.”
CDS needed to develop a plan of attack for their client, and there was no time to waste. A tight schedule for 100,000 documents now had to accommodate 300,000. Given those numbers, CDS and their client quickly determined that Assisted Review would be the best fit.
“Our goal was to quickly identify the relevant documents we’d need to review and produce, and meet the aggressive timeline our client was given,” said Chris.
Using Assisted Review
To start the Assisted Review workflow, CDS needed to identify a seed set of documents to train Relativity to make decisions on the new custodian’s documents. For this case, CDS created a seed set using previously coded documents from the original three custodians, benefiting from the work already completed by the review team. Relativity then categorized the document universe for responsiveness based on the seed set, and CDS began batching random sets of documents from within Relativity, assigning them to reviewers. And with the Assisted Review process being managed by CDS behind the scenes, reviewers could quickly start reviewing documents for the new custodian just as they had with the previous three custodians—without disruption to their workflow or decision making.
After five “training rounds”—in which the reviewers confirmed or corrected the software’s coding decisions—Relativity began showing a very low overturn rate for non-responsive documents, with reviewers agreeing with Relativity’s decisions on these documents close to 90 percent of the time. However, the overturn rate for responsive documents was close to 100 percent, as reviewers were rarely agreeing with Relativity’s decisions regarding responsive documents. At this point, it was clear this was a document universe with an incredibly low rate of responsiveness.
Realizing this universe was atypically non-responsive, CDS adjusted the workflow. First, CDS questioned whether randomly sampling the document universe was enough to ensure reviewers were seeing a sufficient number of responsive documents to properly train the system. To address this, CDS strategically sampled the document universe by using keyword searching to build a more targeted sample set—and batching out only those documents most likely to be responsive.
“Our goal was to pull in documents that would help Relativity better identify the true positives—documents the software categorized as responsive that actually are responsive,” said Chris.
CDS and the client decided that the safest, most conservative workflow would also include additional random sampling from the non-responsive universe. “We wanted to confirm there were no false negatives in the set,” said Chris.
By taking this approach—randomly sampling the non-responsive documents while using targeted sampling to uncover the few documents that might be responsive—CDS could provide more assurance to their client that no responsive documents were being incorrectly categorized as non-responsive, while also providing the software with more examples of responsive documents. The documents CDS was confident were non-responsive could then be set aside, and only potentially responsive documents reviewed—even if the group of potentially responsive documents would be somewhat over-inclusive. “By round 12 of Assisted Review, the overturn rate for non-responsive documents was only 2 percent,” said Chris. “At this point, we felt confident we had identified all potentially responsive documents.”
In the end, the review team looked at only 29,000 documents, of which only 229 were identified as responsive. After 13 rounds of Assisted Review—including the original seed set, a round of keyword searching, some targeted sampling, and multiple rounds of random sampling—more than 170,000 documents were confidently determined to be non-responsive, requiring no additional review.
Achieving the Results
Time and cost were significantly reduced in this case by manually reviewing only those documents most likely to be responsive, while using targeted sampling of the non-responsive universe to confirm that Assisted Review was not creating any false negatives. Less than 15 percent of the total documents in the universe needed to be reviewed to produce highly accurate results in a limited time frame.
“This is a great example of how Assisted Review really works. In this case, Relativity helped us quickly identify the non-responsive documents that did not need to be reviewed and saved our client hundreds of attorney hours,” said Chris. “We will continue to use Assisted Review for quality control and culling non-responsive documents in the future.”