During our live Ask the Doctor webinar in February—hosted by Dr. David Grossman and Jay Leib—attendees asked questions about the workflow, technology, and statistics behind Relativity Assisted Review. To keep the conversation going, we’ve assembled answers for some of the submitted questions we didn’t have time to address during the live event.
Q: After your computer-assisted review project is complete, how do you efficiently pull in family members for coding consistency checks prior to producing—without having to run a full review of responsive documents and their families?
To start, this question implies that the review team may not manually review all of the documents categorized as responsive. To level-set, the decision to review or not review categorized responsive documents and/or their family members rests with the review team.
If the review team decides to review the categorized responsive documents, then Relativity has a variety of features to make this an efficient process. A team can go about the manual review as they normally would—with some added insight into the likely responsiveness of the documents at hand.
Pulling in family members for additional review or production tasks is also a frequent exercise for linear review, and it often comes up during a privilege review. Running a saved search and including family—or creating a view of your results and including families—will provide an interface to review family members of categorized responsive documents.
Q: Do you have established benchmarks for acceptable overturn percentages? How do you know when you’re done?
Because each review is different, the definition of what it means to be “done” will always vary. For example, with traditional review, a review of another party’s production may have different expectations than an outgoing production to a government agency. For this reason, there isn’t an effective benchmark that can be applied to any given case to dictate its conclusion.
As a result, whether or not an Assisted Review project should be concluded is a decision a review team should consider, round by round, throughout the review process. Often, case teams will establish target ranges for their overturn rate or other metrics prior to starting the review. At the end of each round, they will evaluate the metrics they’ve reached and decide whether or not they fall into that target range.
Beyond the overturn rate, it may be informative to understand the volatility of categorization results and to observe if there is little to no change with each incremental quality control round. This plateau may also signify completion, unless a change in review protocol is provided.
Q: In a mixed QC sample, can you identify the margin of error (e.g. +/- 2.5 percent) for the responsive or non-responsive documents separately from one another, or can that only be done in a selective sample?
The margin of error, which results in the confidence interval, is part of the equation that determines the sample size. If you are performing a mixed QC sample, you have asked Assisted Review to randomly select documents from both the responsive and non-responsive populations. Taking this sample is akin to asking the following question: What is my overturn estimate for this population within the confidence level and margin of error I’ve selected?
Because of the mixed nature of that equation and the question it answers, trying to isolate responsive or non-responsive after the mixed sample has been taken will not as effectively answer questions about those individual buckets as utilizing selective samples would.
Q: Is there any advantage to isolating responsive documents to use for sample rounds to increase the number of responsive samples?
For the purposes of this study, we didn’t set out to speed up training or strategically select seed documents. However, our expectations are that providing high-quality seed documents, specifically for the responsive set, may lower the amount of overall training rounds—as you may more quickly begin identifying the responsive documents you need.
Q: How does the precision of computer-assisted review measure up with human coding? Is it more accurate overall?
Precision is the number of relevant and retrieved documents divided by the total number of retrieved documents. In other words, it measures the inverse of the percentage of documents that were coded as responsive by the system, but turned out to be non-responsive. Recall, on the other hand, provides a metric for how many responsive documents were not found.
Our study did not set out to evaluate human review head-to-head with computer-assisted review. However, in recent years, there have been studies of this issue which indicate that computer-assisted review achieves higher precision than manual review. For examples, check out this landmark study by David Blair and M. E. Maron, and this one by Maura Grossman and Gordon Cormack.
Recommended for you
