Your single source for new lessons on legal technology, e-discovery, and the people innovating behind the scenes.

A Practical Assisted Review Tip from the Field

Constantine Pappas

How to Deal with Non-Responsive Documents that Contain Responsive Language

After posting a couple months back about our Reviewer Protocol document—which outlines best practices for identifying good example documents while using computer-assisted review—we received a number of requests for it. Though we realize it’s not the Magna Carta or Detective Comics #27, we hope it can be a helpful reference for anyone conducting a Relativity Assisted Review project. We thought it could be useful to dive into some of its content more deeply, and share some detail on how collaborating with our users has helped us continue to improve the protocol.

Several months ago, we conducted a Relativity Assisted Review training session at a client’s office. (To set the stage, the virtual ink was not yet dry on our protocol document.) As we were going through the document with the attendees, a young attorney asked a question: “Excuse me, but what about early drafts of contracts? If we’ve agreed to carve those out and mark them as non-responsive, won’t that teach the system incorrectly?”

This was an incredibly practical observation that can only come from out in the wild, with the extra end-user perspective that drives this kind of collaboration. How should you handle a document that has responsive language but should be marked as non-responsive for external reasons?

There are two things to keep in mind.

1. The fact that it’s a carve-out makes it technically non-responsive, even if it’s comprised of highly responsive language.

2. Best practice should be to mark this draft as non-responsive and then exclude it as an example document, so the computer won’t be thrown off by the content.

After receiving this insightful suggestion, we went back and added a section to the Reviewer Protocol document called “False Negatives.” The section is meant to be a helpful guide for documents which, through an agreement between both parties, are cut out as non-responsive for more complex reasons (like subject matter, creation date, or point of origin).

Perhaps we didn’t invent the concept of monarchal accountability or introduce the world to Batman, but we did learn something from the field that we were able to share with others. As always, if you’d like to request a copy of the Reviewer Protocol document—or have any questions or suggestions for it—please feel free to email us at advice@relativity.com.


Constantine Pappas is a licensed attorney with more than 15 years of legal experience. He has served as in-house counsel and managed both paper and electronic discovery for large-scale lawsuits and government investigations. As a member of Relativity’s customer success team, Constantine helps Relativity users with workflows for text analytics and computer-assisted review.

The latest insights, trends, and spotlights — directly to your inbox.

The Relativity Blog covers the latest in legal tech and compliance, professional development topics, and spotlights on the many bright minds in our space. Subscribe today to learn something new, stay ahead of emerging tech, and up-level your career.

Interested in being one of our authors? Learn more about how to contribute to The Relativity Blog.