Legal Services Offshore research · Review Consistency
Reviewer agreement in privilege log preparation
A method for studying whether two trained reviewers classify privilege log entries the same way, and where the definitions rather than the documents cause disagreement.

Research context
This research brief is designed for legal operations planning. Use the findings to ask better workflow questions. Test any proposed process against the firm’s jurisdiction, professional duties, systems, and client commitments. Treat every local result as a starting point for review rather than a universal benchmark.
Question and scope
This study asks how consistently two trained reviewers classify the same privilege log entries when they use the same written standard. It compares the author, recipient, date, document type, description, and asserted basis fields. It also records where the reviewers disagree about whether an entry is sufficiently supported by the source document. The scope is administrative preparation, not the legal decision to withhold a document.
The reviewers may flag an entry for attorney attention, but the study does not ask them to decide privilege. That boundary keeps the research focused on the record and avoids turning a quality study into a legal opinion. The result is intended to improve the data dictionary, the examples, and the calibration process. A high agreement rate does not prove that the log is legally correct, and a low rate does not prove that either reviewer is careless.
Entry frame and sampling
The frame is a closed set of privilege log entries from a defined review batch. Record the batch identifier, the number of entries, the review platform, and the date the export was taken. If the log changed after the export, preserve the earlier version and note the change. Draw a sample across document types, custodians, and review outcomes.
Include entries with attachments, entries that were corrected, and entries that the preparer found difficult. A sample of only simple emails will overstate agreement because the fields are easier to read. The unit is one log entry, not one document page. If an entry covers a family of documents, define whether the family is one unit or whether each document is separate. Apply the rule consistently and report the family size so the reader can interpret the result.
Definitions and codebook
The codebook should define each field, the allowed values, and the source that controls. For the author field, specify whether the sender of an email or the signer of a letter controls. For the date field, specify whether the document date or the send date controls. Ambiguity here will appear as reviewer disagreement even when both reviewers are competent.
Include examples of correct and incorrect entries. A short annotated example is often more useful than a long rule. The examples should cover attachments, forwarded messages, distribution lists, and documents with no clear author. These are the cases where reviewers typically diverge. Version the codebook and record which version each reviewer used. If the codebook changes during the study, the later entries are not directly comparable with the earlier ones. Either freeze the codebook for the sample or report the change as a limitation.
Calibration and independence
Train both reviewers on a small set that is not part of the main sample. Discuss the disagreements, revise the codebook if needed, and repeat the calibration. The goal is a shared standard, not a memorized answer key. During the main review, the reviewers should classify independently and should not see each other's answers.
Preserve the initial ratings before any discussion. If the workflow requires immediate adjudication, record the initial rating separately from the final one. A third reviewer can adjudicate disagreements, but the adjudication should not erase the disagreement. Report the initial agreement, the final agreement, and the number of cases that required adjudication. That three-part result is more informative than a single consensus rate.
Measures
Report agreement by field and by overall entry. A log may have strong agreement on dates and weak agreement on descriptions, and the overall rate can hide that difference. Use counts and percentages, and show the number of entries behind each result. For categorical fields, report how often the reviewers chose the same value and how often they chose different values.
For free-text descriptions, define a coding scheme before measuring agreement, such as whether the description identifies the document type, the participants, and the asserted basis. Do not compare free text with an exact-match rule unless the codebook requires exact wording. Report the direction of disagreements where possible. If one reviewer consistently marks a basis as unsupported while the other accepts it, the team may need a clearer rule or a different approval path. A simple disagreement count will not reveal that pattern.
What disagreement can mean
Disagreement can mean the codebook is ambiguous, the source document is hard to read, the reviewers received different training, or the sample includes genuinely close calls. It can also mean one reviewer is applying a stricter standard than the other. The study should distinguish these possibilities rather than treating all disagreement as error. Review a few disagreement cases together and ask each reviewer to explain the rule they applied.
If both can cite the codebook but reach different answers, the codebook needs an example. If one reviewer cannot cite a rule, the training or the reference material may be incomplete. A pattern tied to one document type is a design clue. Attachments, privileged communications with multiple recipients, and documents created by an assistant are common sources of ambiguity. Add examples for those cases before expanding the review.
Limitations
The study uses a sample from one or a few batches, so it cannot describe the firm's entire practice. The reviewers know they are participating in a study, which may change their behavior. The sample may exclude entries that were never logged or documents that were withheld without an entry. The study does not measure legal correctness.
Two reviewers can agree on an entry that the supervising attorney later rejects. Agreement is a consistency measure, not a validity measure. The report should say this clearly so the result is not overread. Resource constraints may limit the number of entries that can be double-reviewed. A small sample can still be useful for finding definition problems, but it cannot support a precise estimate. Report the sample size and the uncertainty in plain language.
Operational use
Use the findings to revise the codebook, add examples, or change the calibration process. If disagreement clusters around a field that the team rarely uses, consider removing the field or replacing it with a controlled list. If disagreement clusters around attachments, add a family-level rule. Set a review cadence and a target for the next round.
The target should be about the process, such as reducing unexplained disagreement in two fields, not about forcing reviewers to agree. A team that understands why it disagrees can improve faster than a team that is simply told to be consistent. Keep the study log with the codebook version and the sample identifiers. A later reviewer should be able to see what was tested and what changed. That record is the foundation for a more reliable privilege log process.
Reporting and the next review round
Report the number of entries in the frame, the number double-reviewed, and the number that required adjudication. Show agreement by field and by document type, with the counts behind each percentage. Preserve examples of disagreement in a restricted appendix so the team can see the kind of ambiguity that produced the result. Do not publish privileged content in the report; use a neutral description and a study key.
Set a date for the next round and name the codebook version that will be used. If the team changes the definitions, treat the next round as a new baseline rather than comparing it directly with the earlier result. Record the training that reviewers received and any change to the review platform. A short decision log with the finding, the change, the owner, and the result will make the next study easier to interpret.
What the report should not claim
The report should not say that the log is legally sufficient or that the firm has met a production obligation. It should say how consistently the reviewers applied the written standard and where the standard was unclear. That distinction matters because agreement can coexist with a substantive error that the attorney later corrects. The report should name the attorney or review owner who is responsible for the legal decision.
It also should not compare agreement rates across firms, matters, or time periods unless the codebook and the review method are comparable. A different document mix or a different definition of the unit can change the result. Keep the comparison inside the defined study and state the version of the standard used. A reader should be able to tell whether the same codebook and the same unit definition were used.
Sources and method
Open each source to confirm its scope, publication date, and applicability before relying on it for a legal workflow decision.