Legal Services Offshore research · Evidence Quality
Redaction QA: detecting missed identifiers in a produced set
A sampling and search protocol for testing whether a redacted production still exposes names, account numbers, or metadata that should have been removed.

Research context
This research brief is designed for legal operations planning. Use the findings to ask better workflow questions. Test any proposed process against the firm’s jurisdiction, professional duties, systems, and client commitments. Treat every local result as a starting point for review rather than a universal benchmark.
Question and decision use
This study asks how often a redacted document production still exposes identifiers that the redaction log says were removed. The decision it informs is whether the production can be released or needs another QA pass. The study measures the presence of detectable identifiers in the produced files and does not decide whether a redaction was legally required. That question belongs to the attorney and the governing order.
The question should name the identifier types the team is testing, such as names, account numbers, email addresses, and matter numbers. A broad question about whether the production is clean is hard to answer consistently. A narrow question about one identifier type can be tested with a defined search and a clear pass rule. If the team needs to test several types, report each separately rather than combining them into one pass rate.
Population and unit
The population should be a frozen list of produced documents with the redaction log and the production version recorded. Include documents that were redacted, documents that were produced without redaction, and documents that were re-produced after a correction. Exclude documents that were formally withdrawn from the production and record the reason. The unit is one document at the production version being tested.
If a document contains several pages, record the page count and test the pages that carry redactions plus a sample of the remaining pages. A document with no redactions is still a unit because a missed identifier can appear on an unredacted page. If the same document appears in more than one production, treat each production as a separate unit and record the relationship. The report should distinguish the document count from the page count.
Evidence dictionary
Define each identifier type and the search method used to detect it. A name search should include known variants and initials. An account number search should include partial matches where the format allows. An email search should cover the visible text and the extracted metadata. Define what counts as a match and what counts as a false positive, such as a common word that happens to match a name.
Define the file states that must be tested: the visible page, the extracted text layer, the document properties, the file name, and any embedded object. A redaction that passes the visible page but fails the text layer is still a failure. The dictionary should also define how to handle a document that cannot be searched, such as a scanned image without OCR. That document should be visually reviewed or flagged as unverifiable rather than counted as a pass.
Sampling and reviewer calibration
Draw the sample from the frozen production list. Stratify by document type, redaction category, and whether the document was re-produced. Oversample the documents with the most redactions and the documents that were corrected, because those are the cases most likely to reveal a process gap. Record the selection method and any replacement. Include a random sample of unredacted pages so the study can detect identifiers that were never logged.
Calibrate reviewers on a small set of documents with known issues. Compare their search results and their classification of false positives. Revise the dictionary and repeat the calibration. During the main review, double-review a subset and preserve the first ratings. A disagreement about whether a partial account number is a match is exactly the kind of case the calibration should resolve. Report the disagreement and the adjudication rule.
Measures and reporting
Report the number of documents and pages tested, the number of identifier matches, and the number of confirmed exposures. Separate a visible-text exposure from a metadata exposure and from a file-name exposure, because each requires a different correction. Report the identifier type and the redaction category for each confirmed exposure. A single count of issues hides whether the problem is a tool failure or a missed review step.
Include the time from production to QA and the time from QA to correction. A production that was tested quickly but corrected slowly may still create risk. Report the documents that could not be searched and the reason. Those documents are not passes, and the report should show them separately. If the team uses a severity scheme, define it in advance and apply it consistently across identifier types.
Analysis and interpretation
Compare exposure rates across document types and redaction categories only when the subgroup is large enough to support the comparison. Inspect the confirmed exposures before assigning a cause. A metadata exposure across many documents may indicate that the export settings were wrong, while a visible-text exposure in one document may indicate a manual error. The remedy should match the stage where the exposure entered.
Do not treat a low exposure rate as proof that the production is safe. A single exposed account number can be significant even if the overall rate is low. Report the confirmed exposures in full and name the documents that need correction. If the sample is small, describe the result as a local observation rather than a population estimate. The report should also note that a search can only detect identifiers the team knew to look for.
Limitations and sources of error
The study tests the identifiers on the search list and cannot detect an unknown identifier or a subtle reference that the list does not cover. It also cannot determine whether a redaction was legally required or whether a produced document is authentic. The result describes the sampled documents and the tested identifier types, not every page in the production. Search tools and OCR quality are additional limitations.
System changes can affect the result. A new export tool, a revised redaction process, or a change in the production format may alter what is detectable. Note the relevant dates and avoid comparing periods as if the instrument stayed constant. A small sample can reveal a tool failure without supporting a precise rate. The report should state that limitation plainly and avoid presenting a local observation as an industry benchmark.
Privacy and role boundaries
Use the minimum document information needed for the study. Replace names and identifiers with study keys in the report, restrict access to the crosswalk, and follow the firm's retention rules. Do not copy an exposed identifier into a shared note when a reference to the document and page is enough. The offshore team may prepare the sample, run approved searches, and maintain the study log.
The attorney decides whether a redaction is required, whether an exposure must be disclosed, and what remediation is appropriate. The study should not be used to make a client-facing statement about the firm's compliance. If a document contains information from another client or a sealed matter, stop and report it through the firm's incident path. The privacy rule applies to the QA study just as it applies to the production.
Worked example
A production contains a spreadsheet with a hidden column of employee identifiers. The visible pages are redacted correctly, and the redaction log lists the column. A QA search of the native file finds the identifiers because the hidden column was not removed. The reviewer records the exposure, returns the file to the production team, and checks whether other spreadsheets used the same template. The correction is logged with the tool setting that caused it, so the team can see whether the export process or the reviewer missed the column.
The example shows why a page-level review is not enough. The document looked clean to a reader and passed a visual check, but the native file still carried the data. The team corrects the export setting rather than only fixing the one file, and the next production uses the same check. The finding is valuable because it changes a repeatable step, not because it identifies a single careless reviewer. That is the difference between a QA pass that improves the process and one that only produces a longer correction list.
From finding to controlled change
Choose one change based on the most common or most consequential exposure. It might be a required native-file check, a metadata export setting, a second-review step, or a clearer redaction category. Name the owner, the due date, and the measure that will show whether the change helped. Recheck the same measure on the next production rather than declaring success after one clean set.
Keep a decision log with the finding, the change, the owner, and the result. The log helps the team see whether a change addressed the cause or only corrected one document. If the measure does not move, inspect the workflow and the tool settings again. The goal is a production that can be released with a documented check, not a production that merely looks clean on the pages a reviewer happened to open.
Sources and method
Open each source to confirm its scope, publication date, and applicability before relying on it for a legal workflow decision.