Testing evidence completeness in conflict packet without a convenience sample

Question and scope

State the observation window, the sites or teams included, and the exclusions before collecting the first case. Exclude a reopened case only when the reopening did not create a new requirement; keep the exclusion count and reason. Distinguish an administrative pause from a failure: a pause waits for a named decision, while a failure leaves the next action unowned. Keep the protocol with the conflict packet batch so the next reviewer can see what was measured. Record this decision in the study log with the case key, the review date, and the person who approved the classification. The scope is one defined population of conflict packet cases observed from first request through verified close in the conflict database. State how duplicate requests are identified and whether they are merged, linked, or counted separately. Distinguish an administrative pause from a failure: a pause waits for a named decision, while a failure leaves the next action unowned. Name the population, the observation window, and the systems included in the extraction. Keep the evidence reference beside the coded value so a reviewer can test the same case without rebuilding the queue.

Population and unit

If a reopened case creates a new requirement, count it as a new case and retain a link to the earlier record. Exclude cases opened outside the window only when the exclusion rule was written before collection. A correction should be classified by the original case and by the correction event, with both dates retained. If the queue was archived during the window, report the loss and exclude the affected period rather than assuming the cases were resolved. If the source changes after the observation, preserve the earlier value and add a dated note instead of replacing it. When several deadline entry updates belong to one request, state whether the request or each update is the unit and apply that rule consistently. Exclude cases opened outside the window only when the exclusion rule was written before collection. The stop event should be the verified destination update, not the last message in the thread. If the queue was archived during the window, report the loss and exclude the affected period rather than assuming the cases were resolved. At Legal Services Offshore, the reviewer should be able to explain this step to supervising attorney from the case record alone.

Definitions and measures

Measure elapsed time from the first recorded request to the verified destination update, and report waiting intervals separately. Use a stable case key and a separate identity crosswalk stored in an approved system. A correction is a change to a closed record; a clarification that does not change the record is a note, not a correction. A definition that depends on the reviewer’s memory is not reproducible enough for this study. Use the same note in the case file and the study extract; do not create a second version in a personal document. If the definition of completion changes, keep the old and new rules side by side for the transition period. Keep the original text and the coded value together so a reviewer can check the classification. If a case is outside the scope rule, record the rule and the date of exclusion rather than removing it silently. Keep a change log for every definition adjustment and the date it took effect. When a case needs an exception, record the reason, the owner, and the date before continuing the review.

Data dictionary and evidence

Keep free-text notes separate from coded fields so the analysis does not depend on inconsistent wording. Use a controlled list for exception reasons; do not let free text become the analysis category. A missing value should have a reason code, not a blank cell or a zero. A case key should remain stable if the case moves between owners or changes status. Keep the measurement note short enough to read with the case and specific enough to support the reported conflict check cycle. Record the version or date of the matter intake and the billing portal value so a later reviewer can see what was known at the first action. Use a controlled list for exception reasons; do not let free text become the analysis category. Run a five-case pilot before the full sample and revise any field that two reviewers interpret differently. Use a separate field for reviewer notes so the coded values stay comparable. If two reviewers disagree, preserve both classifications until the definition or the evidence is resolved. Test the dictionary on five cases before the full sample; ambiguous fields should be fixed before collection, not after. Use a controlled list for exception reasons; do not let free text become the analysis category. Store sensitive evidence in the approved repository and keep only the reference in the analysis file. Use a separate field for reviewer notes so the coded values stay comparable. Link the observation to the exact field in the matter management system and to the conflict packet that supports it.

Sampling and collection

Sample across the full observation window, including the days when the queue was busiest and the days when coverage changed. If a source cannot be accessed, mark it unavailable and record the access problem rather than asking the operator to reconstruct it from memory. Use a random or systematic start rule and then take consecutive cases within each stratum. Record the date of the sample and the version of the queue export. Do not let a summary written later replace the evidence that was available at the first decision. Record the eligible count, the sample size, and every exclusion so another reviewer can reproduce the sample. Keep the collection window and the number of collectors in the report; both affect how the result should be interpreted. Stratify by risk, channel, and owner so the sample is not dominated by the easiest queue. Record the date of the sample and the version of the queue export. If the case moved between owners, include both owner periods in the review note and name the handoff time. Include adverse cases: a missing document index, a conflicting source, an unavailable vendor, and a tight deadline. If a source cannot be accessed, mark it unavailable and record the access problem rather than asking the operator to reconstruct it from memory. If a stratum has fewer cases than planned, report the smaller count instead of replacing it with a convenience case. Record the date of the sample and the version of the queue export. Keep the review note with the case so the next sampler does not have to reconstruct the decision from memory.

Analysis and reporting

Keep case-level detail available for review, because a summary chart without records cannot support a decision. Report denominators with every rate and separate missing evidence from negative evidence. If the result is unstable, say which cases create the variation and what a larger sample would test. Use a case-level table to explain an outlier before treating it as a process signal. When a value is missing, record the missing state and the reason instead of leaving an empty field in the extract. For conflict packet, report the age distribution and the share of cases waiting on a named decision rather than a single average. Keep descriptive findings separate from causal claims. Keep the case-level table available for review and summarize only the measures that answer the question. Show the age distribution and the open-case count by owner, not only the mean. At Legal Services Offshore, the conflict packet review should end with a dated note and a named owner for the next check.

Limitations

Small samples are sensitive to one case; report the instability instead of presenting a percentage as a stable rate. Small samples are unstable; report the range and the cases that create the variation. Record the response or access problems that prevented a case from being reviewed. If the reviewer performed the work, use a second reviewer for a subsample and report the disagreement. If the evidence is sensitive, record the approved reference and access path rather than copying the content. Small samples are sensitive to one case; report the instability instead of presenting a percentage as a stable rate. The study cannot determine whether a professional judgment was correct, only whether the record supports it. If the reviewer also performed the work, note the independence limitation and use a second reviewer for a subsample. State that the result describes the observed sample, not the whole industry or every team in the service. The note should distinguish what was observed, what was inferred, and what still needs an owner decision.

Review and next action

If the action affects billing error, route it to vendor and record the approval before implementation. Choose one change and one measure to test it; a long action list is hard to evaluate. Link the action to the specific conflict packet cases that motivated it. Use a short action log that names the change, the owner, the due date, and the measure that will show whether it worked. Record this decision in the study log with the case key, the review date, and the person who approved the classification. Repeat the protocol after the change with the same definitions, window, and exclusions. Keep the review note with the conflict packet evidence so the next owner can see why the process changed. Set the next sample date before closing the review, so the action does not drift. Schedule the next review before the action begins, not after the result arrives. Keep the evidence reference beside the coded value so a reviewer can test the same case without rebuilding the queue.

Related research

• Testing whether deadline calendar monitoring catches silent failures
• Measuring cycle time for matter intake from request to verified close
• Estimating rework in document index from correction and reopen records