Testing whether deadline calendar monitoring catches silent failures

Question and scope

A useful scope is narrow enough to reproduce and specific enough to show whether missed deadline is being detected. State how duplicate requests are identified and whether they are merged, linked, or counted separately. Define age from the first recorded request, not from the first time someone remembered to open the case. Keep the protocol with the deadline calendar batch so the next reviewer can see what was measured. Keep the measurement note short enough to read with the case and specific enough to support the reported invoice review time. The scope is one defined population of deadline calendar cases observed from first request through verified close in the billing portal. A count without a denominator is not a measurement, especially when the queue changes size during the day. Define the start event, the stop event, the pause rule, and the treatment of a missing privilege log before collecting data. Keep the protocol with the deadline calendar batch so the next reviewer can see what was measured. If two reviewers disagree, preserve both classifications until the definition or the evidence is resolved.

Population and unit

Record how duplicate requests were matched, merged, or kept separate, because that choice changes the denominator. Exclude cases opened outside the window only when the exclusion rule was written before collection. The start event should be the first recorded request, not the first time the team opened the case. Separate cases opened before the window from cases opened and closed inside it, and report both counts. Link the observation to the exact field in the billing portal and to the privilege log that supports it. The unit is one deadline calendar case, from the opening request to the verified close, and linked events stay with that case. Record the queue at the start and end of each observation day so additions and withdrawals are visible. The stop event should be the verified destination update, not the last message in the thread. If the queue was archived during the window, report the loss and exclude the affected period rather than assuming the cases were resolved. Do not let a summary written later replace the evidence that was available at the first decision.

Definitions and measures

A case that has no destination change should be defined as cancelled or rejected, not counted as complete. Use a stable case key and a separate identity crosswalk stored in an approved system. If a case is outside the scope rule, record the rule and the date of exclusion rather than removing it silently. Define whether the measure counts requests, cases, or destination updates, and use that unit consistently. If the case moved between owners, include both owner periods in the review note and name the handoff time. A pause should have a reason, an owner, and a review date; otherwise it is an unowned case with a different label. Define allowed values for status, reason, owner, and evidence state before collection. A correction is a change to a closed record; a clarification that does not change the record is a note, not a correction. State how duplicate, cancelled, and reopened cases enter the numerator and denominator. Keep the review note with the case so the next sampler does not have to reconstruct the decision from memory.

Data dictionary and evidence

Record the version or date of the outside counsel invoice and the document management system value so a later reviewer can see what was known at the first action. Record the extraction time separately from the event time so a late entry is visible. Keep the original evidence reference next to the coded value so the reviewer can verify the classification. Use a separate field for reviewer notes so the coded values stay comparable. When a value is missing, record the missing state and the reason instead of leaving an empty field in the extract. Mark missing, unavailable, and confirmed negative values separately; collapsing them hides the difference between no evidence and evidence of absence. Version the dictionary and report which version was used for each observation period. Run a five-case pilot before the full sample and revise any field that two reviewers interpret differently. Archive the dictionary with the study so a later reviewer can reproduce the categories. At Legal Services Offshore, the deadline calendar review should end with a dated note and a named owner for the next check. Each case needs the source, the observation time, the field supported, the owner, and the reviewer. Give every case a study key that is not the client name, member number, or candidate email. Keep the source reference and the coded value in the same row, and do not overwrite the original with a later correction. Record the source location, not just the file name, so a reviewer can reopen the exact evidence. If the evidence is sensitive, record the approved reference and access path rather than copying the content.

Sampling and collection

Record the eligible count, the sample size, and every exclusion so another reviewer can reproduce the sample. A second reviewer should re-extract a small subsample and record disagreements before the full analysis. If a stratum has fewer cases than planned, report the smaller count instead of replacing it with a convenience case. Use a random start and a fixed interval when the frame is ordered, and record both. The note should distinguish what was observed, what was inferred, and what still needs an owner decision. Include adverse cases: a missing conflict packet, a conflicting source, an unavailable paralegal, and a tight deadline. Record the time of each observation so a later reviewer can distinguish a late entry from a changed decision. A sample of recent closures can miss the oldest unresolved work, which is often the most informative. Include at least one case from each owner in scope when the frame allows it. Record this decision in the study log with the case key, the review date, and the person who approved the classification. Use consecutive deadline calendar cases where possible, then stratify by owner, channel, and risk class. Keep the collection window and the number of collectors in the report; both affect how the result should be interpreted. Do not expand the sample only because the early result looks weak or strong. If a selected case is out of scope, replace it only with the next case in the same stratum and record the substitution. Keep the evidence reference beside the coded value so a reviewer can test the same case without rebuilding the queue.

Analysis and reporting

If the sample is small, report the instability and the cases that drive the result instead of a precise-looking percentage. If two measures disagree, report both and explain what each one counts. Use a simple before-and-after comparison only when the definitions and observation window are the same. When a rate changes, check whether the case mix or the evidence standard changed at the same time. If the source changes after the observation, preserve the earlier value and add a dated note instead of replacing it. A difference between two periods is a signal to inspect, not proof that a process change caused the change. Keep descriptive findings separate from causal claims. Record the sample date, the denominator, the numerator, and the exclusion count for every reported rate. Keep descriptive findings separate from causal claims and label exploratory results. At Legal Services Offshore, the reviewer should be able to explain this step to vendor from the case record alone.

Limitations

Changes in definitions, staffing, or the privilege log during the window can create a trend that is not a process effect. The cited sources describe recordkeeping, privacy, security, or control practice; they do not supply a local benchmark. If the queue changed during the window, explain how the change affects the denominator and the interpretation. A missing record is evidence about the record, not proof that the work was never done. Use the same note in the case file and the study extract; do not create a second version in a personal document. The cited public sources frame recordkeeping, privacy, security, or control practice; they do not provide a local result. The study cannot determine whether a professional judgment was correct, only whether the record supports it. If the queue changed during the window, explain how the change affects the denominator and the interpretation. Keep the limitations with the result so a local observation is not turned into a general claim. When a case needs an exception, record the reason, the owner, and the date before continuing the review.

Review and next action

Choose one next action: fix the source, simplify the privilege log, change the queue, adjust access, or pause the work until the owner decides. If the action changes conflict overlooked, record the approval from client contact before implementation. Choose a measure that can move within one cycle and a second measure that checks for unintended effects. Schedule the next review before the action begins, not after the result arrives. Keep the measurement note short enough to read with the case and specific enough to support the reported deadline completion. Repeat the protocol after the change with the same definitions, window, and exclusions. If the result is inconclusive, say so and choose a narrower question rather than claiming a trend. Set the next sample date before closing the review, so the action does not drift. Link the action to the case evidence that prompted it, not only to the meeting notes. If two reviewers disagree, preserve both classifications until the definition or the evidence is resolved.

Related research

• Testing evidence completeness in conflict packet without a convenience sample
• Measuring cycle time for matter intake from request to verified close
• Estimating rework in document index from correction and reopen records