Method note: this brief proposes a source-led review approach. It does not claim that private client or firm data has been collected.
Question and scope
This protocol asks how often two qualified reviewers reach the same classification when they examine the same time entries. The question is about reliability, not about which reviewer is correct. A defect rate built on inconsistent classifications describes the reviewers as much as the records. The protocol should be run before a defect rate is used to compare teams, periods, or training approaches.
Name the review standard, the entry population, the reviewers, and the period. The reviewers should be qualified to apply the standard and should not be the people who created the entries. If one reviewer is a trainer and the other is new, the disagreement may reflect training rather than the standard. Record the roles and experience so the result can be interpreted.
Unit of analysis
Use one time entry as the unit and classify it against the same checklist independently. If an entry contains several defects, record the primary defect and any secondary defects using the codebook. Do not let reviewers discuss the entry before both have recorded a classification. The independence of the first pass is what makes the agreement result meaningful.
Keep the entry identifier, the reviewer, the classification, and the evidence note for each observation. If a reviewer changes a classification after discussion, preserve the original and record the change separately. The original classification is the data; the discussion is a learning event. Mixing the two makes the agreement rate look better than it is.
Defining the defect categories
Write a codebook with a definition, a positive example, and a negative example for each defect category. Categories might include insufficient narrative, incorrect matter, unsupported time, duplicate entry, and rate mismatch. A category without an example invites each reviewer to use a personal threshold. The codebook should also state what is not a defect so reviewers do not inflate the count.
Test the codebook on a small set of entries before the main review and revise any category that produces confusion. Record the revision and the date so later results can be compared. If a category cannot be made reliable, consider merging it with a broader category or removing it. A smaller reliable codebook is more useful than a detailed one that no two reviewers apply the same way.
Independent review procedure
Give both reviewers the same records, the same standard, and the same time limit. Remove identifying information that could influence the classification, such as the timekeeper's name or the reviewer's prior notes. Ask each reviewer to record a confidence level and a short evidence note for every entry. The confidence field helps distinguish a true disagreement from a guess.
Do not allow one reviewer to see the other's classification until both have submitted. Use separate forms or a system that hides the first response. If the review is conducted in person, collect the forms before any discussion begins. The procedure should be simple enough to repeat without a research team. A shared screen or a visible spreadsheet defeats the independence the study depends on.
Measuring agreement
Report the raw agreement rate and the agreement expected by chance, then calculate a chance-corrected statistic such as Cohen's kappa. Raw agreement alone can look high when one category dominates the sample. Report the counts behind the statistic and the confidence interval when the sample is large enough. If the sample is small, present the counts and avoid overstating precision.
Break the result down by defect category, because a strong overall statistic can hide a weak category. A category with low agreement may need a clearer definition or more training. Also report the entries with no defect, since agreement on clean records is part of the reliability picture. The report should show where the reviewers actually differed.
Resolving disagreements
Discuss a sample of disagreements after the independent pass and record the reasoning, not only the final answer. The discussion should identify whether the codebook, the evidence, or the reviewer's interpretation caused the difference. Update the codebook when the same ambiguity appears more than once. Do not simply tell one reviewer to match the other.
If the reviewers cannot resolve a disagreement, escalate it to the owner of the review standard. That person decides the interpretation and documents the rule for future entries. A decision that lives only in a meeting will not reach the next reviewer. The resolution should become part of the codebook and the training material. Record the date of the ruling so later entries can be compared against the same standard.
Analysis and reporting
Report the number of entries, the number of reviewers, the agreement statistic, and the category-level results. State the review standard and the codebook version used, because a different standard may produce a different result. Keep the observation forms and the disagreement log with the report. A reader should be able to see how the classifications were made.
Separate the reliability finding from any claim about the true defect rate. High agreement means the reviewers applied the standard consistently, not that the standard captures every problem. Low agreement means the defect rate should be treated as provisional until the standard improves. Give the owner a decision about the codebook or the training rather than a verdict about the team.
Limits and sources of error
Two reviewers can agree on a wrong interpretation, and a small sample can produce a misleading statistic. The order of entries, fatigue, and prior familiarity with the matters can also influence classifications. Reviewers who helped design the codebook may apply it differently from reviewers who did not. The report should state these limits and avoid generalizing beyond the observed sample.
The cited public sources frame recordkeeping, privacy, and internal control practice; they do not provide an agreement result for the firm. The protocol measures consistency under the conditions of the test, not the professional worth of any reviewer. Use the result to improve the standard and the training, not to rank people. A reliability check is a tool for better measurement.
Calibration after the study
Use the disagreement examples in a short calibration session with the reviewers who will apply the standard. Ask each person to explain the evidence that drove their classification, not only the label they chose. Update the codebook where the same ambiguity appears more than once. A calibration session is more useful than a general reminder to be consistent.
Record who attended, the codebook version used, and the examples discussed. Re-test a small set of entries after the session to see whether agreement improved. If it did not, the problem may be the evidence rather than the training. Keep the session notes with the study so a later reviewer can see how the standard evolved.
When to trust a defect rate
Set a reliability threshold with the owner of the review standard before using the defect rate for a management decision. A rate built on low agreement should be reported as provisional, with the disagreement examples visible. Do not hide the reliability result in an appendix when the main number depends on it. The reader needs to know how much confidence the classification supports.
Even a reliable rate describes the sample and the standard, not every entry in the population. Report the sample size, the period, and the codebook version beside the result. If the rate is used to compare teams, confirm that both teams used the same standard and the same evidence access. A comparison across different standards is not a comparison of performance.
Review cadence
Repeat the agreement check after a codebook change, a new review system, or a significant change in the reviewer group. Compare the same categories and report what improved and what did not. Set a review date and name the owner of the standard. A one-time agreement study becomes stale when the work changes. Schedule the repeat before the next major queue change rather than waiting for a dispute.
Choose one improvement based on the weakest category and test it in the next round. The improvement might be a new example, a shorter checklist, or a calibration session. Keep the codebook versions and the observation forms together so the change can be evaluated. The goal is a review standard that two qualified people can apply with reasonable consistency.
Put the guidance to work
Choose one recurring billing workflow, identify its authoritative source records, and name the owner who reviews exceptions. We can help turn that boundary into a focused support role.
Discuss your workflow


