Measure a Clinical Process Without Hiding Missing Work

A percentage can look precise while answering the wrong question. A referral audit may count only returned consultation notes, exclude unresolved referrals, and then report excellent completion. A before-and-after comparison may reflect different patients or follow-up periods rather than a better workflow. The remedy begins with a defined question, a reproducible codebook, and transparent handling of uncertainty.

This methods guide includes a complete synthetic dataset, worked calculations, a coding exercise, and a blank analysis worksheet. Every data value is invented for teaching. There are no findings about a real practice, product, physician, or institution.

Define the unit before collecting data

Choose whether the analysis counts patients, encounters, orders, referrals, or tasks. One patient may have several orders; one referral may create several administrative tasks. Treating those units interchangeably can distort the result.

Write the eligibility rule and observation window first. For this exercise, the unit is an investigation order, all 20 orders begin on teaching day zero, and the status is assessed at the end of day 14. “Completed” means the source packet contains evidence that the investigation occurred within that window. It does not mean the result was interpreted, communicated, or clinically beneficial. Those would require additional variables.

Use a codebook that preserves uncertainty

The exercise uses four mutually exclusive statuses: completed, pending, clinically canceled, and unknown. Pending means the packet positively documents an unfinished investigation at the cutoff. Unknown means the available packet does not establish its status. Canceled requires a documented clinical disposition; an unexplained disappearance is not cancellation.

CMS’s denominator-specification guidance cautions against turning systematically missing data into a denominator exclusion. That formal measure-development principle illustrates the problem here: excluding unknown work can make the reported performance look better without improving care.

For a real audit, define source hierarchy, acceptable evidence, duplicate handling, time origins, and how amendments are coded. Keep the codebook version with the dataset so another reviewer can reproduce the classifications.

Fictional dataset: 20 orders at a common cutoff

Entirely synthetic teaching data. “Day” is the documented completion or cancellation day; a dash means that event is not established. All orders share the same day-zero starting point.

Work the calculations and name the question

Documented investigation completion is 12 of 20 orders, or 60%. Pending work is 4 of 20, or 20%; clinically canceled work is 2 of 20, or 10%; unknown status is 2 of 20, or 10%. These categories sum to all 20 orders.

If the question instead concerns documented disposition, completed plus clinically canceled yields 14 of 20, or 70%. That is a different measure. It should not be labeled investigation completion, because cancellation did not perform the investigation.

Among orders not clinically canceled, documented completion is 12 of 18, approximately 66.7%. State that exclusion explicitly. Removing unknown cases as well produces 12 of 16, or 75%, but conceals missingness and changes the question again. A reader should see why those denominators differ before comparing percentages.

For the 12 completed orders, the completion days sum to 79. The mean is approximately 6.6 days, and the median is 6 days. These figures describe completed orders only. They omit pending and unknown cases and therefore do not establish the typical delay for every eligible order. A full time-to-event analysis would require suitable methods and quantitative expertise.

Resolve coding disagreement without deleting its history

Coding exercise: Reviewer A labels S16 pending because an access task remains open. Reviewer B labels it canceled because a scheduling note says “unable to book.” Which code is supported?

Worked answer: Under this codebook, pending is appropriate. There is affirmative evidence of unfinished work and no documented clinical cancellation. Preserve both initial codes, the source reviewed, the adjudicator’s decision, and the reason. If the source itself is ambiguous, revise the uncertainty classification rather than forcing agreement.

If two reviewers initially agree on 19 of 20 records, their observed agreement is 95%. That simple calculation does not by itself establish a reliable measurement instrument. Category prevalence, sampling, ambiguous definitions, and other sources of disagreement still matter. Use qualified methods when making a formal reliability claim.

Compare periods without confusing case mix and improvement

Before comparing two periods, align the unit, eligibility, source access, observation window, and coding rules. Ask whether one period contains more urgent orders, more outside facilities, different service availability, or different patient barriers. A shorter observed delay may reflect a changed population or incomplete follow-up.

Prespecify clinically meaningful categories where possible and report relevant differences. Do not search through many subgroups until one appears favorable and present it as the original question. CMS’s measure-testing guidance emphasizes evaluating exclusions and their potential distortion. A local audit likewise needs to show what was removed and how that affects interpretation.

Report negative and inconclusive findings

A report can conclude that the sample did not demonstrate a change, that information was inadequate, or that a proposed explanation was weakened. Those are useful results. Report the question, data period, eligibility, missingness, method, and limitations with the same care used for a favorable finding.

Do not turn “no difference detected in this small sample” into “the processes are equivalent.” Do not turn an association between delay and an outcome into proof of causation. Distinguish an implementation check from a formal comparative study.

Analysis worksheet

Complete the fields before extracting real data. Preserve revisions with dates.

For real projects, obtain the relevant institutional determinations before data use or intervention. OHRP’s quality-improvement FAQ explains that an intent to publish alone does not determine whether work is research. The FAQ predates the revised Common Rule and says it reflects general thinking; current regulations and the actual activity need separate assessment.

Questions readers ask

Can I count unknown status as failure?

Only if the measure explicitly asks about missing documentation and labels that outcome accurately. Unknown clinical completion is not the same fact as demonstrated noncompletion. Preserve the distinction.

Is a synthetic exercise evidence that the real workflow performs this way?

No. It teaches definitions and calculations. Real findings require real, lawfully obtained data, appropriate governance, and a method suited to the claim.

Does a project become research because we hope to publish it?

Publication intent is insufficient by itself. The activity’s purpose, design, participants, data, and applicable rules require assessment through the appropriate institutional process.

Related resources and sources

Existing foundations: research, methodology, and clinical governance. Continue with result and referral workflow, clinical AI review, and Central Valley access-study methods.

Source and scope note. Primary sources are CMS measure-development guidance and OHRP’s dated FAQ. The dataset and calculations are invented teaching material. This guide releases no real patient data, supplies no empirical findings for the site’s held research products, and does not restore withdrawn downloads. Prepared November 5, 2026 Pacific.