Who Is Responsible When the Algorithm Enters the Exam Room? California’s New Test for Clinical AI

A physician opening a patient’s chart can encounter several different kinds of artificial intelligence before the consultation begins. One program prepares the visit summary. Another estimates deterioration risk. A third suggests diagnoses or medication changes. Somewhere outside the examination room, an insurer’s system may be evaluating whether a proposed service is medically necessary. Calling all of these systems “clinical AI” obscures the question that will define the next phase of policy: who has authority to make the consequential decision, and who must answer when it goes wrong?

California enters its next administration with more of that question already addressed in statute than a campaign-centered account would suggest. Laws enacted before and during 2026 establish duties involving utilization management, independent professional judgment, and biased impacts in clinical decision support. Federal device regulation operates alongside those state responsibilities. The incoming Congress can investigate, legislate, and negotiate funding, but a new majority does not erase the statutes, agency responsibilities, or professional standards already in place. The practical contest in 2027 will concern the meaning of compliance inside actual clinical workflows.¹²³⁴

Our forecast: the most consequential developments over the next two years will involve procurement contracts, monitoring, documentation, insurance coverage, and the allocation of responsibility among clinicians, facilities, insurers, and technology suppliers. A spectacular new federal AI statute is possible. A quieter change in how thousands of institutions deploy software is a more defensible planning assumption. For Fresno and the Central Valley, where public health planning already identifies access to medical, dental, and vision care as a priority, the question is especially concrete: will AI release scarce clinical capacity, or create a new administrative layer that consumes it?⁵

The evidence we have—and what it does not establish

Sources: the randomized trial, enacted bill texts, FDA guidance, and Fresno County’s community health planning materials.²³⁴⁵⁶ The table combines research findings and legal milestones deliberately: a law establishes an obligation, while a study estimates an effect in a defined setting. Neither can substitute for the other.

The trial is a useful warning against a seductive shortcut. A tool that performs impressively when tested alone may not improve a professional’s performance merely because the professional can access it. Integration, interpretation, trust, time pressure, and the design of the task all matter. That is an argument for measuring the combined human-and-software process. It is not evidence that clinical AI is inherently ineffective.⁶

A century of medicine’s unfinished accountability problem

Over the past century, American medicine has repeatedly absorbed technologies that change what clinicians can see and do. The resulting disputes are rarely about whether technology should exist. They concern the terms on which institutions may use it, the competence needed to interpret it, and the distribution of the resulting benefits and risks. The relevant historical continuity is that technological capability and institutional responsibility develop at different speeds.

The creation of Medicare and Medicaid in 1965 enlarged the federal role in paying for care. Subsequent decades brought prospective payment, managed care, increasingly consolidated delivery systems, and electronic records. Each development changed the environment in which a clinician exercises judgment. A treatment decision may be professionally individualized while its available options are shaped by payment rules, staffing, contracts, and information systems. AI intensifies that older tension because software can influence many decisions simultaneously and can be updated without the visibility associated with replacing a physical device.⁷

The last forty years offer a particularly useful lesson. Decision support cannot be evaluated only as a prediction engine. It is also an organizational intervention. A warning shown too frequently can become background noise. A recommendation embedded in an order screen can acquire authority simply because it appears to be the normal next step. A payer’s automated process can change which requests receive individual scrutiny. These are mechanisms to investigate, not assumptions that every product behaves identically.

The governing implication is straightforward. Accountability has to follow the entire decision process: the information collected, the population used to develop the model, the output displayed, the user’s opportunity to disagree, the resources available to act, and the consequences for the patient. Restricting attention to the final clinician click misses much of what a sophisticated organization can control.

What California has actually enacted

Enacted for the next governing period: AB 1979 and SB 503 were approved September 30, 2026. Their ordinary effective date is January 1, 2027; the chaptered texts do not supply a special earlier or later operative date for the duties discussed here. The Secretary of State explains the regular-session effective-date rule.⁹¹⁰ AB 1979 requires specified healthcare organizations to take reasonable steps to preserve a licensed provider’s independent professional judgment when clinical decisions are informed by decision support. It also restricts using AI to direct unlicensed personnel through functions that require licensure or to perform those licensed functions independently, subject to the text’s exceptions. Administrative communication and documentation that do not require professional judgment receive different treatment.²

SB 503 addresses another part of the chain. Its text imposes responsibilities on developers and deployers concerning reasonably foreseeable biased impacts, mitigation, information about intended uses and limitations, and monitoring. Its documentation provisions are relevant to purchasing decisions because an institution cannot assess a product adequately if material limitations are hidden behind a promotional description.³

California’s earlier SB 1120 framework addresses AI and other software used in utilization management. The Department of Insurance’s May 2025 guidance explains requirements for regulated insurers, including the role of individual clinical circumstances and qualified human decision-making. Coverage must be checked carefully: insurance regulation, health-plan regulation, and federal rules governing certain employer plans do not form one interchangeable jurisdiction.¹

These developments should not be summarized as “California protects physicians from AI liability.” The laws establish duties and preserve professional judgment; they do not confer broad immunity. Nor does a requirement for human involvement automatically determine liability in every later dispute. The facts of the deployment, the applicable claim, causation, contracts, professional obligations, and other law remain consequential. That distinction is essential for both physicians and patients.

Federal authority: a boundary that depends on function

Federal law distinguishes software functions that are medical devices from qualifying functions excluded from the device definition. FDA’s January 2026 guidance explains its approach to clinical decision support under that framework. The guidance is nonbinding agency guidance; the underlying statute, regulations, and product-specific circumstances carry distinct legal weight.⁴

The useful question is therefore not whether an institution has “FDA-compliant AI” in the abstract. It is which function the product performs, which regulatory pathway applies, what the authorized or intended use covers, and whether the institution’s implementation matches that use. A single software package can contain multiple functions. A product sold for administrative assistance should not quietly become the institution’s diagnostic authority without a new analysis.

Congress could change these boundaries through legislation, require reporting, fund evaluation, or direct agencies within constitutional limits. It can also scrutinize federal purchases and reimbursement policies. But oversight does not by itself revise a product’s legal status. Appropriations language, substantive amendments, agency action, and judicial review are different instruments with different consequences. A credible forecast names the instrument rather than treating a hearing as an enacted safeguard.

A Democratic congressional majority and a Republican president create both friction and opportunities. Consumer protection, fraud, rural access, competition, and federal purchasing can attract overlapping interests even where the parties disagree about broader regulation. Our base case is selective agreement around visible failures and administrative controls, accompanied by more contested debates over national standards and federal preemption. The strongest contrary case is a major incident that rapidly creates a larger bipartisan coalition.

The real liability question begins before the patient encounter

Consider a hypothetical health system purchasing software to prioritize referrals. The vendor supplies a risk score. The hospital selects the threshold. A department decides which patients will receive additional review. A clinician eventually signs an order. If a patient is harmed after a referral is delayed, it would be analytically weak to focus only on the final signature.

The vendor may argue that the system was used outside its intended population. The institution may argue that clinicians retained discretion. The clinician may argue that the organization provided neither the relevant validation information nor an alternative pathway. A payer may have imposed a separate authorization delay. Each proposition requires evidence. A responsible publication should resist assigning liability from a short scenario before examining the applicable law and the actual record.

For institutional governance, however, the scenario exposes actionable questions. Was the local population represented in evaluation? Was the threshold chosen to improve safety, manage volume, or reduce expense? Could users override it without unreasonable delay? Were exceptions reviewed? Did the organization keep a record of model versions and material changes? Was there a mechanism to identify delayed care after deployment?

Contracts should reflect those questions. A promise that a product is “accurate” is less useful than specific commitments about performance in intended uses, notification of changes, incident cooperation, audit access, data handling, and responsibility for unsupported claims. Contractual allocation cannot eliminate statutory or professional obligations. It can improve the information and cooperation available when something goes wrong.

Human review must be a functioning process

The phrase “human in the loop” can describe very different realities. In one organization, a clinician receives an intelligible recommendation, has time to investigate, can disagree, and can obtain a second opinion. In another, a worker faces a queue of hundreds of decisions and an expectation of near-universal acceptance. Both systems contain a human; their practical safeguards differ substantially.

The appropriate test is whether the reviewer possesses relevant competence, usable information, sufficient time, and actual authority to change the result. Institutions should also ask what happens after disagreement. An override that triggers burdensome extra work may discourage appropriate exceptions. An override that disappears from monitoring prevents the organization from learning where the tool fails. Conversely, a system that treats every disagreement as evidence of model failure may miss inconsistent practice that deserves investigation.

A workable approach measures the review process itself. How often are recommendations accepted, changed, or escalated? Which patient groups experience delays? Are clinicians correcting the same type of error repeatedly? Does the organization respond to those corrections? These questions move the debate from a ceremonial signature to the quality of the decision process.

NIST’s voluntary AI Risk Management Framework offers a useful vocabulary for organizing governance and evaluation. It is not a substitute for California law, FDA requirements, or a specialty-specific clinical standard. Its value is in helping an institution connect responsibilities across technical, clinical, procurement, and executive teams.⁸

Fresno should buy solutions to local problems

Fresno County’s health improvement plan identifies access, chronic disease, maternal and child health, mental health, environmental conditions, and other connected priorities. That provides a better starting point for local AI policy than a vendor’s general promise of transformation. The public question should be which documented problem a purchase addresses and what evidence would show improvement.⁵

An administrative assistant that reduces time spent preparing records may be valuable if it gives clinicians more time with patients. Its evaluation should include correction time, missing information, privacy obligations, and whether freed capacity actually reaches patients. A referral tool may help if it reduces avoidable delays, but its assessment should examine the entire pathway from referral to completed visit. A favorable risk score is not an appointment.

Language access deserves particular attention in any diverse service area. An organization should evaluate whether translation, summarization, and patient messaging preserve clinical meaning, including uncertainty and the difference between instructions and historical information. It should not assume that performance measured in one language transfers automatically to another. This is a proposed evaluation standard, not a claim that all local systems currently fail it.

The Valley also needs a realistic purchasing strategy for smaller practices. An independent clinic may lack a dedicated evaluation team. Shared procurement expertise, common reporting formats, university partnerships, and publicly accessible evidence could lower the cost of responsible adoption. A policy that requires every small practice to independently reproduce a sophisticated validation program may unintentionally favor larger institutions with greater administrative resources.

Who gains, who pays, and whose risk becomes invisible?

Patients can gain through faster access, better coordination, earlier recognition of deterioration, and clearer communication. Clinicians can gain through less repetitive work and more usable information. Payers and employers can gain when care is more effective or administration less expensive. Developers can earn returns by solving real problems. None of those possibilities should be presumed to occur in every deployment.

The costs are equally distributed only in a fictional model. A hospital may capture administrative savings while physicians absorb verification work. An insurer may reduce processing expense while patients spend more time appealing decisions. A vendor may improve a product using feedback generated by uncompensated clinicians. A system may show acceptable average performance while a smaller subgroup bears a disproportionate share of errors. The distributional analysis belongs beside the performance analysis.

There is also a competition question. If liability uncertainty and evaluation expense become fixed costs that only large organizations can manage, governance intended to protect patients could accelerate consolidation. That is not a reason to abandon safeguards. It is a reason to design shared evidence resources and proportionate obligations, and to distinguish a low-risk documentation tool from a high-consequence clinical function.

Public purchasers should therefore report more than estimated labor savings. They should identify which party receives the saving, what new work is created, how patients experience the change, and whether the total cost of safe operation remains favorable after correction and monitoring. Those are testable questions. A press release about the number of automated tasks is not enough.

Three plausible paths through 2028

Base case: uneven adoption with stronger institutional controls. California’s enacted framework prompts revised contracts, clinical governance committees, training, and monitoring. Larger systems move faster because they can spread evaluation costs. Smaller practices adopt administrative tools more readily than high-consequence decision support. Federal agencies continue product-specific and program-specific activity, while Congress uses oversight and selected legislation. Confidence in this direction is moderate; confidence in the speed and consistency of implementation is lower.

Improvement case: evidence becomes reusable. Academic and public partners develop evaluation methods that smaller organizations can use. Vendors disclose meaningful limitations. Purchasers reward products that reduce total workload and improve patient outcomes. Clinician feedback becomes a documented source of model improvement. In this scenario, governance supports adoption by reducing uncertainty, and regional providers can participate without building an entire technical department.

Failure case: formal compliance masks weak practice. Institutions add disclaimers and signatures while maintaining workflows that encourage uncritical acceptance. Vendors respond to scrutiny with narrow contractual language rather than better evidence. Smaller providers delay useful purchases, while larger buyers lock themselves into systems that are difficult to replace. A serious incident then triggers hurried restrictions. The early warning sign would be growth in compliance documentation without corresponding improvement in review quality, outcomes, or incident learning.

These are conditional pathways, not numerical predictions. They should be updated when implementing guidance, enforcement actions, procurement standards, and credible outcome studies become available. A dramatic demonstration should not outweigh a weak local evaluation; neither should a single failure establish that every application is unusable.

The strongest case against this forecast

The best challenge is that the analysis overestimates the importance of governance. Models could improve quickly enough that better performance becomes the dominant driver of adoption, while institutions learn through ordinary experience. Heavy requirements might delay useful tools, preserve avoidable clinician workload, and protect incumbent organizations from competitors.

That challenge deserves a substantive response. Governance can become wasteful if it demands identical procedures for every application, measures paperwork instead of outcomes, or forces local organizations to repeat evaluations that could be shared. The answer is proportionality and learning: match the intensity of review to the consequence of error, make evidence portable where appropriate, and retire requirements that do not improve decisions.

But rapid technical improvement does not resolve every organizational issue. A more capable model can still be used for an unsuitable task, integrated into a poorly designed workflow, or marketed beyond its evidence. Responsibility for monitoring, updates, overrides, and incident response remains necessary even when average performance rises. The forecast would change most if rigorous real-world studies showed reliable benefits across diverse settings with low verification burden and effective safeguards that small practices can afford.

What should be measured before the next election?

A public scorecard should follow patient-relevant outcomes, not just the count of tools purchased. For an access application, measure referral completion and time to care. For documentation, measure total clinician time including correction. For clinical decision support, evaluate outcomes appropriate to the function and examine performance across relevant groups. For utilization management, track decision timeliness, reasons for denial, appeal outcomes, and whether required clinical review is substantive.

The legal watchlist should distinguish agency guidance from binding rules, filed litigation from court orders, and enacted bills from proposals. Publication of a procurement policy is a milestone; demonstrated compliance is another. Institutions should retain enough information about product versions and deployment changes to explain whether a later result belongs to the system originally evaluated.

For Fresno, a practical regional initiative would connect county priorities, local providers, academic expertise, and patient representatives around a small number of measurable problems. The purpose would be to compare tools against current practice and against simpler alternatives. Sometimes better scheduling, clearer referral criteria, or additional staff will be the more effective intervention. Sometimes software will help deliver those improvements at scale.

A credible public bargain

The next administration and Congress will face pressure to promise both rapid innovation and stronger protection. Those objectives can coexist if policy makes responsibility visible. Patients need a route to question consequential decisions. Clinicians need real authority and usable information. Institutions need evidence they can act on. Developers need clear expectations and opportunities to demonstrate value.

The central forecast is therefore institutional: clinical AI will increasingly be judged as part of a system of care, with attention to the people, contracts, incentives, and public authorities surrounding it. California’s new laws make that transition harder to ignore. Whether it improves life in Fresno will depend on what happens after the legislation is signed—inside the referral queue, the insurance review, and the examination room.

Related reporting

Continue with The Medi-Cal Test: Whether Coverage Survives the Journey from Washington to a Fresno Exam Room on Medi-Cal coverage and capacity, The Price of an Appointment: What the New Congress and California Can Actually Change About Healthcare Costs on healthcare prices and primary care, Who Gets the Last Word? AI, Work, and Education After the Election on AI at work and school, and Prevention Is a Governing System: Mental Health, Nutrition and the Places That Keep Communities Well on public health infrastructure.

Source notes

Numbered notes support the sentence that carries the number. A source supports the claim it is cited for, with its own date and scope; it is not an endorsement by the issuing institution.

1. California Department of Insurance, SB 1120 guidance, May 5, 2025 ↩↩

2. California AB 1979, chapter 854 of 2026, chaptered text reproduced by LegiScan ↩↩↩

3. California SB 503, chaptered text, 2026 ↩↩↩

4. FDA, Clinical Decision Support Software, final guidance, January 29, 2026 ↩↩↩

5. Fresno County, Community Health Assessment and Community Health Improvement Plan ↩↩↩

6. Goh and colleagues, Large Language Model Influence on Diagnostic Reasoning, randomized clinical trial, 2024 ↩↩

7. CMS, program history ↩

8. NIST, Artificial Intelligence Risk Management Framework 1.0, January 26, 2023 ↩

9. California Secretary of State, bill chapters and effective-date rules ↩

10. Governor of California, September 30, 2026 signing announcement ↩