III.B. Algorithm Design

Our machine learning estimator of risk |$\widehat{m}(\cdot )$| is an ensemble model that combines gradient boosted trees and LASSO. It takes as its input vector X _ij 16,405 characteristics of patient i , observable at the start of visit j . ¹⁹ This includes patient demographics; diagnoses, procedures, laboratory results, and quantitative vital signs, measured over the two years prior to the visit; and the symptom recorded at the ED triage desk at the start of the visit. We train estimator |$\widehat{m}(X_{ij})$| to predict the yield of testing Y _ij among the tested, as a close proxy for risk of blockage at the time of an ED visit. ²⁰ To leverage risk information contained in the much larger set of untested patients, we also use predictions on adverse events A _ij = 1 among untested patients as inputs to the model predicting Y _ij . Training happens in a random 75% sample of patients, and all results below are shown in the remaining 25% hold-out set, except where noted. We split our data set at the patient, not the observation, level, so that all visits from a given patient are assigned to either the training or hold-out set. More details can be found in Online Appendix A.4 . Although we cannot share patient-level information to protect privacy, our code repository is publicly available on GitLab. ²¹

We emphasize that Lemma 1 is valid even if the algorithm is inefficient (or even inconsistent) since it applies to any subset, however identified. Inefficient algorithms may fail to find under- or overtested subsets if they do exist. But if they find one that satisfies the inequalities, then it will be an inefficiency, irrespective of the algorithm’s accuracy. It should be added that even a perfect algorithm where m ( X ) = E [ Y | X ] may fail to find all inefficiencies because it does not have access to Z and so may (for example) miss physician errors involving Z .

IV. Results

IV.A. Overtesting

Figure I , Panel A shows how well our risk model predicts the yield of testing. In the hold-out set, we sort tested patients into decile bins based on predicted risk. For each bin ( x -axis), we calculate the yield of testing ( y -axis). Comfortingly, realized yield rises with predicted yield. The algorithm also produces a wide dispersion in realized yields—from 0.01 in the lowest decile to 0.55 in the highest decile.

Figure I

Open in new tab Download slide

Yield and Cost-Effectiveness of Testing in Tested Patients

Realized yield of testing (Panel A) and cost-effectiveness (Panel B) of tests ( y -axis; sample mean shown with an arrow) in the tested, by decile bins of predicted risk ( x -axis). The cost-effectiveness line shows our preferred specification, and the shaded interval shows sensitivity to a range of estimated treatment effects from the literature. For comparison, we include cost-effectiveness estimates of several other tests and treatments.

Figure I , Panel B converts these yields into cost-effectiveness. As in the top panel, patients are sorted by predicted risk, but this time into quintile bins ( x -axis). ²² The y -axis now shows the implied cost-effectiveness of testing patients in a bin, in units of thousands of dollars per life-year. The y -axis shows a commonly used threshold for judging cost-effectiveness, |${\$}$| 150,000, as well as the cost-effectiveness of selected other procedures for comparison. This illustrates a great deal of inefficient testing. The bottom bin of tests is extremely cost-ineffective: |${\$}$| 1,352,466 per life-year. For comparison, biologics for rare diseases (some of the least cost-effective technologies that health systems sometimes pay for) are typically estimated at around |${\$}$| 300,000 per quality adjusted life-year. ²³ Even the second-lowest bin is very cost-ineffective at |${\$}$| 318,603 dollars per life-year.

With these data, we can calculate a precise policy counterfactual as described in Lemma 1 : dropping individual tests whose cost-effectiveness predictably falls below a threshold. For example, at a |${\$}$| 150,000 life-year valuation, we would drop 62.4% of the lowest-value tests, with a combined cost-effectiveness of |${\$}$| 265,114 per life-year. ²⁴ These results only deal with one kind of counterfactual: eliminating the particular tests physicians decided to do (i.e., stress tests or catheterizations) on patients in a given risk bin. Since we have two types of tests, Online Appendix A.3 also explores other counterfactuals. A notable finding is that stress testing (as opposed to catheterization) is so low-value that eliminating it altogether would improve welfare, as has been previously suggested ( Prasad, Cheung, and Cifu 2012 ). Taken together, the results in Figure I and these policy counterfactuals suggest a great deal of overtesting.

IV.B. Undertesting

At the same time, testing in the high-risk bins appears very cost effective. Table III , column (2) shows that in the highest-risk quintile bins, tests cost only |${\$}$| 46,017 per life-year, comparable with cost-effective interventions like dialysis. In column (3), we show testing rate by predicted risk for all patients (for comparability, these bins are formed using the same bin cutoffs used in the tested set, so they are not equally sized). We see that physicians do test higher-risk patients more. But strikingly, many high-risk patients go untested—only 38.3% in the top bin are actually tested.

Of course, this only tells us that the physician and the model disagree, not who is right. ²⁵ The physician has access to a host of information unavailable to the model: how the patient looks, what they say, or crucial data in the ED such as X-rays or ECGs. These data elements are likely to be predictive of yield; if they are also predictive of testing, this private information will create selection bias: untested patients will have far lower yield than predicted based on observables.

Because we lack test results on the untested, we have no way to quantify the magnitude of the problem. But a simple calculation suggests a large bias. The hold-out set has 266 positive tests. Taking model predictions at face value would imply 10 times as many positives (2,738) were we to test the predicted high-risk untested—implausibly large. To show the role of private information more directly, Online Appendix A.5 incorporates data from ECGs, observed by the physician but not routinely observable in health data sets, into risk predictions. ²⁶ For patients with ECG data available, we show that several ECG features (e.g., ST-elevation, ST-depression) predict both the physician’s test decision and the yield of testing, conditional on |$\hat{m}(X)$|⁠ : physicians are using these data effectively. We then directly incorporate the ECG waveform into new risk predictions, via a deep learning model. This decreases model-predicted risk for 97.5% of patients, and 100% of the highest-risk untested. So the model without the ECG was significantly overestimating the risk of the untested patients. Of course, the ECG is just one of many critical variables we do not (and cannot) observe.

Following Lemma 1 , we look for evidence of undertesting in the form of adverse events resulting from untreated blockages, in the 30 days after visits. Among all eligible untested patients, the rate of adverse events is 1.1%, well below the 2% clinical threshold, implying (reassuringly) that testing all untested patients does not make sense. ²⁷ Figure II shows these adverse-event rates ( y -axis) by decile bins of predicted risk. Again, for comparability we use bin cutoffs defined in the tested, meaning bins are of unequal sizes in the untested: in particular, because the untested are lower risk than the tested, bin size decreases in risk. Panel A shows that patients in high-risk bins have very high 30-day adverse-event rates. For example, the highest-risk bin contains 0.15% of the untested, 15.6% of which go on to have an adverse event. The second-highest-risk bin contains 0.75% of the untested and has an adverse event rate of 6.81%; together the top two bins have an adverse event rate of 8.26%. In fact, the crossover point where the adverse event rate becomes statistically indistinguishable from the 2% threshold is the sixth risk bin, which means that the top four bins—which make up 6.9% of the untested—all have high enough adverse event rates to merit consideration for testing under current guidelines.

Figure II

Open in new tab Download slide

Adverse Events in Untested Patients (30 Days after Visits)

Thirty-day adverse event rates among untested patients ( y -axis), by decile bins of predicted risk ( x -axis). Risk bin cutoffs are defined in the tested population, so bins here are not equally sized: the percent in each bin is shown above the x -axis. Panel A shows the total adverse-event rate (the top of the highest 95% confidence interval is truncated). The horizontal line shows the 2% threshold above which testing is recommended by clinical guidelines; the highest-risk 14% (top six bins) have a rate significantly above 2%. The bottom panels show two subset categories of adverse events that make up the total: (B) diagnosed adverse events (heart damage, confirmed with laboratory biomarkers; and cardiac arrest), (C) death (via linkage to Social Security data); bins here are quartiles of predicted risk (because outcomes are less frequent).

These adverse events are not simply billing codes, which might exaggerate the incidence of actual health problems, due to incentives to overtest or treat. Panel B shows the subset of adverse events related to diagnosed blockage, all confirmed with biomarker evidence of damage to the heart muscle (positive troponin laboratory results), as well as dangerous arrythmias (ventricular fibrillation and tachycardia, or procedure codes for defibrillation or CPR). In the highest-risk bin, 4.9% have one of these events. Panel C shows 30-day mortality. The highest-risk bin experiences death at a rate of 3.3%, comprising nearly half (45%) of all adverse events in this bin. These data alone suggest a great deal of undertesting. However, there is a potential confound, which we address next.

IV.C. Accounting for Differences in Treatment Benefits

These high adverse event rates establish that predicted high-risk patients who go untested are indeed high risk. But it does not establish that failing to test them was a mistake. Adverse events rule out private information by physicians about risk, but not private information about the suitability of treatment. It is possible that physicians recognized these patients as being high risk, but also recognized them as having lower return to treatment and chose not to test them for that reason. In particular, we may have mismeasured K _ij . In excluding patients K _ij = 0 from our sample (by excluding those with prior ill health and by excluding untested patients in whom the physician appears to suspect heart problems), our measure K _ij may have failed to capture other elements of K that the physician observes. One fact provides prima facie evidence that these unobservables are not large: the average age of the untested we flag for testing is 58.5 (close to the mean age of the tested, 57.8), whereas the average age of those with observed contraindications is 68.5. At least on this crucial observable, the high-risk untested look more like the tested than the too frail to test.

To address this problem more thoroughly, we use a clinical fact. When physicians suspect a blockage, even if the patient is ineligible for testing or treatment, there are still important actions they can and must take. At a minimum, everyone the physician suspects of a blockage will be given an ECG—a low-cost, noninvasive test. Even for treatment-ineligible patients, the ECG guides medications (e.g., blood thinners) and decisions about intensity of monitoring (e.g., whether to admit to the ICU). Similarly, the troponin blood test will also be checked, as it provides critical information on the nature and extent of any blockage. So if we remove patients with an ECG or troponin from our calculations, we will have removed all patients in whom physicians had even the slightest suspicion of a heart problem, leaving us with a pool of unsuspected patients. ²⁸ Within the remaining unsuspected pool, we recalculate the adverse event rate. If the high adverse event rates in the whole population are due to physicians knowingly leaving some high-risk patients untested, because they are unsuitable for treatment, then this unsuspected pool should have a very low adverse event rate, and specifically the rates should be below the clinical threshold for testing.

The top two panels of Figure III first show the fraction of all patients who are both untested, and did not receive an ECG (Panel A) or troponin (Panel B), by quartile bin of predicted risk. As expected, higher-risk patients are on average perceived as such by physicians: they are less likely to be untested and lack one of these tests. Though decreasing, the fractions nonetheless remain substantial in the highest-risk bin: 19.1% are untested and lack an ECG (versus 77.7% in the lowest-risk bin), and 41.2% are untested and lack a troponin result (versus 93.3% in the lowest-risk bin). The bottom two panels show the adverse event rates in only these untested patients without an ECG or without troponin. For the highest-risk untested patients without such suspicion for heart attack, adverse-event rates remain high: 4.3% in those without an ECG, and 6.6% in those without a troponin. These rates are 3.2 percentage points (std. err. 1.3) and 1.2 percentage points (std. err. 1.1) lower than the 7.5% rate in the full population above, respectively; but they still significantly exceed the clinical threshold for testing of 2%. ²⁹ Together, these results suggest that physicians do have private information both about the risk of blockage and about suitability for treatment—but that even after accounting for them, there is still substantial undertesting.

Figure III

Open in new tab Download slide

Adverse Events in Untested and Unsuspected Patients

Top panels: fraction of patients in whom physicians do not appear to suspect blockage. Panel A shows the fraction untested and lacking an electrocardiogram (ECG); Panel B shows the fraction untested and lacking a troponin laboratory test. Both ECG and troponin are low-cost tests used to screen for blockage; they are done even in patients who may be ineligible for invasive treatment. Fractions are shown by quartile risk bins, with bin cutoffs defined in the tested population (so bins here are not equally sized). Bottom panels: rate of 30-day adverse events (diagnosed events and death) after visits ( y -axis), by bin of predicted risk ( x -axis), among untested patients lacking (C) an ECG, and (D) a troponin. The horizontal line shows the clinical threshold above which testing is recommended by clinical guidelines.

IV.D. Natural Experiment

Although these data provide clear evidence of undertesting, this evidence is indirect, based on clinical thresholds. It would be reassuring to have more direct evidence that testing these untested high-risk patients would affect their health. Ideally, we would measure the effect of testing some high-risk patients at random, and see if in fact mortality and long-term adverse event rates decrease significantly. While such an experiment is beyond the scope of this article, we can exploit natural variation in our data that might serve as a (limited) proxy for it. ³⁰

When a patient arrives at the ED, they are seen by a team of providers, largely nurses, at the triage desk. As Chan and Gruber (2020) note, the triage process can influence many downstream decisions by physicians, including testing. For example, a nurse can notice that a patient with chest pain is sweaty or not; he can ascribe it to the hot and humid weather or not; and he can share his impressions with the physician when he brings the patient back into the room or not. As a result, we hypothesized that the testing rate, while ultimately determined by the physician, could be affected by the particular make-up of the team working the triage desk. Because who is present varies over time, this creates a natural experiment based on the exact time a patient showed up. Because shifts are not perfectly synchronized with the calendar, we can also control for day of week and hour of day.

Our data do not track the exact identity of the triage team, but we know the times at which shifts begin and end. This lets us calculate the average testing rate of all other patients seen on a shift, |$\bar{T}_{-j}$|⁠ , to instrument for whether patient-visit j is tested. For this to be a valid instrument, we assume that (i) the triage shift affects long-term health outcomes only through testing, and (ii) patients are balanced on unobservables across shifts; we discuss both assumptions below. We perform this analysis on a slightly different sample than used so far. To maximize power, we use the full data set, not just the hold-out. To avoid overfitting, we use fivefold cross-validation to predict risk. In addition, to address nonindependence of health outcomes across visits, we restrict the sample to each patient’s first visit. ³¹

Overall, there is reasonable variation in likelihood of testing across shifts: for example, a patient in the highest-risk bin arriving on a Monday evening is 18% more likely to be tested by the highest- (19.9%) versus lowest-decile (16.8%) shifts. Regressing a visit’s test ( T _j ) on the leave-one-out shift testing rate ( ⁠|$\bar{T}_{-j}$|⁠ ), controlling for time fixed effects (year, week of year, day of week, and hour of day) and patient risk, we find that a one standard deviation increase in shift testing rate (2.3 percentage points) increases individual testing probability by 0.19 percentage points (std. err. 0.06), or 6.7% of the base test rate (see Online Appendix Table A.12 ). ³²

Figure IV shows how patient observables compare across shifts. The top panel shows the results of regressing a pretriage variable X _j on the shift testing rate. We do find statistically significant differences in predicted risk across triage testing rates ( p = .051), but they are very small in magnitude: a 1 standard deviation increase in |$\bar{T}_{-j}$| implies a 0.007 standard deviation difference in predicted risk. But reassuringly, we find no statistically significant difference when we test for differences in predicted risk nonlinearly (by risk bin), nor in age, sex, self-reported race, income, or risk factors for heart disease. Together, these results suggest that observables are (largely) balanced across shifts. In the bottom panel, we plot for each shift the average testing rate for all patients who arrive in that shift (in percentile terms, x -axis) and the average predicted risk of those patients ( y -axis). We see that at every level of testing rate, there is large variability in predicted risk.

Figure IV

Open in new tab Download slide

Balance and Risk Variation across Triage Shifts

Panel A shows balance checks in a quasi-experiment, in which patients arriving during different triage shifts are tested at higher or lower rates. Each point shows the coefficient and confidence interval on the leave-one-out shift testing rate ( ⁠|$\bar{T}_{-j}$|⁠ ), from a regression of a given pretriage variable on |$\bar{T}_{-j}$|⁠ . Panel B plots, for each shift, the average testing rate for all patients who arrive in that shift (in percentile terms, x -axis) and the average predicted risk of those patients ( y -axis). Each point represents one of 3,951 shifts in our data set, and the density plot on the right shows the overall distribution of mean risk. *Age is divided by 100 for scale.

In Online Appendix Table A.12 , as another test for balance, we regress test T _j on predicted risk and its interaction with |$\bar{T}_{-j}$|⁠ . If patients in high-testing shifts are riskier on unobservables, they should have higher yield than expected based on risk, leading the interaction term to be positive. In fact, there is no significant interaction. While estimates are imprecise, they do argue against large imbalance on unobservables.

As before, the average effect may conceal a great deal of heterogeneity: undertesting is not universal but only in high-risk patients. So we reestimate equation (1) , but include an interaction term |$\bar{T}_{-j} \times \widehat{m}(X_{j})$|⁠ , which allows the effect of testing to vary by predicted risk. Table IV , Panel B shows this interaction term to be large, negative, and significant, indicating lower rates of diagnosed events and death in higher-risk patients. To scale this coefficient, the implied reduction in one-year mortality for the highest-risk quintile is 2.6 percentage points (34%) if they arrive on the highest- versus lowest-testing shifts. This confirms that physician private information about treatment heterogeneity cannot account for our findings: increased testing improves health in high-risk patients. It also provides some reassurance regarding the exclusion restriction in our experiment: if triage affected long-term outcomes in ways unrelated to testing for blockage, we would expect to see broader effects, not just among the predicted high-risk for blockage. We emphasize that this does not imply that all high-risk untested patients would benefit from testing: we are constrained by the extent of variation in testing rates in our quasi-experiment and can say nothing about patients who are never tested (i.e., even in the highest-testing shifts).

We use these estimates to simulate counterfactuals that bound the extent of undertesting. We first estimate a random effects model of a shift’s testing rates and group shifts into quartiles based on its random effect. ³⁴ Suppose we know a predicted risk bin has positive benefits from testing. Our counterfactual assumes all such patients are assigned to the highest-testing shifts: so the difference between a patient’s actual shift testing rate and highest-quartile shift test rate is counted as undertesting. The key assumption is which risk bins have positive benefits from testing.

We take two approaches. First and most conservative, we assume only those with significant one-year mortality reductions from testing qualify, which based on Table IV includes only the top risk quintile. (Recall that bins are defined using bins in the tested, so the top quintile is far less than 20% of the whole population.) Reassigning all patients in this bin from their actual shift (mean 18.1% test rate) to the highest-quartile shift (32.3% test rate), generates additional tests equal to 0.48% of all untested patients or 15.6% of the tested set. A second approach allows for other testing benefits beyond decreasing one-year mortality, for example, reductions in immediate heart attack (size and extent), as well as longer-term outcomes. To simulate this, for each risk bin, we take the cost-effectiveness estimates from the tested and (naively) apply them to the untested. By testing patients who appear to be cost-effective based on risk, we would add new tests equal to 3.0% of the set of all untested, and 99.5% of the current tested set. ³⁵

Taken together, the evidence tells us three facts about high-risk untested patients, all suggesting they ought to have been tested. First, they go on to have high adverse-event rates of the kind that suggest undiagnosed blockage. Second, physicians do not appear to have recognized their risk: many were not screened with simple tests given to everyone suspected of any heart problems (ECG or troponin), but nonetheless had high adverse event rates. Finally, plausibly exogenous increases in testing improve their health, but not the health of lower-risk patients. Each finding has its limitations, but together, they make the case that testing high-risk untested patients would increase welfare as strongly as possible without a randomized trial.

IV.E. Nationally Representative Data

These results come from a single hospital. To check their generality, we replicate them in a nationally representative 20% sample of Medicare fee-for-service patients, from January 2009 through June 2013. These data are limited in several important ways. Because they are based on insurance claims, not EHR data, they contain very limited patient information. For example, we do not have ECGs, lab values, or other biomarkers, nor do we have arrival time and shift timing data that would let us re-create our natural experiment. These caveats aside, these data do let us replicate our estimates of over- and undertesting from Sections IV.A and IV.B . Applying similar exclusions to those used in the single-hospital data, we arrive at a final sample of 4,425,247 Medicare visits by 1,602,501 patients, of whom 4.4% were tested. Of the tested, 12.4% received treatments. Of the untested, 5.3% had 30-day adverse events. This higher rate reflects the older and sicker Medicare population, but also our inability to confirm diagnosis codes with biomarker evidence of heart attack as above.

Online Appendix Figure A.7 shows that yield of testing and cost-effectiveness both increase in predicted risk (as in Figure I ), with many tests being predictably cost-ineffective. We also find many high-risk untested patients with adverse-event rates above clinical thresholds. Online Appendix Figure A.8 shows that 3.8% of the highest-risk patients are diagnosed with an adverse event, and an additional 1.5% die (as in Figure II ). In summary, we find both overtesting (52.6% of all tests) and undertesting (at least 17.9% of the tested). ³⁶

V. Why Do Physicians Make Testing Errors?

We have shown that physicians mispredict: they test predictably low-risk patients and fail to test predictably high-risk patients. In this section, we try to better understand the nature of physician misprediction. To do so, we examine how physician testing decisions deviate from predicted risk. Our approach builds on a long tradition of research comparing clinical judgment to statistical models as a way to gain insights into decision-making, often among physicians ( Dawes, Faust, and Meehl 1989 ; Elstein 1999 ; Redelmeier et al. 2001 ; Ægisdóttir et al. 2006 ), as well as the clinical literature on diagnostic error ( Croskerry 2002 ; Graber, Franklin, and Gordon 2005 ; IOM 2015 ). We view this as exploratory: a way to shed light on potential psychology at work, rather than to structurally estimate a specific model of physician decisions.

V.A. Boundedness in Physician Judgments

One reason physicians may make errors is that the optimal risk model is quite complex: our own machine learning model uses 16,405 variables. Bounded rationality may lead them to use a simpler approximation. Such simplification is analogous to regularization in machine learning ( Camerer 2019 ). To avoid overfitting, algorithms do not pick the model that fits best in sample. Instead they estimate a best-fit model for each level of complexity, then choose a complexity level by asking which of these best-fit models produces best out-of-sample fit. To study physicians, we use this same set of best-fit models for each complexity. But we ask which model complexity best predicts physician choices, not out-of-sample risk. If physicians are boundedly rational, the model that best predicts their choices should be simpler than the one that best predicts actual risk, measured by yield of testing.

We implement this procedure using the LASSO model of risk, one component of our full ensemble model, because it has a straightforward measure of complexity: the number of nonzero coefficients included in its linear model. ³⁷ For k ∈ [0, 1,500] we train and retain the set of best-fit LASSO models that has exactly k nonzero coefficients. ³⁸ In our hold-out set, we correlate each model with test outcomes and testing decisions. Two caveats are worth noting. First, we do not assume anything about the model selection properties of LASSO: the particular variables the LASSO chooses is somewhat arbitrary in the setting of correlated, noisy input variables. We are interested only in the complexity of these models, which is likely a more stable quantity ( Mullainathan and Spiess 2017 ). Second, we can only focus on the variables in our data: so we only test hypotheses related to boundedness on observables, not on the variables physicians may use that are unobservable to us.

Figure V visually displays the results of this exercise. On the x -axis is k , the measure of complexity. On the y -axis is R ² , a measure of goodness of fit (though our results are not specific to this setup: Online Appendix Figure A.12 shows similar results with AUC instead of R ² , trees instead of LASSO, and the Medicare population). The gray line shows, at each level of complexity, how well a model predicts out-of-sample risk: R ² increases at first, then decreases as additional variables lead to overfitting. The yellow line shows how well the same model predicts physician testing decisions. Here we see in part a similar pattern: R ² increases with complexity, then decreases. Importantly, the two curves hit their peaks at very different levels. For physicians, the empirical optimum is at 49 variables, and for risk it is at 224 variables. The model that best predicts actual risk is much more complex than the one that best predicts test decisions.

Figure V

Open in new tab Download slide

Explanatory Power of Simple versus Complex Models of Risk

Using a LASSO model of predicted risk (part of our full ensemble risk model), we preserve all risk models along the regularization path for k ∈ [0, 1500]: the best-fit linear model that uses at most k nonzero coefficients. The x -axis shows k , the number of variables retained as the regularization penalty is decreased, moving from left to right. The y -axis shows the explanatory power of these risk models of varying complexity for physician testing decisions (dark gray line), and patient risk (yield of testing: yellow line), measured by R ² . The 95% confidence intervals are the shaded intervals, calculated by bootstrapping. The two vertical lines show the complexity of the model that explains the most variance in physician decisions (left, at k = 49) and risk (right, at k = 224).

These results provide suggestive evidence that physicians are boundedly attentive: they only pay attention to some variables. But how accurately do they weigh the variables they attend to? Figure VI shows, for the 49 variables in |$\widehat{m}_{\mathsf {simple}}(X_{ij})$|⁠ , their correlation with test outcome ( x -axis) and test decision ( y -axis). ⁴⁰ We see a tight, strongly positive relationship ( R ² = 0.433). While far from proof of rationality, this does suggest that physicians (mostly) correctly weight the variables they do use.

Figure VI

Open in new tab Download slide

Simple Risk Variables: Correlation with Testing and Predicted Risk

For the simple risk model (with complexity k = 49) that best predicts physicians’ testing decisions, we show univariate correlations of each included variable with the physician’s testing decision ( y -axis) and patient risk ( x -axis). Each point is one of the 49 included variables, with separate shapes denoting different categories of inputs. Some outlier points of interest are labeled.

To assess how important boundedness is in explaining under- and overtesting, we look at how much riskier (or less risky) a patient appears if only simple risk is accounted for. We measure this with |$\widehat{m}_{\mathsf {simple}}(X_{ij}) - \widehat{m}(X_{ij})$| and inspect its distribution for both low-risk tested patients (the overtested) and high-risk untested patients (the undertested). As shown in Online Appendix Figure A.13 , a full 35.5% of the overtested come from the top quintile of |$\widehat{m}_{\mathsf {simple}}(X_{ij}) - \widehat{m}(X_{ij})$|⁠ , meaning their simple risk is much larger than their actual risk (compared with 14.5% in the lowest quintile). Likewise, among the undertested, 74.2% come from the bottom quintile, meaning their simple risk is much smaller than their actual risk (compared with 7.4% in the top quintile). Boundedness thus appears to be quantitatively important as well for misprediction. Physicians identify a handful of good risk predictors that they use, if not perfectly, at least modestly well; at the same time, they neglect many other variables which, while individually small, together provide much explanatory power.

Our evidence on boundedness deviates from the traditional perspective of Dawes, Faust, and Meehl (1989) , who suggest that people use too complex a model: a statistical model does better by being simpler. In contrast, we find physicians use too simple a model: a statistical model does better by being more complex. The difference may arise because modern statistical tools can better fit complex natural phenomena, echoing recent findings that sparse models, despite their appeal (to humans), fit economic phenomena poorly ( Gabaix 2014 ; Giannone, Lenza, and Primiceri 2021 ). In both cases, reality is complicated, while human judgments are simple.

V.B. Biases in Physician Judgments

Figure VI , while largely consistent with bounded rationality, also hints at another phenomenon: physicians might over- or underweight specific variables. In particular, a suggestive example is “Reason for visit: chest pain,” a clear outlier. A complaint of chest pain does correlate with risk, but it correlates even more with testing. This indicates that those with chest pain may be tested at rates above and beyond what is justified by their (heightened) risk. ⁴¹

Chest pain has two features that make it particularly interesting from a behavioral point of view, suggesting two broader behavioral hypotheses for why an input might be overweighted. First, it is highly salient ( Tversky and Kahneman 1974 ; Bordalo, Gennaioli, and Shleifer 2012 ). Second, it is highly representative of blockage: it is a (perhaps the) stereotypical symptom in textbooks and in public understanding ( Bordalo et al. 2016 ). This motivates our exploration of bias: we ask whether variables that are either salient or representative are generally overweighted.

We study these hypotheses in turn, using a similar empirical approach. To assess whether physicians are biased in their use of some subset of variables |$\mathcal {W}$|⁠ , we create a new risk predictor that uses only those variables in |$\mathcal {W}$|⁠ . Except for the restriction on input variables, this estimator, |$\widehat{m}_{\mathcal {W}}$|⁠ , is built in the training set exactly the same as the original risk predictor, and for simplicity of notation takes the same input X _ij but ignores the variables not in |$\mathcal {W}$|⁠ . In the hold-out set, we first regress yield on full risk (our usual risk predictor |$\widehat{m}(X_{ij})$|⁠ ) as well as this limited risk model |$\widehat{m}_{\mathcal {W}}(X_{ij})$|⁠ , analogous to equation (2) . ⁴² We do this to verify that, as expected, conditional on full risk, |$\widehat{m}_{\mathcal {W}}$| does not provide additional information. Then, as our test of whether |$\mathcal {W}$| is misused, we regress the test decision T _ij on full risk |$\widehat{m}(X_{ij})$| and |$\widehat{m}_{\mathcal {W}}(X_{ij})$|⁠ . If physicians overweight the variables in |$\mathcal {W}$|⁠ , the coefficient on |$\widehat{m}_{\mathcal {W}}(X_{ij})$| should be positive; if they underweight, it should be negative. ⁴³

1. Symptom Salience

Building on the chest pain insight above, we implement this procedure first for symptoms: the most salient and immediate thing the physician sees about a patient, often stressed in medical education and vignettes. Table VI , column (1) shows the results of regressing testing on the full risk predictor; column (2) then adds the new symptom-only risk predictor. ⁴⁴ We see here that the risk from symptoms is additionally predictive of testing, suggesting that symptoms as a category are overweighted. ⁴⁵

We expand this exercise to the entire universe of inputs. We form a set of risk predictors, one for each subset of variables, grouped into the following categories: demographics, prior diagnoses, past procedures done on the patient, and prior labs and vital signs. The categories are formed to reflect coherent types of inputs physicians may treat differently. For example, medical case reports and pedagogy use a standard structure, stressing age, sex, and symptoms (e.g., “A 43-year-old man with chest pain,” as in the NEJM ’s Case Records). So we conjectured that demographics and symptoms would be highly salient and thus overweighted. By contrast, the complex, quantitative time series contained in previous laboratory studies and vital signs are harder to process and likely less salient. Finally, while some prior diagnoses (e.g., diabetes, prior blockage) and procedures (e.g., prior stenting) relevant to blockages may be salient, these categories are far broader, including hundreds of other types of information that we also expect to be less salient.

Column (3) shows how these risk predictors correlate with the testing decision. Even after including risk from all other variable subsets, risk from symptoms stays positive (i.e., overweighted), as is risk from demographic information: a patient in the top quartile of symptom risk is 5.26 percentage points more likely to be tested, relative to other patients, and 0.78 percentage points for demographic risk. ⁴⁶ This is equivalent to a patient moving from the 50th percentile of true (full) risk to the 89th and 62nd percentile, respectively. Prior quantitative information from laboratory studies and vital signs, though, has a negative sign, suggesting that physicians underweight or neglect this information. Finally, diagnoses are slightly overweighted while procedures are slightly underweighted. Taken together, these results are generally supportive of the salience model: risk signals from clearly salient inputs—demographics and symptoms—are attended to more than they should be, while more complex, less salient information—past quantitative vital signs and labs—are neglected.

2. Representativeness

This model has a crisp empirical prediction: at the same predicted risk, patients with more (less) representative symptoms are more (less) likely to be tested. We investigate this by first identifying the set of symptoms that are potentially representative of blockage. To make this list, we identify those tested patients ultimately found to have blockages after testing and look back at their presenting symptom (limiting to 16 symptoms with frequency over 0.5% in this population; see Online Appendix Table A.16 ). For each symptom M , we calculate its representativeness for blockage: |$\frac{Pr(M=1|B=1)}{Pr(M=1|B=0)}$|⁠ . Nine symptoms have a ratio over 1, which we consider representative of blockage. Some are very common in the general population (e.g., chest pain, shortness of breath) and others are quite rare (e.g., presenting to the ER after a referral for a concern of possible blockage or because they were found unresponsive or in cardiac arrest by paramedics). The remaining seven symptoms are more common in the general population than in those with blockage (e.g., dizziness, nausea).

This allows us to build yet another risk predictor, restricting to representative symptoms. Table VI , column (4) shows the results of adding this to the regression we described previously (column (2)) with the predictor formed from all symptoms. With representative symptoms included, the all-symptom-based predictor becomes small and insignificant. The coefficient on the representative symptom-based predictor in column (4) is nearly double the magnitude of the all-symptom-based predictor in column (3). ⁴⁷ This argues that while symptoms as a whole may be salient, representative symptoms drive physicians to test far more: they effectively cue the physician’s mind to consider blockage. This effect is quantitatively large: the 7% in the highest quintile of representative symptom risk are 16.2 percentage points more likely to be tested, corresponding to an increase from the 50th to the 98th percentile of true risk.

Further, as shown in Online Appendix Figure A.14 , patients whose risk comes disproportionately from representative symptoms (i.e., large |$[\widehat{m}_{\mathsf {represent}}(X_{ij}) - \widehat{m}(X_{ij})]$|⁠ ) are overrepresented in testing errors. Those in the top quintile of representativeness risk (relative to true risk) make up 34.3% of the low-risk tested; while the bottom quintile makes up 99.4% of the high-risk untested. ⁴⁸

V.C. Implications for Incentive Policies

The simultaneous presence of over- and underuse suggests that simple views of health care like “less is more” or “more is more” are insufficiently nuanced. Our results thus add to the growing body of work in health economics arguing for richer models of physician behavior ( Kolstad 2013 ; Abaluck et al. 2016 ; Chandra and Staiger 2020 ; Chan, Gentzkow, and Yu 2022 ). Policy makers have long viewed health care through the lens of misaligned incentives that make physicians too eager to test. Implicit in this model is that physicians estimate risk correctly but simply set too low a threshold. This “less is more” model, which suggests that high-testing providers are wasteful relative to low-testing ones, has a clear practical implication that drives much of health policy in the United States and internationally: create incentives to test less, for example, via reimbursement schemes or capacity constraints. Yet our finding of systematic biases by physicians calls this approach into question: if physicians mispredict risk, incentives to cut care may do harm as well as good.

We empirically examine these potentially perverse consequences by asking, when physicians test less, which tests do they cut? The view of traditional models—and the hope of health policy—is that they cut the low-value tests. The top panel of Figure VII shows that this is not the case. Here we graph the probability of testing against predicted risk separately for each of the testing quartiles in our quasi-experiment (using the random effects model described above). Low-testing shifts do cut back on low-value tests: the lowest-risk patients are tested only 0.4% of the time, versus 3.0% on the highest-testing shifts. But they also cut back on high-value tests: the highest-risk patients are tested 5.8% of the time, versus 32.3% on the highest-testing shifts. In absolute terms, high-value tests suffer the biggest decline—26.5% fewer in low- versus high-testing regimes. In relative terms, low- and high-value tests fall by similar amounts: 87% versus 82%, respectively. In other words, less testing means less testing for everyone, regardless of risk. The bottom panel replicates these results in our nationally representative Medicare sample, where we sort hospitals into quintiles based on their testing rate, and again graph testing versus predicted risk for each quintile. We see the same result: hospitals that test more test everyone more. ⁴⁹

Figure VII

Open in new tab Download slide

Variation in Testing Rates by Predicted Risk

Panel A shows variation in testing rates by predicted risk, in our quasi-experiment where patients are tested at higher or lower rates based on the triage staff working when they arrive. Panel B shows variation in testing rate by predicted risk, across all hospitals in the United States. Hospitals are binned into quartiles based on the overall testing rate of the hospital referral region in which they are located, to mirror cross-sectional analyses in the literature.

These data provide a reminder that reducing care leads to cutbacks in what is perceived to be low value. But when there are prediction errors, what is perceived to be low value might in fact be extremely valuable. The problem is analogous to behavioral hazard in patient decision making, where copays lead patients to cut back on both low- and high-value care ( Chandra, Gruber, and McKnight 2010 ; Baicker, Mullainathan, and Schwartzstein 2015 ; Handel and Kolstad 2015 ; Brot-Goldberg et al. 2017 ; Chandra, Flack, and Obermeyer 2021 ). Incentives to reduce care can have perverse consequences throughout the health care system.

V.D. The Role of Physician Experience

If incentives do not reduce inefficiency, what does? A natural candidate is physician experience, which we observe in our data. Though we cannot causally identify the effect of experience, correlations can be suggestive. In particular, we study how the correlation between physician decisions and patient risk varies with physician experience (as measured by years since residency). In Table VI , we regress testing on predicted risk, experience, and an interaction term between experience and patient risk. Column (5) shows that more experienced physicians test less on average: 1.68 percentage points or 0.05% for every year since residency. At the same time, experienced physicians are better able to match testing decisions to risk: with every year of experience, they test the lowest-risk patients 0.04 percentage points (2.81%) less, and the highest-risk 0.58 percentage points (1.06%) more. ⁵⁰ These correlations provide suggestive evidence that physicians may learn over time, becoming more accurate with experience.

The results on experience in this section and the results on high- versus low-testing regimes tell distinct stories. On one hand, experienced physicians both test less and are more accurate. This echoes Chan, Gentzkow, and Yu (2022) , who show a negative relationship between skill and testing levels. On the other hand, in Section V.C , we saw that less testing was uncorrelated with accuracy: testing fell across the risk distribution, including high-risk patients. This suggests that the relationship between testing level and accuracy is complex, and that care is needed to characterize it accurately. Understanding what leads physicians to be more or less accurate—and how that relates to testing level—is an important and open question.

VI. Conclusions

Much of machine learning applied to health care focuses on building tools to aid or substitute for humans: for example, algorithms that can match radiologists’ performance on X-rays. Our work suggests a very different use for machine learning in health care: as a tool to understand humans and the health systems they work in.

This approach allows us to precisely characterize inefficiencies. Current empirical approaches in health policy rely largely on aggregates: for example, do tests on average yield enough positives to justify their costs ( Weinstein et al. 1996 ; Sanders et al. 2016 )? By that metric, testing appears highly efficient, at only |${\$}$| 89,714 per life-year in our data. The granularity of algorithmic predictions, by contrast, reveals both under- and overuse. This reframes the discussion away from how many people get tested—too many, or too few?—to one about who gets tested. In a very conservative simulation of optimal testing, total testing would drop by 47%, but the composition of the tested would change radically: 29% of efficient tests would be new, in patients physicians do not currently test; and tests would go from costing |${\$}$| 89,714 to |${\$}$| 59,390 per life-year. The importance of composition in turn calls into question the central role of incentives in policy. By changing the level of testing alone, they may improve one inefficiency (overuse) while aggravating another (underuse).

Despite the great promise of algorithms for diagnosing and improving human inefficiencies, great care is needed when comparing human decisions and algorithmic predictions. As we saw, when physician and algorithm disagree, we cannot just assume the algorithm is correct: unobserved variables confound algorithmic predictions. This selection bias pervades machine learning applications in health and elsewhere, appearing whenever algorithms are trained on data produced by the humans they seek to influence. ⁵¹ Once acknowledged, we show these problems can be tackled: by developing new labels grounded in domain expertise, and via quasi-experimental methods from the causal inference toolkit. But ignoring this bias risks stacking the deck in favor of algorithms: assuming away physician private information means algorithms can, by construction, never do worse than the human—a misleading comparison.

Finally, our findings suggest a role for algorithmic predictions in interventions to increase efficiency. Most obviously, because they are built on EHR data, our predictions can be delivered to physicians in real time. Rather than replacing their judgment, they can be combined with physician private information. At the payment level, a system of precision pricing could tie incentives and reimbursements for testing to patient-level predicted risk and testing outcomes. Or predictions could be used as an educational tool, during physician training or as continuing medical education. We found accuracy improves with experience, but using algorithms to hasten the learning process would be valuable: human trial and error is a costly way to learn in medicine.

Data Availability

Code replicating the tables and figures in this article can be found in Mullainathan and Obermeyer (2021) in the Harvard Dataverse, https://doi.org/10.7910/DVN/IUMIO6 .

Footnotes

References