Because of limited definitive sources of information about grant-level terminations, EPA awards have a more complex status determination logic than other agencies.
\nDOGE provides data on terminated awards, although we have found that it is incomplete or occasionally incorrect, so for EPA we use it only in combination with other sources as a signal of grant termination.
\nWe use this source primarily for metadata including grantee information, grant assistance listing (CFDA), grant funding opportunity (also called notice of funding announcement or NOFO), and total obligated and total outlaid amounts.\nWe also track changes to project end dates (i.e. project end date moved earlier, a \"cutoff\", or later, a \"reextension\"), and deobligation events (i.e. a negative amount obligated) which we use in combination as signals of disruption.
\nFile C is released roughly once a month (except that Sep/Oct are combined into one release).\nWe use this source primarily for outlay actions which we use to confirm reinstatements.
\nEPA's Next Generation Grants System (NGGS) provides an API to download grant-level information and we snapshot this data regularly.\u00a0\nWe use it as a source of grant metadata such as grant description, grantee information, etc.\nNotably, it is our only source for grant titles.\nUnfortunately, it does not carry a clear signal of grant termination, but we use project end date cutoffs and deobligation events in combination with similar signals from USAspending award data in our status determination.
\nWe have information from a trusted confidential source about particular EPA assistance listings (also called CFDAs) that have been terminated that we use in combination with other signals to determine grant status.
\nThe Our Environment and Public Works Senate Committee released a list of terminated EPA awards in March 2025. This is one of our definitive sources of grant terminations.
\nIn March 2025, EPA terminated eight National Clean Investment Fund (NCIF) and Clean Communities Investment Accelerator (CCIA) awards.\nThe CCIA and NCIF programs appear as though they were fully outlaid on USAspending.\nHowever, the outlaid assets are in accounts frozen by the federal government.\nWe do not know exactly how much has been spent, but it is closer to $0 than to the entire award value, so we have manually adjusted the 'funds spent before disruption' and 'funds remaining before disruption' values for these awards.
\nAt the time of writing, there are two court decisions ordering the reinstatement of EPA awards\u2014Sustainability Institute v. Trump and Green and Healthy Homes Initiative Inc. vs EPA.\nWe use these documents as a definitive source of award reinstatement and will continue to add court documents for similar decisions in the future.
\nEPA awards are marked as terminated if any of the following conditions are met:
\nAwards have been reported as such by PIs or trusted sources
\nOur sources show the awards' assistance listing has been terminated and they have experienced a project end date cutoff or are reported terminated by DOGE, or have a deobligation.
\nAwards appear in any of our sources of known terminations (see Our Environment and Public Works Senate Committee and Greenhouse Gas Reduction Fund above)
\nAwards are reported terminated by DOGE and have a project end date cutoff
\nAwards appear in court documents ordering reinstatement (we assume that if they were ordered reinstated, they must have been terminated for EPA)
\nEPA awards are considered reinstated if they have had their end date re-extended after a date cutoff, appear in court documents ordering reinstatement, or are reported as reinstated by PIs or trusted sources.\nReinstatements are confirmed after they receive an outlay.
\nThe funds obligated and spent at time of earliest disruption are calculated using the current total obligated amount minus obligations since the earliest termination date and the total outlays minus outlays since the earliest termination date, respectively.\nThe outlay and obligation events come from File C, while the current total obligated and outlaid numbers are from the more regularly updating USAspending awards data.\nAs a result, these estimates may fluctuate week-to-week.
","doi":"https://doi.org/10.59350/tq3c6-7x998","guid":"https://doi.org/10.59350/tq3c6-7x998","language":"en","license":"https://creativecommons.org/licenses/by/4.0/legalcode","published_at":1792713600,"rid":"z98pv-wa015","summary":"Because of limited definitive sources of information about grant-level terminations, EPA awards have a more complex status determination logic than other agencies.","tags":["Methods","Environmental Protection Agency"],"title":"EPA Grant Disruption Methodology","updated_at":1792793705,"url":"https://grantwitness.org/epa/updates/methodology","version":"v1"}},{"document":{"authors":[{"affiliation":[{"name":"CSM Advising LLC"}],"contributor_roles":[],"family":"Marcum","given":"Christopher","url":"https://orcid.org/0000-0002-0899-6143"}],"blog":{"authors":null,"community_id":"8bdb1ae7-4621-4fa5-ad1a-3a639417dfd5","created":1768694400,"current_feed_url":null,"description":"Perspectives on science, data, and technology that don't fit anywhere else.","doi":"https://doi.org/10.59350/chrismarcum","favicon":"https://rogue-scholar.org/api/communities/8bdb1ae7-4621-4fa5-ad1a-3a639417dfd5/logo","feed_format":"application/atom+xml","feed_url":"http://chrismarcum.com/marcum-blog/feed.atom","filter":null,"generator":"Jekyll","home_page_url":"https://www.chrismarcum.com/marcum-blog/","issn":null,"language":"eng","license":"https://creativecommons.org/licenses/by/4.0/legalcode","prefix":"10.59350","relative_url":null,"secure":true,"slug":"chrismarcum","status":"active","subfield":"3312","title":"Open Evidence","updated":1792713600,"use_api":null},"blog_name":"Open Evidence","blog_slug":"chrismarcum","content_html":"In July, we showed that anyone could inflate the \"comments received\" number displayed on regulations.gov \u2013 that is, the number of comments submitted during a public comment period. When you attached a file to a comment, the form asked whether the file contained comments from more than one person or entity. Check yes, and you would be asked for the \"Number of persons/entities represented\" by your comment. You could put any number from 1 to 999,999, and whatever number you picked would be added to the docket's \"comments submitted\" count. We tested it on a Bureau of Land Management docket, and the count jumped from 310 to over a million by the next day.
\nAnd this topline \"comments received\" number drives news coverage. Reporters cited 500,000 comments on OMB's proposed revisions to the Uniform\nGuidance (2 CFR 200) as a measure of public interest. The true count was still very high at over 164K, but that's a far cry from the half million reported by the press.
\nLast weekend, we noticed that this had been fixed, and looked back at archived copies of regulations.gov to see when. We found that on September 4, the General Services Administration (GSA), which runs regulations.gov, removed the question from the comment form. A week later, on September 10, GSA relabeled the count and rewrote its description. We don't know for sure whether our post or the press coverage we received on it had anything to do with the decision to fix the count bug. Either way, credit to the civil servants at GSA who made the change happen. That's good governance.
\nRegulations.gov runs on a single JavaScript file that generates the comment form and the count box. Every time GSA updates the site, the file gets a new name. The Internet Archive's Wayback Machine saves copies of those files as it crawls, so we can line up every version from the spring through today and check each one \u2013 which we did.\nEvery prior version we checked until midday on September 4, 2026 includes the \"persons/entities\" question on the comment form. Every version after does not. We were first clued into this because we noticed that, despite a very high rate of comments being submitted from campaigns to the US Census Bureau's proposed rulemaking on residency requirements for the 2030 census \u2013 a controversial rulemaking where we would expect such gaming \u2013 the comments submitted to comments published had a relatively consistent difference over time. We were tracking this because of our work on handsoffourcens.us.
\nWe investigated further to find out if GSA had indeed fixed the bug.
\n| Version | \nSaved by Wayback | \nQuestion on form? | \nCount description | \n
|---|---|---|---|
| e4870fb6 | \nSep 4, 12:15 PM ET | \nYes | \n\"as of 11:59 PM yesterday\" | \n
| c583eabb | \nSep 4, 3:14 PM ET | \nNo | \n\"as of 11:59 PM yesterday\" | \n
| c92c9d40 | \nSep 10, 7:56 PM ET | \nNo | \n\"including unpublished\" | \n
\n\nThe new count description on the dockets is better, too. It now reads:\n\"The number of comment submissions that have been received (including unpublished) to all documents within this docket.\"
\n
The counts for dockets prior to this change are still on this site. For instance, our BLM test docket still shows 1,037,743 submissions received against 4,477 posted, and the OMB docket still shows 496,606 against 164,654. Neither was corrected.
\nWe also wanted to know whether anyone else had used the loophole before we wrote about it. So we pulled every proposed rule on regulations.gov whose comment period closed between January 2025 and early September 2026, took a random sample of 300, and compared the number received to the number posted. Most line up closely, which is what you'd expect: the difference is usually comments an agency is still reviewing or chose not to post. A few don't line up at all.\nThe standout is an FAA airworthiness directive for a single Airbus Canada aircraft model. These are technical maintenance orders that usually draw a handful of comments. This one shows 656,575 received and 4 posted. A Copyright Office rule about paying royalties through Pay.gov shows 91,510 received and none posted. We can't prove those numbers were inflated, but we also can't think of another way they got there.
\nNot every big gap reflects a user entering a large number into \"persons/entities\" field. Sometimes, agencies just don't post every individual comment for a mass-mail campaign, and so the posted count will remain significantly below the received count. The EPA in particular will post one representative letter to stand in for thousands of identical ones. EPA's Waters of the United States rule, which closed October 9, shows 121,798 received and 2,095 posted in the API backend. But we can see what's driving this: the EPA posted 47 \"mass comment campaign\" representative records on that rule, which likely account for much of the gap.
\n| Docket | \nComment period closed | \nReceived | \nPosted | \nDifference | \n
|---|---|---|---|---|
| Closed before the fix | \n\u00a0 | \n\u00a0 | \n\u00a0 | \n\u00a0 | \n
| OMB-2026-0034 | \nJul 13, 2026 | \n496,606 | \n164,654 | \n331,952 | \n
| BLM-2026-0068 | \nAug 14, 2026 | \n1,037,743 | \n4,477 | \n1,033,266 | \n
| FAA-2025-0199-0008 | \nNov 10, 2025 | \n656,575 | \n4 | \n656,571 | \n
| COLC-2024-0004-0001 | \nJan 13, 2025 | \n91,510 | \n0 | \n91,510 | \n
| FWS-R4-ES-2024-0073-0001 | \nJan 24, 2025 | \n70,899 | \n38,045 | \n32,854 | \n
| EPA-HQ-OAR-2024-0419-0001 | \nApr 15, 2025 | \n20,979 | \n194 | \n20,785 | \n
| IRS-2025-0235-0001 | \nOct 22, 2025 | \n16,903 | \n321 | \n16,582 | \n
| DOT-OST-2025-0831 | \nAug 21, 2026 | \n1,357 | \n1,354 | \n3 | \n
| OCC-2026-0331 | \nAug 21, 2026 | \n524 | \n23 | \n501 | \n
| EPA-HQ-OW-2026-1090 | \nMay 26, 2026 | \n47 | \n47 | \n0 | \n
| Comment period open after the fix | \n\u00a0 | \n\u00a0 | \n\u00a0 | \n\u00a0 | \n
| EPA-HQ-OW-2025-0322-15447 | \nOct 9, 2026 | \n121,798 | \n2,095 | \n119,703 | \n
| FS-2025-0001-342438 | \nOct 6, 2026 | \n19,826 | \n19,826 | \n0 | \n
| FAR-2026-0003-0001 | \nOct 19, 2026 | \n95 | \n95 | \n0 | \n
| OPM-2025-0012-0678 | \nSep 29, 2026 | \n81 | \n76 | \n5 | \n
| NRC-2025-1204-0004 | \nOct 13, 2026 | \n52 | \n52 | \n0 | \n
\n\nCounts pulled from the regulations.gov API on October 23, 2026 (October 18 for the OMB, BLM, DOT, OCC, and EPA water docket rows). Rows linked to a docket count the whole docket. Rows linked to a document count that one comment period for a specific document within a docket.
\n
All data come from the public regulations.gov API, the same one the website uses. \"Received\" is the API's received count, and \"posted\" is the number of comments the API returns. The before-fix sample was drawn at random from about 2,600 whose comment periods closed between January 1, 2025 and September 3, 2026. The after-fix dockets represent every proposed rule, 35 in all, whose comment period opened September 5 or later and closed by October 22. The table shows the five largest by number received from each group, plus our original docket use-cases. Only two of the after-fix rules drew more than 100 submissions, so this sample can't tell us much about how counts behave now. We also checked the figures against snapshots captured by Wayback Machine.
\nThe proof of the fix is the document upload form itself: the entities count question is gone.
\nWe're really glad that this is fixed going forward, and we think a few small tweaks would complete the change. First, recalculate the counts on dockets where the \"persons/entities\" number was used, or at least flag them. Second, GSA should publish a short note explaining the change, so the public knows the count means something different now. Third, GSA should update the language about comment counts on its FAQ page, which is now out-of-date. We're following this issue closely, but it still took us over a month to catch this: other people who use this data might miss this important detail unless GSA points this out.
","doi":"https://doi.org/10.59350/nd0y0-agy83","guid":"https://www.chrismarcum.com/marcum-blog/2026/10/23/GSA-Closed-the-Comment-Loophole-on-Regulations-dot-gov","language":"en","license":"https://creativecommons.org/licenses/by/4.0/legalcode","published_at":1792713600,"rid":"b68eb-yej13","summary":"In July, we showed that anyone could inflate the \"comments received\" number displayed on regulations.gov \u2013 that is, the number of comments submitted during a public comment period. When you attached a file to a comment, the form asked whether the file contained comments from more than one person or entity. Check yes, and you would be asked for the \"Number of persons/entities represented\" by your comment.","tags":["General"],"title":"GSA Closed the Comment Count Loophole on Regulations.gov","updated_at":1792790704,"url":"https://www.chrismarcum.com/marcum-blog/2026/10/23/GSA-Closed-the-Comment-Loophole-on-Regulations-dot-gov.html","version":"v1"}},{"document":{"authors":[{"affiliation":[{"id":"https://ror.org/05dq2gs74","name":"Vanderbilt University Medical Center"}],"contributor_roles":[],"family":"Harrell","given":"Frank","url":"https://orcid.org/0000-0002-8271-5493"}],"blog":{"authors":[{"name":"Frank Harrell","url":"https://orcid.org/0000-0002-8271-5493"}],"community_id":"e74e284b-0d7e-4e86-bfae-58b71f06a7cf","created":1792713600,"current_feed_url":null,"description":"Parametric Models vs. Empirical Distributions and Ordinal Regression.","doi":"https://doi.org/10.59350/fharell","favicon":"https://rogue-scholar.org/api/communities/e74e284b-0d7e-4e86-bfae-58b71f06a7cf/logo","feed_format":"application/rss+xml","feed_url":"https://www.fharrell.com/index.xml","filter":null,"generator":"Quarto","home_page_url":"https://www.fharrell.com/","issn":null,"language":"eng","license":"https://creativecommons.org/licenses/by/4.0/legalcode","prefix":"10.59350","relative_url":null,"secure":null,"slug":"fharrell","status":"active","subfield":"2613","title":"Statistical Thinking","updated":1786251600,"use_api":null},"blog_name":"Statistical Thinking","blog_slug":"fharrell","content_html":"Harrell FE (2017). Introduction. Statistical Thinking. https://doi.org/10.59350/hepkr-mgd06
\nStatistics is a field that is a science unto itself and that benefits all other fields and everyday life. What is unique about statistics is its proven tools for decision making in the face of uncertainty, understanding sources of variation and bias, and most importantly, statistical thinking. Statistical thinking is a different way of thinking that is part detective, skeptical, and involves alternate takes on a problem. An excellent example of statistical thinking is statistician Abraham Wald's analysis of British bombers surviving to return to their base in World War II: his conclusion was to reinforce bombers in areas in which no damage was observed. For other great examples watch my colleague Chris Fonnesbeck's Statistical Thinking for Data Science.
Some of my personal philosophy of statistics can be summed up in the list below:
\nStatistics has multiple challenges today, which I break down into three major sources:
\nI will be discussing several of the issues in future blogs, especially item 1 above and items 2 and 4 below. Complacency in the field of statistics and in statistical education has resulted in
\nI was interviewed by Kevin Gray in July, 2017 where more of my opinions about statistics may be found.
\nSee this article for related thoughts.
\nMy top role model is Sherlock Holmes. See this video for an excellent example.
\n\n@online{harrell2017,\n author = {Harrell, Frank and Harrell, Frank},\n title = {Introduction},\n date = {2017-01-13},\n url = {https://www.fharrell.com/post/introduction/},\n doi = {10.59350/hepkr-mgd06},\n langid = {en}\n}\nHarrell FE (2017). p-values and Type I Errors are Not the Probabilities We Need. Statistical Thinking. https://doi.org/10.59350/f27qd-grr25
\nIn trying to guard against false conclusions, researchers often attempt to minimize the risk of a \"false positive\" conclusion. In the field of assessing the efficacy of medical and behavioral treatments for improving subjects' outcomes, falsely concluding that a treatment is effective when it is not is an important consideration. Nowhere is this more important than in the drug and medical device regulatory environments, because a treatment thought not to work can be given a second chance as better data arrive, but a treatment judged to be effective may be approved for marketing, and if later data show that the treatment was actually not effective (or was only trivially effective) it is difficult to remove the treatment from the market if it is safe. The probability of a treatment not being effective is the probability of \"regulator's regret.\" One must be very clear on what is conditioned upon (assumed) in computing this probability. Does one condition on the true effectiveness or does one condition on the available data? Type I error conditions on the treatment having no effect and does not entertain the possibility that the treatment actually worsens the patients' outcomes. Can one quantify evidence for making a wrong decision if one assumes that all conclusions of non-zero effect are wrong up front because H0 was assumed to be true? Aren't useful error probabilities the ones that are not based on assumptions about what we are assessing but rather just on the data available to us?
Statisticians have convinced regulators that long-run operating characteristics of a testing procedure should rule the day, e.g., if we did 1000 clinical trials where efficacy was always zero, we want no more than 50 of these trials to be judged as \"positive.\" Never mind that this type I error operating characteristic does not refer to making a correct judgment for the clinical trial at hand. Still, there is a belief that type I error is the probability of regulator's regret (a false positive), i.e., that the treatment is not effective when the data indicate it is. In fact, clinical trialists have been sold a bill of goods by statisticians. No probability derived from an assumption that the treatment has zero effect can provide evidence about that effect. Nor does it measure the chance of the error actually in question. All probabilities are conditional on something, and to be useful they must condition on the right thing. This usually means that what is conditioned upon must be knowable.
\nThe probability of regulator's regret is the probability that a treatment doesn't work given the data. So the probability we really seek is the probability that the treatment has no effect or that it has a backwards effect. This is precisely one minus the Bayesian posterior probability of efficacy.
\nIn reality, there is unlikely to exist a treatment that has exactly zero effect. As Tukey argued in 1991, the effects of treatments A and B are always different, to some decimal place. So the null hypothesis is always false and the type I error could be said to be always zero.
\nThe best paper I've read about the many ways in which p-values are misinterpreted is Statistical tests, P values, confidence intervals, and power: a guide to misinterpretations written by a group of renowned statisticians. One of my favorite quotes from this paper is
\nThus to claim that the null P value is the probability that chance alone produced the observed association is completely backwards: The P value is a probability computed assuming chance was operating alone. The absurdity of the common backwards interpretation might be appreciated by pondering how the P value, which is a probability deduced from a set of assumptions (the statistical model), can possibly refer to the probability of those assumptions.
\nIn 2016 the American Statistical Association took a stand against over-reliance on p-values. This would have made a massive impact on all branches of science had it been issued 50 years ago but better late than never.
\nThough believed to be true by many non-statisticians, p-values are not the probability that H0 is true, and to turn them into such probabilities requires Bayes' rule. If you are going to use Bayes' rule you might as well formulate the problem as a full Bayesian model. This has many benefits, not the least of them being that you can select an appropriate prior distribution and you will get exact inference. Attempts by several authors to convert p-values to probabilities of interest (just as sensitivity and specificity are converted to probability of disease once one knows the prevalence of disease) have taken the prior to be discontinuous, putting a high probability on H0 being exactly true. In my view it is much more sensible to believe that there is no discontinuity in the prior at the point represented by H0, encapsulating prior knowledge instead by saying that values near H0 are more likely if no relevant prior information is available.
\nReturning to the non-relevance of type I error as discussed above, and ignoring for the moment that long-run operating characteristics do not directly assist us in making judgments about the current experiment, there is a subtle problem that leads researchers to believe that by controlling type I \"error\" they think they have quantified the probability of misleading evidence. As discussed at length by my colleague Jeffrey Blume, once an experiment is done the probability that positive evidence is misleading is not type I error. And what exactly does \"error\" mean in \"type I error?\" It is the probability of rejecting H0 when H0 is exactly true, just as the p-value is the probability of obtaining data more impressive than that observed given H0 is true. Are these really error probabilities? Perhaps \u2026 if you have been misled earlier into believing that we should base conclusions on how unlikely the observed data would have been observed under H0. Part of the problem is in the loaded word \"reject.\" Rejecting H0 by seeing data that are unlikely if H0 is true is perhaps the real error.
\nThe \"error quantification\" truly needed is the probability that a treatment doesn't work given all the current evidence, which as stated above is simply one minus the Bayesian posterior probability of positive efficacy.
\nType I error control is an indirect way to being careful about claims of effects. It should never have been the preferred method for achieving that goal. Seen another way, we would choose type I error as the quantity to be controlled if we wanted to:
\nThe Bayesian approach focuses on the chances you give effects to be positive and does not have multiplicity issues (potential issues such as examining treatment effects in multiple subgroups are handled by the shrinkage that automatically results when you use the 'right' Bayesian hierarchical model).
\nThe p-value is the chance that someone else would observe data more extreme than mine if the effect is truly zero (if they could exactly replicate my experiment) and not the probability of no (or a negative) effect of treatment given my data.
\nAs discussed in Gamalo-Siebers at al DOI: 10.1002/pst.1807 the type I error is the probability of making an assertion of an effect when no such effect exists. It is not the probability of regret for a decision maker, e.g., it is not the probability of a drug regulator's regret. The probability of regret is the probability that the drug doesn't work or is harmful when the decision maker had decided it was helpful. It is the probability of harm or no benefit when an assertion of benefit is made. This is best thought of as the probability of harm or no benefit given the data which is one minus the probability of efficacy. Prob(assertion|no benefit) is not equal to 1-Prob(benefit|data).
\nType I (\"false positive\") error probability would be a useful concept while a study is being designed. Frequentists speak of type I error control, but after a study is completed, the only way to commit a type I error is to know with certainty that an effect is exactly zero. But then the study would not have been necessary. So type I error remains a long-run operating characteristic for a sequence of hypothetical studies.
\nThinking of p-values that a sequence of hypothetical studies might provide, when the type I error is \u03b1 this means P(p-value < \u03b1 | zero effect) = \u03b1. Neither a single p-value nor \u03b1 is the probability of a decision error. They are \"what if\" probabilities, if the effect is zero. The p-value for a single study is merely the probability that data more extreme than ours would have been observed had the effect been exactly zero and the experiment was capable of being re-run infinitely often. It is nothing more than this. It is not a false positive probability for the experiment at hand. To compute the false positive probability one would need a prior distribution for the effect (and for the p-value to be perfectly accurate which is rare), and one might as well be fully Bayesian and enjoy all the Bayesian benefits.
\nFrank Harrell: You are correct; sorry for the omission of the condition that H0 is true. Thanks for the comments.
\nHenry: When you write the following, \"The p-value is the chance that someone else would observe data more extreme than mine (if they could exactly replicate my experiment) and not the probability of no (or a negative) effect of treatment given my data,\" I am guessing that you left out, \"conditional that the null hypothesis is true.\" While a nitpick, I thought that this is an important and meaningful addendum as its absence changes the meaning to a non-stats reader.
\nGreat fan of your book and your posts on Stackexchange.
\nFH: Thanks for the perceptive comments. I agree in spirit with all of them until you get into multiplicity where the fundamental problems with p-values just makes everything implode. But in general, and in line with your thoughts, I think that it's not the case that p-values are not related to evidence, it's just that they are not calibrated to provide the quantification of evidence we need. So I need to say these points more clearly, e.g., p-values are the correct functions of the data, and except for not being able to deal with prior knowledge, they are measures of evidence but are not calibrated to what is needed for direct action/decision making.
\nMichael Lew: I like very much the update of January 20, but I think the last sentence misses the important nature of the P-value.
\nThe best way to ensure care is taken about claims of effects is to make sure that scientists think scientifically: carefully, critically, and consider more than just the statistical result of any particular experiment or study component.
\nBayesian approaches help, presumably, but my opinion is that scientists would be more directly helped in their thinking if the statistics of evidence were emphasised over the statistics of posterior probabilities or the statistics of errors.
\nFisher's P-values have an evidential meaning that has nothing to do with error probabilities. They may not be as rich a depiction of evidence as a likelihood function, but they communicate evidence nonetheless.
\n**FH*: These are great points. And I should have said \"stand\" instead of \"strong stand\", or \"expressed significant caution\". I linked to another blog instead of the ASA because the other blog linked to ASA and had some other statements I liked. I would like to note that evidence against H0 is not the kind of evidence I seek in science. A recent paper I co-authored delves into the Fisher vs Neyman-Pearson approaches - see this. One reason there are the two schools of thought is that the entire NHST premise is flawed in my humble opinion.
\nMichael Lew: While I emphatically agree with the title of this post, I think that you are not presenting a complete picture of the P-value. In particular, you are your summary of the ASA statement on p-values as a \"strong stand against\" is incorrect. (I was a participant in the discusssions and forum that led to the statement.) Further, you have linked a blog that seems to be that of an anti-p-value polemicist rather than the ASA statement itself. Perhaps that is a useful strategy to obtain support for a strong stand, but it is bad practice.
\nNo commentary on p-values can avoid further muddying the waters without addressing the necessary distinction between the evidential account of p-values as indices of strength of evidence against the null hypothesised values of the model parameter of interest (Fisherian significance testing) and the alternative long run error rate measure of the analytical method (Neyman-Pearsonian hypothesis testing). See this and this.
\nFH: IMHO the discussion is best conducted without reference to data, sample size, and shrinking confidence intervals. The \"true unknown population value\" of a parameter is not always easy to define but it is not difficult for me to believe that, for example, in a blood pressure study where one treatment arm involves giving patients weather forecasts there is a miniscule but nonzero effect on blood pressure. So H0 is formally not true in this setting. The effect parameter needs to be exactly equal to zero for H0 to pertain. This is not a sampling issue in my view.
\nDaniel Lakens: There has been a lot of criticism on the simplistic arguments in Tukey and Cohen that the null is never true (and I believe general agreement they got it wrong). You either have to take the view that everything causally impacts everything to tiny degrees we can never measure, or you have to admit the null can be true. The latter is more scientific. I've explained this in a blog post: The Null Is Always False (Except When It Is True).
\n\n@online{harrell2017,\n author = {Harrell, Frank and Harrell, Frank},\n title = {P-Values and {Type} {I} {Errors} Are {Not} the\n {Probabilities} {We} {Need}},\n date = {2017-01-14},\n url = {https://www.fharrell.com/post/pvalprobs/},\n doi = {10.59350/f27qd-grr25},\n langid = {en}\n}\nHarrell FE (2017). Null Hypothesis Significance Testing Never Worked. Statistical Thinking. https://doi.org/10.59350/rxzw7-6q428
\nMuch has been written about problems with our most-used statistical paradigm: frequentist null hypothesis significance testing (NHST), p-values, type I and type II errors, and confidence intervals. Rejection of straw-man null hypotheses leads researchers to believe that their theories are supported, and the unquestioning use of a threshold such as p<0.05 has resulted in hypothesis substitution, search for subgroups, and other gaming that has badly damaged science. But we seldom examine whether the original idea of NHST actually delivered on its goal of making good decisions about effects, given the data.
NHST is based on something akin to proof by contradiction. The best non-mathematical definition of the p-value I've ever seen is due to Nicholas Maxwell: \"the degree to which the data are embarrassed by the null hypothesis.\" p-values provide evidence against something, never in favor of something, and are the basis for NHST. But proof by contradiction is only fully valid in the context of rules of logic where assertions are true or false without any uncertainty. The classic paper The Earth is Round (p<.05) by Jacob Cohen has a beautiful example pointing out the fallacy of combining probabilistic ideas with proof by contradiction in an attempt to make decisions about an effect.
\nThe following is almost but not quite the reasononing of null hypothesis rejection:
\nIf the null hypothesis is correct, then this datum (D) can not occur. It has, however, occurred. Therefore the null hypothesis is false.
\nIf this were the reasoning of H0 testing, then it would be formally correct. \u2026 But this is not the reasoning of NHST. Instead, it makes this reasoning probabilistic, as follows:
\nIf the null hypothesis is correct, then these data are highly unlikely. These data have occurred. Therefore, the null hypothesis is highly unlikely.
\nBy making it probabilistic, it becomes invalid. \u2026 the syllogism becomes formally incorrect and leads to a conclusion that is not sensible:
\nIf a person is an American, then he is probably not a member of Congress. (TRUE, RIGHT?) This person is a member of Congress. Therefore, he is probably not an American. (Pollard & Richardson, 1987)
\u2026 The illusion of attaining improbability or the Bayesian Id's wishful thinking error \u2026
Induction has long been a problem in the philosophy of science. Meehl (1990) attributed to the distinguished philosopher Morris Raphael Cohen the saying \"All logic texts are divided into two parts. In the first part, on deductive logic, the fallacies are explained; in the second part, on inductive logic, they are committed.\"
\nSometimes when an approach leads to numerous problems, the approach itself is OK and the problems can be repaired. But besides all the other problems caused by NHST (including need for arbitrary multiplicity adjustments, need for consideration of investigator intentions and not just her actions, rejecting H0 for trivial effects, incentivizing gaming, interpretation difficulties, etc.) it may be the case that the overall approach is defective and should not have been adopted.
\nWith all of the amazing things Ronald Fisher gave us, and even though he recommended against the unthinking rejection of H0, his frequentist approach and dislike of the Bayesian approach did us all a disservice. He called the Bayesian method invalid and was possibly intellectually dishonest when he labeled it as \"inverse probability.\" In fact the p-value is an indirect inverse probability and Bayesian posterior probabilities are direct forwards probabilities that do not condition on a hypothesis, and the Bayesian approach has not only been shown to be valid, but it actually delivers on its promise.
\n@online{harrell2017,\n author = {Harrell, Frank and Harrell, Frank},\n title = {Null {Hypothesis} {Significance} {Testing} {Never} {Worked}},\n date = {2017-01-14},\n url = {https://www.fharrell.com/post/nhst-never/},\n doi = {10.59350/rxzw7-6q428},\n langid = {en}\n}\nHarrell FE (2017). Classification vs.\u00a0Prediction. Statistical Thinking. https://doi.org/10.59350/17jxr-hf782
\nIt is important to distinguish prediction and classification. In many decisionmaking contexts, classification represents a premature decision, because classification combines prediction and decision making and usurps the decision maker in specifying costs of wrong decisions. The classification rule must be reformulated if costs/utilities or sampling criteria change. Predictions are separate from decisions and can be used by any decision maker.
Classification is best used with non-stochastic/deterministic outcomes that occur in say 0.3 - 0.7 of the observations, and not when the simplest classifer (always outputting \"positive\" or always outputting \"negative\") is highly accurate or when two individuals with identical inputs can easily have different outcomes. For these situations, modeling tendencies (i.e., probabilities) is key.
Classification should be used when outcomes are distinct and predictors are strong enough to provide, for all subjects, a probability near 1.0 for one of the outcomes.
The field of machine learning arose somewhat independently of the field of statistics. As a result, machine learning experts tend not to emphasize probabilistic thinking. Probabilistic thinking and understanding uncertainty and variation are hallmarks of statistics. By the way, one of the best books about probabilistic thinking is Nate Silver's The Signal and The Noise: Why So Many Predictions Fail But Some Don't. In the medical field, a classic paper is David Spiegelhalter's Probabilistic Prediction in Patient Management and Clinical Trials.
By not thinking probabilistically, machine learning advocates frequently utilize classifiers instead of using risk prediction models. The situation has gotten acute: many machine learning experts actually label logistic regression as a classification method (it is not). It is important to think about what classification really implies. Classification is in effect a decision. Optimum decisions require making full use of available data, developing predictions, and applying a loss/utility/cost function to make a decision that, for example, minimizes expected loss or maximizes expected utility. Different end users have different utility functions. In risk assessment this leads to their having different risk thresholds for action. Classification assumes that every user has the same utility function and that the utility function implied by the classification system is that utility function.
\nClassification is a forced choice. In marketing where the advertising budget is fixed, analysts generally know better than to try to classify a potential customer as someone to ignore or someone to spend resources on. Instead, they model probabilities and create a lift curve, whereby potential customers are sorted in decreasing order of estimated probability of purchasing a product. To get the \"biggest bang for the buck\", the marketer who can afford to advertise to n persons picks the n highest-probability customers as targets. This is rational, and classification is not needed here.
\nA frequent argument from data users, e.g., physicians, is that ultimately they need to make a binary decision, so binary classification is needed. This is simply not true. First of all, it is often the case that the best decision is \"no decision; get more data\" when the probability of disease is in the middle. In many other cases, the decision is revocable, e.g., the physician starts the patient on a drug at a lower dose and decides later whether to change the dose or the medication. In surgical therapy the decision to operate is irrevocable, but the choice of when to operate is up to the surgeon and the patient and depends on severity of disease and symptoms. At any rate, if binary classification is needed, it must be done at the point of care when all utilities are known, not in a data analysis.
\nWhen are forced choices appropriate? I think that one needs to consider whether the problem is mechanistic or stochastic/probabilistic. Machine learning advocates often want to apply methods made for the former to problems where biologic variation, sampling variability, and measurement errors exist. It may be best to apply classification techniques instead just to high signal:noise ratio situations such as those in which there there is a known gold standard and one can replicate the experiment and get almost the same result each time. An example is pattern recognition - visual, sound, chemical composition, etc. If one creates an optical character recognition algorithm, the algorithm can be trained by exposing it to any number of replicates of attempts to classify an image as the letters A, B, \u2026 The user of such a classifier may not have time to consider whether any of the classifications were \"close calls.\" And the signal:noise ratio is extremely high. In addition, there is a single \"right\" answer for each character. This situation is primarily mechanistic or non-stochastic. Contrast that with forecasting death or disease where two patients with identical known characteristics can easily have different outcomes.
\nWhen close calls are possible, or when there is inherent randomness to the outcomes, probability estimates are called for. One beauty of probabilities is that they are their own error measures. If the probability of disease is 0.1 and the current decision is not to treat the patient, the probability of this being an error is by definition 0.1. A probability of 0.4 may lead the physician to run another lab test or do a biopsy. When the signal:noise ratio is small, classification is usually not a good goal; there one must model tendencies, i.e., probabilities.
\nThe U.S. Weather Service has always phrased rain forecasts as probabilities. I do not want a classification of \"it will rain today.\" There is a slight loss/disutility of carrying an umbrella, and I want to be the one to make the tradeoff.
\nWhether engaging in credit risk scoring, weather forecasting, climate forecasting, marketing, diagnosis a patient's disease, or estimating a patient's prognosis, I do not want to use a classification method. I want risk estimates with credible intervals or confidence intervals. My opinion is that machine learning classifiers are best used in mechanistic high signal:noise ratio situations, and that probability models should be used in most other situations.
\nThis is related to a subtle point that has been lost on many analysts. Complex machine learning algorithms, which allow for complexities such as high-order interactions, require an enormous amount of data unless the signal:noise ratio is high, another reason for reserving some machine learning techniques for such situations. Regression models which capitalize on additivity assumptions (when they are true, and this is approximately true is much of the time) can yield accurate probability models without having massive datasets. And when the outcome variable being predicted has more than two levels, a single regression model fit can be used to obtain all kinds of interesting quantities, e.g., predicted mean, quantiles, exceedance probabilities, and instantaneous hazard rates.
\nA special problem with classifiers illustrates an important issue. Users of machine classifiers know that a highly imbalanced sample with regard to a binary outcome variable Y results in a strange classifier. For example, if the sample has 1000 diseased patients and 1,000,000 non-diseased patients, the best classifier may classify everyone as non-diseased; you will be correct 0.999 of the time. For this reason the odd practice of subsampling the controls is used in an attempt to balance the frequencies and get some variation that will lead to sensible looking classifiers (users of regression models would never exclude good data to get an answer). Then they have to, in some ill-defined way, construct the classifier to make up for biasing the sample. It is simply the case that a classifier trained to a 1/2 prevalence situation will not be applicable to a population with a 1/1000 prevalence. The classifier would have to be re-trained on the new sample, and the patterns detected may change greatly. Logistic regression on the other hand elegantly handles this situation by either (1) having as predictors the variables that made the prevalence so low, or (2) recalibrating the intercept (only) for another dataset with much higher prevalence. Classifiers' extreme dependence on prevalence may be enough to make some researchers always use probability estimators like logistic regression instead. One could go so far as to say that classifiers should not be used at all when there is little variation in the outcome variable (prevalence is near 0 or 1), and that only tendencies (probabilities) should be modeled.
\nOne of the key elements in choosing a method is having a sensitive accuracy scoring rule with the correct statistical properties. Experts in machine classification seldom have the background to understand this enormously important issue, and choosing an improper accuracy score such as proportion classified correctly will result in a bogus model. This is discussed in detail here.
\nThanks to Andrew Howe for suggesting some wording clarifications.
\n@online{harrell2017,\n author = {Harrell, Frank and Harrell, Frank},\n title = {Classification Vs. {Prediction}},\n date = {2017-01-15},\n url = {https://www.fharrell.com/post/classification/},\n doi = {10.59350/17jxr-hf782},\n langid = {en}\n}\nHarrell FE (2017). Ideas for Future Articles. Statistical Thinking. https://doi.org/10.59350/bk0rc-5jv57
\nSuggestions for future articles are welcomed as comments to this entry. Some topics I intend to write about are listed below.
A few weeks ago we had a small discussion at CrossValidated about the pros and cons of matching here. I am sorry that I did not had enough time to elaborate further on the support of matching procedures (in my field researchers do not focus much on a bias-variance tradeoff but they prioritize on minimizing biases. For that reason, they like matching procedures). Now, I have seen that you started a blog recently (congratulations!). I would like to encourage to take up the topic of matching because it is probably interesting for many applied researchers. I think in your 'philosophy', this would belong to the point \"Preserve all the information in the data\". Here, perhaps some input for a blog post. Back then, you wrote:
\nMatching on continuous variables results in an incomplete adjustment because the variables have to be binned.
\nWhat about propensity score matching?
\nMatching throws away good data from observations that would be good matches.
\nExtrapolation bias is only a significant problem if there is a covariate by group interaction, and users of matching methods ignore interactions anyway.
\nHere, you go too far (in my view). You can add interactions, again for example with propensity score matching. Imbens and Rubin (2015) suggest a procedure using quadratic and interaction terms of the covariates.
\nComment: Nice to know this exists but I've never seen a paper that used matching attempt to explore interactions. If you don't want to make regression assumptions that are unverifiable, remove observations outside the overlap region just as with matching.
\nWhich assumptions do you refer to? I think that treating everyone the same (statistically) is also an unverifiable assumption (do you disagree?). What is your opinion about weighted least squares?
\nComment: This is the no-interaction assumption. If you assume additivity then it's more OK to have a no-overlap region, otherwise throw-away non-overlap regions and do a conditional analysis. Not clear on the need for weighting here. In general I like conditioning over weighting.
\nArne Jonas Warnke
Labour Markets, Human Resources and Social Policy
Internet: zew.de
@online{harrell2017,\n author = {Harrell, Frank and Harrell, Frank},\n title = {Ideas for {Future} {Articles}},\n date = {2017-01-16},\n url = {https://www.fharrell.com/post/ideas-articles/},\n doi = {10.59350/bk0rc-5jv57},\n langid = {en}\n}\nHarrell FE (2017). Fundamental Principles of Statistics. Statistical Thinking. https://doi.org/10.59350/njpzb-qcd80
\nThere are many principles involved in the theory and practice of statistics, but here are the ones that guide my practice the most.
\u2026 the statistician must be instinctively and primarily a logician and a scientist in the broader sense, and only secondarily a user of the specialized statistical techniques.
In considering the refinements and modifications of the scientific method which particularly apply to the work of the statistician, the first point to be emphasized is that the statistician is always dealing with probabilities and degrees of uncertainty. He is, in effect, a Sherlock Holmes of figures, who must work mainly, or wholly, from circumstantial evidence.
Malcolm C Rorty: Statistics and the Scientific Method
JASA 26:1-10, 1931
The statistical method is more than an array of techniques. The statistical method is a mode of thought; it is sharpened thinking; it is power.
William Edwards Deming, International Statistical Institute Meeting, Sept 1953
Science is not about falsifying hypotheses. Science is about determining what component of a set of observations are reproducible, and then trying to determine the mechanism or rule to explain those observations\u2026 again, reasoning from data, not by starting with an axiom.
David J Glass
See this post for related thoughts.
\n\n@online{harrell2017,\n author = {Harrell, Frank and Harrell, Frank},\n title = {Fundamental {Principles} of {Statistics}},\n date = {2017-01-18},\n url = {https://www.fharrell.com/post/principles/},\n doi = {10.59350/njpzb-qcd80},\n langid = {en}\n}\nHarrell FE (2017). Split-Sample Model Validation. Statistical Thinking. https://doi.org/10.59350/hqg7h-8p324
\nMethods used to obtain unbiased estimates of future performance of statistical prediction models and classifiers include data splitting and resampling. The two most commonly used resampling methods are cross-validation and bootstrapping. To be as good as the bootstrap, about 100 repeats of 10-fold cross-validation are required.
As discussed in more detail in Section 5.3 of Regression Modeling Strategies Course Notes and the same section of the RMS book, data splitting is an unstable method for validating models or classifiers, especially when the number of subjects is less than about 20,000 (fewer if signal:noise ratio is high). This is because were you to split the data again, develop a new model on the training sample, and test it on the holdout sample, the results are likely to vary significantly. Data splitting requires a significantly larger sample size than resampling to work acceptably well. See also here.
\nThere are also very subtle problems:
\nThere is a less subtle problem but one that is ordinarily not addressed by investigators: unless both the training and test samples are huge, split-sample validation is not nearly as accurate as the bootstrap. See for example the section Studies of Methods Used in the Text here. As shown in a simulation appearing there, bootstrapping is typically more accurate than data splitting and cross-validation that does not use a large number of repeats. This is shown by estimating the \"true\" performance, e.g., the R2 or c-index on an infinitely large dataset (infinite here means 50,000 subjects for practical purposes). The performance of an accuracy estimate is taken as the mean squared error of the estimate against the model's performance in the 50,000 subjects.
\nData are too precious to not be used in model development/parameter estimation. Resampling methods allow the data to be used for both development and validation, and they do a good job in estimating the likely future performance of a model. Data splitting only has an advantage when the test sample is held by another researcher to ensure that the validation is unbiased.
\nMany investigators have been told that they must do an \"external\" validation, and they split the data by time or geographical location. They are sometimes surprised that the model developed in one country or time does not validate in another. They should not be; this is an indirect way of saying there are time or country effects. Far better would be to learn about and estimate time and location effects by including them in a unified model. Then rigorous internal validation using the bootstrap, accounting for time and location all along the way. The end result is a model that is useful for prediction at times and locations that were at least somewhat represented in the original dataset, but without assuming that time and location effects are nil.
\n@online{harrell2017,\n author = {Harrell, Frank and Harrell, Frank},\n title = {Split-Sample {Model} {Validation}},\n date = {2017-01-23},\n url = {https://www.fharrell.com/post/split-val/},\n doi = {10.59350/hqg7h-8p324},\n langid = {en}\n}\nHarrell FE (2017). Clinicians' Misunderstanding of Probabilities Makes Them Like Backwards Probabilities Such As Sensitivity, Specificity, and Type I Error. Statistical Thinking. https://doi.org/10.59350/td3jn-6cf74
\nOptimum decision making in the presence of uncertainty comes from probabilistic thinking. The relevant probabilities are of a predictive nature: P(the unknown given the known). Thresholds are not helpful and are completely dependent on the utility/cost/loss function.
Corollary: Since p-values are P(someone else's data are more extreme than mine if H0 is true) and we don't know whether H0 is true, it is a non-predictive probability that is not useful for decision making.
\nImagine watching a baseball game, seeing the batter get a hit, and hearing the announcer say \"The chance that the batter is left handed is now 0.2!\" No one would care. Baseball fans are interested in the chance that a batter will get a hit conditional on his being right handed (handedness being already known to the fan), the handedness of the pitcher, etc. Unless one is an archaeologist or medical examiner, the interest is in forward probabilities conditional on current and past states. We are interested in the probability of the unknown given the known and the probability of a future event given past and present conditions and events.
Clinicians are people trained in the science and practice of medicine, and most of them are very good at it. They are also very good at many aspects of research. But they are generally not taught probability, and this can limit their research skills. Many excellent clinicians even let their limitations in understanding probability make them believe that their clinical decision making is worse than it actually is. I have taught many clinicians who say \"I need a hard and fast rule so I know how to diagnose or treat patients. I need a hard cutoff on blood pressure, HbA1c, etc. so that I know what to do, and the fact that I either treat or not treat the patient means that I don't want to consider a probability of disease but desire a simple classification rule.\" This makes the clinician try to influence the statistician to use inefficient, arbitrary methods such as categorization, stratification, and matching.
\nIn reality, clinicians do not act that way when treating patients. They are smart enough to know that if a patient has cholesterol just over someone's arbitrary threshold they may not start statin therapy right away if the patient has no other risk factors (e.g., smoking) going against him. They know that sometimes you start a patient on a lower dose and see how she responds, or start one drug and try it for a while and then switch drugs if the efficacy is unacceptable or there is a significant side effect.
\nSo I emphasize the need to understand probabilities when I'm teaching clinicians. A probability is a self-contained summary of the current information, except for the patient's risk aversion and other utilities. Clinicians need to be comfortable with a probability of 0.5 meaning \"we don't know much\" and not requesting a classification of disease/normal that does nothing but cover up the problem. A classification does not account for gray zones or patient and physician utility functions.
\nEven physicians who understand the meaning of a probability are often not understanding conditioning. Conditioning is all important, and conditioning on different things massively changes the meaning of the probabilities being computed. Every physician I've known has been taught probabilistic medical diagnosis by first learning about sensitivity (sens) and specificity (spec). These are probabilities that are in backwards time- and information flow order. How did this happen? Sensitivity, specificity, and receiver operating characteristic curves were developed for radar and radio research in the military. It was a important to receive radio signals from distant aircraft, and to detect an incoming aircraft on radar. The ability to detect something that is really there is definitely important. In the 1950s, virologists appropriated these concepts to measure the performance of viral cultures. Virus needs to be detected when it's present, and not detected when it's not. Sensitivity is the probability of detecting a condition when it is truly present, and specificity is the probability of not detecting it when it is truly absent. One can see how these probabilities would be useful outside of virology and bacteriology when the samples are retrospective, as in a case-control studies. But I believe that clinicians and researchers would be better off if backward probabilities were not taught or were mentioned only to illustrate how not to think about a problem.
\nBut the way medical students are educated, they assume that sens and spec are what you first consider in a prospective cohort of patients! This gives the professor the opportunity of teaching Bayes' rule and requires the use of a supposedly unconditional probability known as prevalence which is actually not very well defined. The students plugs everything into Bayes' rule and fails to notice that several quantities cancel out. The result is the following: the proportion of patients with a positive test who have disease, and the proportion with a negative test who have disease. These are trivially calculated from the cohort data without knowing anything about sens, spec, and Bayes. Instead of computing the obvious, the previous backwards way of thinking harms the student's understanding for years to come and influences those who later engage in clinical and pharmaceutical research to believe that type I error and p-values are directly useful.
\nThe situation in medical diagnosis gets worse when referral bias (also called workup bias) is present. When certain types of patients do not get a final diagnosis, sens and spec are biased. For example, younger women with a negative test may not get the painful procedure that yields the final diagnosis. There are formulas that must be used to correct sens and spec. But wait! When Bayes' rule is used to obtain the probability of disease we needed in the first place, these corrections completely cancel out when the usual correction methods are used! Using forward probabilities in the first place means that one just conditions on age, sex, and result of the initial diagnostic test and no special methods other than (sometimes) logistic regression are required.
\nThere is an analogy to statistical testing. p-values and type I error are affected by sequential testing and a host of other factors, but forward-time probabilities (Bayesian posterior probabilities) are not. Posterior probabilities condition on what is known and does not have to imagine alternate paths to getting to what is known (as do sens and spec when workup bias exists). p-values and type I errors are backwards-information-flow measures, and clinical researchers and regulators come to believe that type I error is the error of interest. They also very frequently misinterpret p-values. Type I error is one minus spec, and power is sens. The posterior probability is exactly analogous to the probability of disease.
\nSens and spec are so pervasive in medicine, bioinformatics, and biomarker research that we don't question how silly they would be in other contexts. Do we dichotomize a response variable so that we can compute the probability that a patient is on treatment B given a \"positive\" response? On the contrary we want to know the full continuous distribution of the response given the assigned treatment. Again this represents forward probabilities.
\nSander Greenland: \"researchers and regulators\u2026 also very frequently misinterpret p-values.\" So do statisticians: \"The p-value is one minus spec, and power is sens.\" NO, that's just wrong. The Type-I error rate (capped by alpha) is 1-spec, and power = sens (so beta = Type-II error rate = 1-sens). These are properties of the test given the design and P is not the first one. P is random, not a property of the test or design; it is a property of the data in relation to the model from which it is computed. That means it varies from data set to data set. In contrast, error rates are calculated across a sequence of data sets.
\nHardly anyone interprets P-values correctly, it seems\u2026but is that that their fault or the fault of statisticians who confuse them with alpha levels or Type-I error rates (test size)?
\nFrank Harrell: Thanks for the correction Sander, although I could imagine a nicer way to state this. I'll correct the analogy in the post from p-value to type I error.
\nSander Greenland: Sorry for the jarring tone, I was reacting to the cottage industry of P-bashing. That industry would be fine except the more I examine each case of supposed P-badness, the more it looks like P is just being misinterpreted - sometimes in obviously wrong ways (as the posterior probability of a point hypothesis) and sometimes in not so obvious ways as in your passage. I should have said that I agree with most everything you said and thought the over-all point was great, so that's why this mistake was so jarring\u2026Well that and I've been battling P-bashers for years now on just these grounds. Not all are like you in terms of seeing and admitting a mistake. Consider the howler \"P-values overstate evidence\" - no they don't; people overinterpret P-values in terms of cutoffs and their meaning. Senn pointed out the confusion here 16 years ago.
\nFH: Thanks very much for that Sander. I agree with you. I have been sloppy in mixing p-values with type I error in a couple of other places so I appreciate the wake-up call.
\nFH: I think you got what I was trying to convey Martin. Diagnostic models require a cohort study or a non-oversampled cross-sectional study. You can also use an oversampled study such as case-control if you can apply a correction to the model's intercept (no other corrections usually needed). Yes you need a nomogram or a prediction calculator to get the baseline risk and then to update it by the (usually continuous) test result. The technology is there now. I agree with your interpretation of the figure, until you know more about action thresholds the clinician/patient use.
\nMartin: Dr.\u00a0Harrell, I read the chapter of your book and I think I'm getting the idea. Please correct me if I'm wrong:
\nWhat you propose is to condition on other clinical variables (which are theoretically a part of the clinical diagnostic process) to \"predict\" the post-test log odds of the disease. A good diagnostic test would add information to a pre-test model (you picture this as the absolute diagnostic yield).
\nAnd that would be the way of knowing which test to choose in a specific clinical setting, instead of sensitivity and specificity (and LRs, I guess).
\nThis model relies on an analytical cohort design. Most of diagnostic studies I've read about are based on a cross-sectional design. I wouldn't know if this can change your proposal, but it doesn't look like it does.
\nMy other concern is the translation into practice. I guess we would need the nomogram or clinical diagnostic rule (mobile app) for it to work. This can be cumbersome in some settings.
\nPS: How can one interpret figure 19.6? The test is most useful with pretest probs < 80%? I'm having a hard time interpreting this figure.
\nFH: Thanks for continuing this discussion because it will help me better understand the clinical issues. I should say that it's not they they are unimportant but that they are too indirect. Chapter 19 of BBR gives my suggestions. Briefly I want a partial effect plot for each predictor, whether it's a test output or a regular baseline variable. For a test output this is the contribution to the log odds of disease holding all the baseline variables constant. As shown in that chapter once you know the regression fit (odds ratio for a binary test) you can show its impact on absolute risk of disease as a function of baseline risk in general (regardless of what gave the patient a high risk, for example). For the rare case where the test is binary, the odds ratio is the product of the likelihood ratio positive and the likelihood ratio negative.
\nMartin: Thank you for your reply, Dr.\u00a0Harrell.
\nMaybe we're talking past each other, and since this is an important topic, I'll try my best to understand your point of view.
\nTo answer your question, in the context of a RCT it would not make any sense.
\nMy question to you is, if sensitivity and specificity are not important, what would be the criteria a physician should have to choose a diagnostic test for a given clinical situation?
\nFH: Nothing you discussed motivates the need for sensitivity and specificity. What I need is the effect of each piece of information, e.g., the effect of age, the effect of having a test result of 17.35 on the log odds of disease. Give me a very specific example where sens and spec were needed and probability of disease or probability of outcome did not do the job.
\nIf you were not diagnosing a patient but were interpreting RCT results would you take time to compute the probability that a patient was on treatment B if she improved her blood pressure?
\nSteve Pitts: I believe I understand your point of view about sn/sp vs probabilities. The main advantage of sensitivity and specificity is the intuitive one: Dr.\u00a0Yerushalmy was onto something with these adjectives. We all know people or pets that are \"sensitive\". It enables medical students to understand intuitively what is actually a pretty complicated subject. The main first order of business is to get them past the TV-show level of understanding, where a positive test means you have disease, and negative means you don't. Tests have properties that are more constant than individual posterior probabilities, whether the test is a clinical finding or an expensive machine.
\nFH: Thanks for the comments Steve. I think it is very appropriate to show people how to move past positive=disease, negative=no disease thinking. I just don't see the role an an indirect way to do this that is based on P(positive given disease) etc. I'd rather go for the jugular: P(disease given all test results and background variables). And beware of assuming that sensitivity and specificity are quite that constant.
\nMartin: Dr.\u00a0Harrell, diagnostic tests have different uses in clinical practice. They can confirm or discard a disease when there's suspicion of disease, discard a disease in an asymptomatic patient (screening), they also have prognostic capabilities, and they can help us decide which therapy is better.
\nNow, we cannot mix them up, these are different questions, although some tests can be both diagnostic and prognostic.
\nSensitivity and specificity help us to choose which test to order according to the clinical context. For example, an EKG could not be used to discard a myocardial infarction and a D-dimer could not be used to confirm pulmonary thromboembolism.
\nRegarding prevalence. It works as a starting point for prior probabilities, which take into account specific patient characteristics. This and the likelihood of the test result gives us the posterior probabilities. Since rarely are we ever 100% sure of the diagnosis, based on some specified (subjective) thresholds we decide to treat, discard, or take another test.
\nSo, I would have to disagree that sensitivity and specificity are not important in medical training.
\nFH: I disagree. The confusion caused by sens and spec is massive and even more clearly they take valuable class and study time away from the real need: how to specify probability models and design studies that will yield the data need to estimate their parameters. The only place I can thing of for sens and spec is when you don't have data, or at least the needed data. Do we ever teach physicians how to compute the probability that a patient is male given his disease status? Why is a test different from the patient's sex?
\nClyde Schechter: The only thing I disagree with here is your implication that clinicians should not be taught about sensitivity and specificity. It really is important to know those concepts, along with prevalence and Bayes' theorem. The reason is that some tests are used in different populations and settings. The same test may be used to mass screen asymptomatic populations, and also as a confirmatory diagnostic test in people with symptoms suggestive of the disease in question. In the former setting the prior probability of disease is typically very low, and in the latter it is usually moderate or even high. The sensitivity and specificity are, under reasonably broad circumstances, the same in both populations as they are generally physico-chemical properties of the test itself. The clinician needs to understand that, and why, the same test result has very different implications in these two uses.
\nIn fact, I think that this is not sufficiently emphasized in medical training. Most of the clinicians I deal with, even those relatively recently trained, are shocked to learn how low the positive predictive value of most screening tests are. But once they have mastered that, they also need to understand why the very same test can be very informative when applied to a patient who is likely to have the disease in question.
\ntherandomtexan: This merits a series of 4 or 5 lectures that tie all these ideas together and get out students past mere sensitivity, specificity, and the PPV. You've just nailed my foot the floor for a couple of days this summer, writing those lectures.
\n\n@online{harrell2017,\n author = {Harrell, Frank and Harrell, Frank},\n title = {Clinicians' {Misunderstanding} of {Probabilities} {Makes}\n {Them} {Like} {Backwards} {Probabilities} {Such} {As} {Sensitivity,}\n {Specificity,} and {Type} {I} {Error}},\n date = {2017-01-25},\n url = {https://www.fharrell.com/post/backwards-probs/},\n doi = {10.59350/td3jn-6cf74},\n langid = {en}\n}\nBecause of limited definitive sources of information about grant-level terminations, EPA awards have a more complex status determination logic than other agencies.
\nDOGE provides data on terminated awards, although we have found that it is incomplete or occasionally incorrect, so for EPA we use it only in combination with other sources as a signal of grant termination.
\nWe use this source primarily for metadata including grantee information, grant assistance listing (CFDA), grant funding opportunity (also called notice of funding announcement or NOFO), and total obligated and total outlaid amounts.\nWe also track changes to project end dates (i.e. project end date moved earlier, a \"cutoff\", or later, a \"reextension\"), and deobligation events (i.e. a negative amount obligated) which we use in combination as signals of disruption.
\nFile C is released roughly once a month (except that Sep/Oct are combined into one release).\nWe use this source primarily for outlay actions which we use to confirm reinstatements.
\nEPA's Next Generation Grants System (NGGS) provides an API to download grant-level information and we snapshot this data regularly.\u00a0\nWe use it as a source of grant metadata such as grant description, grantee information, etc.\nNotably, it is our only source for grant titles.\nUnfortunately, it does not carry a clear signal of grant termination, but we use project end date cutoffs and deobligation events in combination with similar signals from USAspending award data in our status determination.
\nWe have information from a trusted confidential source about particular EPA assistance listings (also called CFDAs) that have been terminated that we use in combination with other signals to determine grant status.
\nThe Our Environment and Public Works Senate Committee released a list of terminated EPA awards in March 2025. This is one of our definitive sources of grant terminations.
\nIn March 2025, EPA terminated eight National Clean Investment Fund (NCIF) and Clean Communities Investment Accelerator (CCIA) awards.\nThe CCIA and NCIF programs appear as though they were fully outlaid on USAspending.\nHowever, the outlaid assets are in accounts frozen by the federal government.\nWe do not know exactly how much has been spent, but it is closer to $0 than to the entire award value, so we have manually adjusted the 'funds spent before disruption' and 'funds remaining before disruption' values for these awards.
\nAt the time of writing, there are two court decisions ordering the reinstatement of EPA awards\u2014Sustainability Institute v. Trump and Green and Healthy Homes Initiative Inc. vs EPA.\nWe use these documents as a definitive source of award reinstatement and will continue to add court documents for similar decisions in the future.
\nEPA awards are marked as terminated if any of the following conditions are met:
\nAwards have been reported as such by PIs or trusted sources
\nOur sources show the awards' assistance listing has been terminated and they have experienced a project end date cutoff or are reported terminated by DOGE, or have a deobligation.
\nAwards appear in any of our sources of known terminations (see Our Environment and Public Works Senate Committee and Greenhouse Gas Reduction Fund above)
\nAwards are reported terminated by DOGE and have a project end date cutoff
\nAwards appear in court documents ordering reinstatement (we assume that if they were ordered reinstated, they must have been terminated for EPA)
\nEPA awards are considered reinstated if they have had their end date re-extended after a date cutoff, appear in court documents ordering reinstatement, or are reported as reinstated by PIs or trusted sources.\nReinstatements are confirmed after they receive an outlay.
\nThe funds obligated and spent at time of earliest disruption are calculated using the current total obligated amount minus obligations since the earliest termination date and the total outlays minus outlays since the earliest termination date, respectively.\nThe outlay and obligation events come from File C, while the current total obligated and outlaid numbers are from the more regularly updating USAspending awards data.\nAs a result, these estimates may fluctuate week-to-week.
","doi":"https://doi.org/10.59350/tq3c6-7x998","guid":"https://doi.org/10.59350/tq3c6-7x998","language":"en","license":"https://creativecommons.org/licenses/by/4.0/legalcode","published_at":1792713600,"rid":"z98pv-wa015","summary":"Because of limited definitive sources of information about grant-level terminations, EPA awards have a more complex status determination logic than other agencies.","tags":["Methods","Environmental Protection Agency"],"title":"EPA Grant Disruption Methodology","updated_at":1792793705,"url":"https://grantwitness.org/epa/updates/methodology","version":"v1"},{"authors":[{"affiliation":[{"name":"CSM Advising LLC"}],"contributor_roles":[],"family":"Marcum","given":"Christopher","url":"https://orcid.org/0000-0002-0899-6143"}],"blog":{"authors":null,"community_id":"8bdb1ae7-4621-4fa5-ad1a-3a639417dfd5","created":1768694400,"current_feed_url":null,"description":"Perspectives on science, data, and technology that don't fit anywhere else.","doi":"https://doi.org/10.59350/chrismarcum","favicon":"https://rogue-scholar.org/api/communities/8bdb1ae7-4621-4fa5-ad1a-3a639417dfd5/logo","feed_format":"application/atom+xml","feed_url":"http://chrismarcum.com/marcum-blog/feed.atom","filter":null,"generator":"Jekyll","home_page_url":"https://www.chrismarcum.com/marcum-blog/","issn":null,"language":"eng","license":"https://creativecommons.org/licenses/by/4.0/legalcode","prefix":"10.59350","relative_url":null,"secure":true,"slug":"chrismarcum","status":"active","subfield":"3312","title":"Open Evidence","updated":1792713600,"use_api":null},"blog_name":"Open Evidence","blog_slug":"chrismarcum","content_html":"In July, we showed that anyone could inflate the \"comments received\" number displayed on regulations.gov \u2013 that is, the number of comments submitted during a public comment period. When you attached a file to a comment, the form asked whether the file contained comments from more than one person or entity. Check yes, and you would be asked for the \"Number of persons/entities represented\" by your comment. You could put any number from 1 to 999,999, and whatever number you picked would be added to the docket's \"comments submitted\" count. We tested it on a Bureau of Land Management docket, and the count jumped from 310 to over a million by the next day.
\nAnd this topline \"comments received\" number drives news coverage. Reporters cited 500,000 comments on OMB's proposed revisions to the Uniform\nGuidance (2 CFR 200) as a measure of public interest. The true count was still very high at over 164K, but that's a far cry from the half million reported by the press.
\nLast weekend, we noticed that this had been fixed, and looked back at archived copies of regulations.gov to see when. We found that on September 4, the General Services Administration (GSA), which runs regulations.gov, removed the question from the comment form. A week later, on September 10, GSA relabeled the count and rewrote its description. We don't know for sure whether our post or the press coverage we received on it had anything to do with the decision to fix the count bug. Either way, credit to the civil servants at GSA who made the change happen. That's good governance.
\nRegulations.gov runs on a single JavaScript file that generates the comment form and the count box. Every time GSA updates the site, the file gets a new name. The Internet Archive's Wayback Machine saves copies of those files as it crawls, so we can line up every version from the spring through today and check each one \u2013 which we did.\nEvery prior version we checked until midday on September 4, 2026 includes the \"persons/entities\" question on the comment form. Every version after does not. We were first clued into this because we noticed that, despite a very high rate of comments being submitted from campaigns to the US Census Bureau's proposed rulemaking on residency requirements for the 2030 census \u2013 a controversial rulemaking where we would expect such gaming \u2013 the comments submitted to comments published had a relatively consistent difference over time. We were tracking this because of our work on handsoffourcens.us.
\nWe investigated further to find out if GSA had indeed fixed the bug.
\n| Version | \nSaved by Wayback | \nQuestion on form? | \nCount description | \n
|---|---|---|---|
| e4870fb6 | \nSep 4, 12:15 PM ET | \nYes | \n\"as of 11:59 PM yesterday\" | \n
| c583eabb | \nSep 4, 3:14 PM ET | \nNo | \n\"as of 11:59 PM yesterday\" | \n
| c92c9d40 | \nSep 10, 7:56 PM ET | \nNo | \n\"including unpublished\" | \n
\n\nThe new count description on the dockets is better, too. It now reads:\n\"The number of comment submissions that have been received (including unpublished) to all documents within this docket.\"
\n
The counts for dockets prior to this change are still on this site. For instance, our BLM test docket still shows 1,037,743 submissions received against 4,477 posted, and the OMB docket still shows 496,606 against 164,654. Neither was corrected.
\nWe also wanted to know whether anyone else had used the loophole before we wrote about it. So we pulled every proposed rule on regulations.gov whose comment period closed between January 2025 and early September 2026, took a random sample of 300, and compared the number received to the number posted. Most line up closely, which is what you'd expect: the difference is usually comments an agency is still reviewing or chose not to post. A few don't line up at all.\nThe standout is an FAA airworthiness directive for a single Airbus Canada aircraft model. These are technical maintenance orders that usually draw a handful of comments. This one shows 656,575 received and 4 posted. A Copyright Office rule about paying royalties through Pay.gov shows 91,510 received and none posted. We can't prove those numbers were inflated, but we also can't think of another way they got there.
\nNot every big gap reflects a user entering a large number into \"persons/entities\" field. Sometimes, agencies just don't post every individual comment for a mass-mail campaign, and so the posted count will remain significantly below the received count. The EPA in particular will post one representative letter to stand in for thousands of identical ones. EPA's Waters of the United States rule, which closed October 9, shows 121,798 received and 2,095 posted in the API backend. But we can see what's driving this: the EPA posted 47 \"mass comment campaign\" representative records on that rule, which likely account for much of the gap.
\n| Docket | \nComment period closed | \nReceived | \nPosted | \nDifference | \n
|---|---|---|---|---|
| Closed before the fix | \n\u00a0 | \n\u00a0 | \n\u00a0 | \n\u00a0 | \n
| OMB-2026-0034 | \nJul 13, 2026 | \n496,606 | \n164,654 | \n331,952 | \n
| BLM-2026-0068 | \nAug 14, 2026 | \n1,037,743 | \n4,477 | \n1,033,266 | \n
| FAA-2025-0199-0008 | \nNov 10, 2025 | \n656,575 | \n4 | \n656,571 | \n
| COLC-2024-0004-0001 | \nJan 13, 2025 | \n91,510 | \n0 | \n91,510 | \n
| FWS-R4-ES-2024-0073-0001 | \nJan 24, 2025 | \n70,899 | \n38,045 | \n32,854 | \n
| EPA-HQ-OAR-2024-0419-0001 | \nApr 15, 2025 | \n20,979 | \n194 | \n20,785 | \n
| IRS-2025-0235-0001 | \nOct 22, 2025 | \n16,903 | \n321 | \n16,582 | \n
| DOT-OST-2025-0831 | \nAug 21, 2026 | \n1,357 | \n1,354 | \n3 | \n
| OCC-2026-0331 | \nAug 21, 2026 | \n524 | \n23 | \n501 | \n
| EPA-HQ-OW-2026-1090 | \nMay 26, 2026 | \n47 | \n47 | \n0 | \n
| Comment period open after the fix | \n\u00a0 | \n\u00a0 | \n\u00a0 | \n\u00a0 | \n
| EPA-HQ-OW-2025-0322-15447 | \nOct 9, 2026 | \n121,798 | \n2,095 | \n119,703 | \n
| FS-2025-0001-342438 | \nOct 6, 2026 | \n19,826 | \n19,826 | \n0 | \n
| FAR-2026-0003-0001 | \nOct 19, 2026 | \n95 | \n95 | \n0 | \n
| OPM-2025-0012-0678 | \nSep 29, 2026 | \n81 | \n76 | \n5 | \n
| NRC-2025-1204-0004 | \nOct 13, 2026 | \n52 | \n52 | \n0 | \n
\n\nCounts pulled from the regulations.gov API on October 23, 2026 (October 18 for the OMB, BLM, DOT, OCC, and EPA water docket rows). Rows linked to a docket count the whole docket. Rows linked to a document count that one comment period for a specific document within a docket.
\n
All data come from the public regulations.gov API, the same one the website uses. \"Received\" is the API's received count, and \"posted\" is the number of comments the API returns. The before-fix sample was drawn at random from about 2,600 whose comment periods closed between January 1, 2025 and September 3, 2026. The after-fix dockets represent every proposed rule, 35 in all, whose comment period opened September 5 or later and closed by October 22. The table shows the five largest by number received from each group, plus our original docket use-cases. Only two of the after-fix rules drew more than 100 submissions, so this sample can't tell us much about how counts behave now. We also checked the figures against snapshots captured by Wayback Machine.
\nThe proof of the fix is the document upload form itself: the entities count question is gone.
\nWe're really glad that this is fixed going forward, and we think a few small tweaks would complete the change. First, recalculate the counts on dockets where the \"persons/entities\" number was used, or at least flag them. Second, GSA should publish a short note explaining the change, so the public knows the count means something different now. Third, GSA should update the language about comment counts on its FAQ page, which is now out-of-date. We're following this issue closely, but it still took us over a month to catch this: other people who use this data might miss this important detail unless GSA points this out.
","doi":"https://doi.org/10.59350/nd0y0-agy83","guid":"https://www.chrismarcum.com/marcum-blog/2026/10/23/GSA-Closed-the-Comment-Loophole-on-Regulations-dot-gov","language":"en","license":"https://creativecommons.org/licenses/by/4.0/legalcode","published_at":1792713600,"rid":"b68eb-yej13","summary":"In July, we showed that anyone could inflate the \"comments received\" number displayed on regulations.gov \u2013 that is, the number of comments submitted during a public comment period. When you attached a file to a comment, the form asked whether the file contained comments from more than one person or entity. Check yes, and you would be asked for the \"Number of persons/entities represented\" by your comment.","tags":["General"],"title":"GSA Closed the Comment Count Loophole on Regulations.gov","updated_at":1792790704,"url":"https://www.chrismarcum.com/marcum-blog/2026/10/23/GSA-Closed-the-Comment-Loophole-on-Regulations-dot-gov.html","version":"v1"},{"authors":[{"affiliation":[{"id":"https://ror.org/05dq2gs74","name":"Vanderbilt University Medical Center"}],"contributor_roles":[],"family":"Harrell","given":"Frank","url":"https://orcid.org/0000-0002-8271-5493"}],"blog":{"authors":[{"name":"Frank Harrell","url":"https://orcid.org/0000-0002-8271-5493"}],"community_id":"e74e284b-0d7e-4e86-bfae-58b71f06a7cf","created":1792713600,"current_feed_url":null,"description":"Parametric Models vs. Empirical Distributions and Ordinal Regression.","doi":"https://doi.org/10.59350/fharell","favicon":"https://rogue-scholar.org/api/communities/e74e284b-0d7e-4e86-bfae-58b71f06a7cf/logo","feed_format":"application/rss+xml","feed_url":"https://www.fharrell.com/index.xml","filter":null,"generator":"Quarto","home_page_url":"https://www.fharrell.com/","issn":null,"language":"eng","license":"https://creativecommons.org/licenses/by/4.0/legalcode","prefix":"10.59350","relative_url":null,"secure":null,"slug":"fharrell","status":"active","subfield":"2613","title":"Statistical Thinking","updated":1786251600,"use_api":null},"blog_name":"Statistical Thinking","blog_slug":"fharrell","content_html":"Harrell FE (2017). Introduction. Statistical Thinking. https://doi.org/10.59350/hepkr-mgd06
\nStatistics is a field that is a science unto itself and that benefits all other fields and everyday life. What is unique about statistics is its proven tools for decision making in the face of uncertainty, understanding sources of variation and bias, and most importantly, statistical thinking. Statistical thinking is a different way of thinking that is part detective, skeptical, and involves alternate takes on a problem. An excellent example of statistical thinking is statistician Abraham Wald's analysis of British bombers surviving to return to their base in World War II: his conclusion was to reinforce bombers in areas in which no damage was observed. For other great examples watch my colleague Chris Fonnesbeck's Statistical Thinking for Data Science.
Some of my personal philosophy of statistics can be summed up in the list below:
\nStatistics has multiple challenges today, which I break down into three major sources:
\nI will be discussing several of the issues in future blogs, especially item 1 above and items 2 and 4 below. Complacency in the field of statistics and in statistical education has resulted in
\nI was interviewed by Kevin Gray in July, 2017 where more of my opinions about statistics may be found.
\nSee this article for related thoughts.
\nMy top role model is Sherlock Holmes. See this video for an excellent example.
\n\n@online{harrell2017,\n author = {Harrell, Frank and Harrell, Frank},\n title = {Introduction},\n date = {2017-01-13},\n url = {https://www.fharrell.com/post/introduction/},\n doi = {10.59350/hepkr-mgd06},\n langid = {en}\n}\nHarrell FE (2017). p-values and Type I Errors are Not the Probabilities We Need. Statistical Thinking. https://doi.org/10.59350/f27qd-grr25
\nIn trying to guard against false conclusions, researchers often attempt to minimize the risk of a \"false positive\" conclusion. In the field of assessing the efficacy of medical and behavioral treatments for improving subjects' outcomes, falsely concluding that a treatment is effective when it is not is an important consideration. Nowhere is this more important than in the drug and medical device regulatory environments, because a treatment thought not to work can be given a second chance as better data arrive, but a treatment judged to be effective may be approved for marketing, and if later data show that the treatment was actually not effective (or was only trivially effective) it is difficult to remove the treatment from the market if it is safe. The probability of a treatment not being effective is the probability of \"regulator's regret.\" One must be very clear on what is conditioned upon (assumed) in computing this probability. Does one condition on the true effectiveness or does one condition on the available data? Type I error conditions on the treatment having no effect and does not entertain the possibility that the treatment actually worsens the patients' outcomes. Can one quantify evidence for making a wrong decision if one assumes that all conclusions of non-zero effect are wrong up front because H0 was assumed to be true? Aren't useful error probabilities the ones that are not based on assumptions about what we are assessing but rather just on the data available to us?
Statisticians have convinced regulators that long-run operating characteristics of a testing procedure should rule the day, e.g., if we did 1000 clinical trials where efficacy was always zero, we want no more than 50 of these trials to be judged as \"positive.\" Never mind that this type I error operating characteristic does not refer to making a correct judgment for the clinical trial at hand. Still, there is a belief that type I error is the probability of regulator's regret (a false positive), i.e., that the treatment is not effective when the data indicate it is. In fact, clinical trialists have been sold a bill of goods by statisticians. No probability derived from an assumption that the treatment has zero effect can provide evidence about that effect. Nor does it measure the chance of the error actually in question. All probabilities are conditional on something, and to be useful they must condition on the right thing. This usually means that what is conditioned upon must be knowable.
\nThe probability of regulator's regret is the probability that a treatment doesn't work given the data. So the probability we really seek is the probability that the treatment has no effect or that it has a backwards effect. This is precisely one minus the Bayesian posterior probability of efficacy.
\nIn reality, there is unlikely to exist a treatment that has exactly zero effect. As Tukey argued in 1991, the effects of treatments A and B are always different, to some decimal place. So the null hypothesis is always false and the type I error could be said to be always zero.
\nThe best paper I've read about the many ways in which p-values are misinterpreted is Statistical tests, P values, confidence intervals, and power: a guide to misinterpretations written by a group of renowned statisticians. One of my favorite quotes from this paper is
\nThus to claim that the null P value is the probability that chance alone produced the observed association is completely backwards: The P value is a probability computed assuming chance was operating alone. The absurdity of the common backwards interpretation might be appreciated by pondering how the P value, which is a probability deduced from a set of assumptions (the statistical model), can possibly refer to the probability of those assumptions.
\nIn 2016 the American Statistical Association took a stand against over-reliance on p-values. This would have made a massive impact on all branches of science had it been issued 50 years ago but better late than never.
\nThough believed to be true by many non-statisticians, p-values are not the probability that H0 is true, and to turn them into such probabilities requires Bayes' rule. If you are going to use Bayes' rule you might as well formulate the problem as a full Bayesian model. This has many benefits, not the least of them being that you can select an appropriate prior distribution and you will get exact inference. Attempts by several authors to convert p-values to probabilities of interest (just as sensitivity and specificity are converted to probability of disease once one knows the prevalence of disease) have taken the prior to be discontinuous, putting a high probability on H0 being exactly true. In my view it is much more sensible to believe that there is no discontinuity in the prior at the point represented by H0, encapsulating prior knowledge instead by saying that values near H0 are more likely if no relevant prior information is available.
\nReturning to the non-relevance of type I error as discussed above, and ignoring for the moment that long-run operating characteristics do not directly assist us in making judgments about the current experiment, there is a subtle problem that leads researchers to believe that by controlling type I \"error\" they think they have quantified the probability of misleading evidence. As discussed at length by my colleague Jeffrey Blume, once an experiment is done the probability that positive evidence is misleading is not type I error. And what exactly does \"error\" mean in \"type I error?\" It is the probability of rejecting H0 when H0 is exactly true, just as the p-value is the probability of obtaining data more impressive than that observed given H0 is true. Are these really error probabilities? Perhaps \u2026 if you have been misled earlier into believing that we should base conclusions on how unlikely the observed data would have been observed under H0. Part of the problem is in the loaded word \"reject.\" Rejecting H0 by seeing data that are unlikely if H0 is true is perhaps the real error.
\nThe \"error quantification\" truly needed is the probability that a treatment doesn't work given all the current evidence, which as stated above is simply one minus the Bayesian posterior probability of positive efficacy.
\nType I error control is an indirect way to being careful about claims of effects. It should never have been the preferred method for achieving that goal. Seen another way, we would choose type I error as the quantity to be controlled if we wanted to:
\nThe Bayesian approach focuses on the chances you give effects to be positive and does not have multiplicity issues (potential issues such as examining treatment effects in multiple subgroups are handled by the shrinkage that automatically results when you use the 'right' Bayesian hierarchical model).
\nThe p-value is the chance that someone else would observe data more extreme than mine if the effect is truly zero (if they could exactly replicate my experiment) and not the probability of no (or a negative) effect of treatment given my data.
\nAs discussed in Gamalo-Siebers at al DOI: 10.1002/pst.1807 the type I error is the probability of making an assertion of an effect when no such effect exists. It is not the probability of regret for a decision maker, e.g., it is not the probability of a drug regulator's regret. The probability of regret is the probability that the drug doesn't work or is harmful when the decision maker had decided it was helpful. It is the probability of harm or no benefit when an assertion of benefit is made. This is best thought of as the probability of harm or no benefit given the data which is one minus the probability of efficacy. Prob(assertion|no benefit) is not equal to 1-Prob(benefit|data).
\nType I (\"false positive\") error probability would be a useful concept while a study is being designed. Frequentists speak of type I error control, but after a study is completed, the only way to commit a type I error is to know with certainty that an effect is exactly zero. But then the study would not have been necessary. So type I error remains a long-run operating characteristic for a sequence of hypothetical studies.
\nThinking of p-values that a sequence of hypothetical studies might provide, when the type I error is \u03b1 this means P(p-value < \u03b1 | zero effect) = \u03b1. Neither a single p-value nor \u03b1 is the probability of a decision error. They are \"what if\" probabilities, if the effect is zero. The p-value for a single study is merely the probability that data more extreme than ours would have been observed had the effect been exactly zero and the experiment was capable of being re-run infinitely often. It is nothing more than this. It is not a false positive probability for the experiment at hand. To compute the false positive probability one would need a prior distribution for the effect (and for the p-value to be perfectly accurate which is rare), and one might as well be fully Bayesian and enjoy all the Bayesian benefits.
\nFrank Harrell: You are correct; sorry for the omission of the condition that H0 is true. Thanks for the comments.
\nHenry: When you write the following, \"The p-value is the chance that someone else would observe data more extreme than mine (if they could exactly replicate my experiment) and not the probability of no (or a negative) effect of treatment given my data,\" I am guessing that you left out, \"conditional that the null hypothesis is true.\" While a nitpick, I thought that this is an important and meaningful addendum as its absence changes the meaning to a non-stats reader.
\nGreat fan of your book and your posts on Stackexchange.
\nFH: Thanks for the perceptive comments. I agree in spirit with all of them until you get into multiplicity where the fundamental problems with p-values just makes everything implode. But in general, and in line with your thoughts, I think that it's not the case that p-values are not related to evidence, it's just that they are not calibrated to provide the quantification of evidence we need. So I need to say these points more clearly, e.g., p-values are the correct functions of the data, and except for not being able to deal with prior knowledge, they are measures of evidence but are not calibrated to what is needed for direct action/decision making.
\nMichael Lew: I like very much the update of January 20, but I think the last sentence misses the important nature of the P-value.
\nThe best way to ensure care is taken about claims of effects is to make sure that scientists think scientifically: carefully, critically, and consider more than just the statistical result of any particular experiment or study component.
\nBayesian approaches help, presumably, but my opinion is that scientists would be more directly helped in their thinking if the statistics of evidence were emphasised over the statistics of posterior probabilities or the statistics of errors.
\nFisher's P-values have an evidential meaning that has nothing to do with error probabilities. They may not be as rich a depiction of evidence as a likelihood function, but they communicate evidence nonetheless.
\n**FH*: These are great points. And I should have said \"stand\" instead of \"strong stand\", or \"expressed significant caution\". I linked to another blog instead of the ASA because the other blog linked to ASA and had some other statements I liked. I would like to note that evidence against H0 is not the kind of evidence I seek in science. A recent paper I co-authored delves into the Fisher vs Neyman-Pearson approaches - see this. One reason there are the two schools of thought is that the entire NHST premise is flawed in my humble opinion.
\nMichael Lew: While I emphatically agree with the title of this post, I think that you are not presenting a complete picture of the P-value. In particular, you are your summary of the ASA statement on p-values as a \"strong stand against\" is incorrect. (I was a participant in the discusssions and forum that led to the statement.) Further, you have linked a blog that seems to be that of an anti-p-value polemicist rather than the ASA statement itself. Perhaps that is a useful strategy to obtain support for a strong stand, but it is bad practice.
\nNo commentary on p-values can avoid further muddying the waters without addressing the necessary distinction between the evidential account of p-values as indices of strength of evidence against the null hypothesised values of the model parameter of interest (Fisherian significance testing) and the alternative long run error rate measure of the analytical method (Neyman-Pearsonian hypothesis testing). See this and this.
\nFH: IMHO the discussion is best conducted without reference to data, sample size, and shrinking confidence intervals. The \"true unknown population value\" of a parameter is not always easy to define but it is not difficult for me to believe that, for example, in a blood pressure study where one treatment arm involves giving patients weather forecasts there is a miniscule but nonzero effect on blood pressure. So H0 is formally not true in this setting. The effect parameter needs to be exactly equal to zero for H0 to pertain. This is not a sampling issue in my view.
\nDaniel Lakens: There has been a lot of criticism on the simplistic arguments in Tukey and Cohen that the null is never true (and I believe general agreement they got it wrong). You either have to take the view that everything causally impacts everything to tiny degrees we can never measure, or you have to admit the null can be true. The latter is more scientific. I've explained this in a blog post: The Null Is Always False (Except When It Is True).
\n\n@online{harrell2017,\n author = {Harrell, Frank and Harrell, Frank},\n title = {P-Values and {Type} {I} {Errors} Are {Not} the\n {Probabilities} {We} {Need}},\n date = {2017-01-14},\n url = {https://www.fharrell.com/post/pvalprobs/},\n doi = {10.59350/f27qd-grr25},\n langid = {en}\n}\nHarrell FE (2017). Null Hypothesis Significance Testing Never Worked. Statistical Thinking. https://doi.org/10.59350/rxzw7-6q428
\nMuch has been written about problems with our most-used statistical paradigm: frequentist null hypothesis significance testing (NHST), p-values, type I and type II errors, and confidence intervals. Rejection of straw-man null hypotheses leads researchers to believe that their theories are supported, and the unquestioning use of a threshold such as p<0.05 has resulted in hypothesis substitution, search for subgroups, and other gaming that has badly damaged science. But we seldom examine whether the original idea of NHST actually delivered on its goal of making good decisions about effects, given the data.
NHST is based on something akin to proof by contradiction. The best non-mathematical definition of the p-value I've ever seen is due to Nicholas Maxwell: \"the degree to which the data are embarrassed by the null hypothesis.\" p-values provide evidence against something, never in favor of something, and are the basis for NHST. But proof by contradiction is only fully valid in the context of rules of logic where assertions are true or false without any uncertainty. The classic paper The Earth is Round (p<.05) by Jacob Cohen has a beautiful example pointing out the fallacy of combining probabilistic ideas with proof by contradiction in an attempt to make decisions about an effect.
\nThe following is almost but not quite the reasononing of null hypothesis rejection:
\nIf the null hypothesis is correct, then this datum (D) can not occur. It has, however, occurred. Therefore the null hypothesis is false.
\nIf this were the reasoning of H0 testing, then it would be formally correct. \u2026 But this is not the reasoning of NHST. Instead, it makes this reasoning probabilistic, as follows:
\nIf the null hypothesis is correct, then these data are highly unlikely. These data have occurred. Therefore, the null hypothesis is highly unlikely.
\nBy making it probabilistic, it becomes invalid. \u2026 the syllogism becomes formally incorrect and leads to a conclusion that is not sensible:
\nIf a person is an American, then he is probably not a member of Congress. (TRUE, RIGHT?) This person is a member of Congress. Therefore, he is probably not an American. (Pollard & Richardson, 1987)
\u2026 The illusion of attaining improbability or the Bayesian Id's wishful thinking error \u2026
Induction has long been a problem in the philosophy of science. Meehl (1990) attributed to the distinguished philosopher Morris Raphael Cohen the saying \"All logic texts are divided into two parts. In the first part, on deductive logic, the fallacies are explained; in the second part, on inductive logic, they are committed.\"
\nSometimes when an approach leads to numerous problems, the approach itself is OK and the problems can be repaired. But besides all the other problems caused by NHST (including need for arbitrary multiplicity adjustments, need for consideration of investigator intentions and not just her actions, rejecting H0 for trivial effects, incentivizing gaming, interpretation difficulties, etc.) it may be the case that the overall approach is defective and should not have been adopted.
\nWith all of the amazing things Ronald Fisher gave us, and even though he recommended against the unthinking rejection of H0, his frequentist approach and dislike of the Bayesian approach did us all a disservice. He called the Bayesian method invalid and was possibly intellectually dishonest when he labeled it as \"inverse probability.\" In fact the p-value is an indirect inverse probability and Bayesian posterior probabilities are direct forwards probabilities that do not condition on a hypothesis, and the Bayesian approach has not only been shown to be valid, but it actually delivers on its promise.
\n@online{harrell2017,\n author = {Harrell, Frank and Harrell, Frank},\n title = {Null {Hypothesis} {Significance} {Testing} {Never} {Worked}},\n date = {2017-01-14},\n url = {https://www.fharrell.com/post/nhst-never/},\n doi = {10.59350/rxzw7-6q428},\n langid = {en}\n}\nHarrell FE (2017). Classification vs.\u00a0Prediction. Statistical Thinking. https://doi.org/10.59350/17jxr-hf782
\nIt is important to distinguish prediction and classification. In many decisionmaking contexts, classification represents a premature decision, because classification combines prediction and decision making and usurps the decision maker in specifying costs of wrong decisions. The classification rule must be reformulated if costs/utilities or sampling criteria change. Predictions are separate from decisions and can be used by any decision maker.
Classification is best used with non-stochastic/deterministic outcomes that occur in say 0.3 - 0.7 of the observations, and not when the simplest classifer (always outputting \"positive\" or always outputting \"negative\") is highly accurate or when two individuals with identical inputs can easily have different outcomes. For these situations, modeling tendencies (i.e., probabilities) is key.
Classification should be used when outcomes are distinct and predictors are strong enough to provide, for all subjects, a probability near 1.0 for one of the outcomes.
The field of machine learning arose somewhat independently of the field of statistics. As a result, machine learning experts tend not to emphasize probabilistic thinking. Probabilistic thinking and understanding uncertainty and variation are hallmarks of statistics. By the way, one of the best books about probabilistic thinking is Nate Silver's The Signal and The Noise: Why So Many Predictions Fail But Some Don't. In the medical field, a classic paper is David Spiegelhalter's Probabilistic Prediction in Patient Management and Clinical Trials.
By not thinking probabilistically, machine learning advocates frequently utilize classifiers instead of using risk prediction models. The situation has gotten acute: many machine learning experts actually label logistic regression as a classification method (it is not). It is important to think about what classification really implies. Classification is in effect a decision. Optimum decisions require making full use of available data, developing predictions, and applying a loss/utility/cost function to make a decision that, for example, minimizes expected loss or maximizes expected utility. Different end users have different utility functions. In risk assessment this leads to their having different risk thresholds for action. Classification assumes that every user has the same utility function and that the utility function implied by the classification system is that utility function.
\nClassification is a forced choice. In marketing where the advertising budget is fixed, analysts generally know better than to try to classify a potential customer as someone to ignore or someone to spend resources on. Instead, they model probabilities and create a lift curve, whereby potential customers are sorted in decreasing order of estimated probability of purchasing a product. To get the \"biggest bang for the buck\", the marketer who can afford to advertise to n persons picks the n highest-probability customers as targets. This is rational, and classification is not needed here.
\nA frequent argument from data users, e.g., physicians, is that ultimately they need to make a binary decision, so binary classification is needed. This is simply not true. First of all, it is often the case that the best decision is \"no decision; get more data\" when the probability of disease is in the middle. In many other cases, the decision is revocable, e.g., the physician starts the patient on a drug at a lower dose and decides later whether to change the dose or the medication. In surgical therapy the decision to operate is irrevocable, but the choice of when to operate is up to the surgeon and the patient and depends on severity of disease and symptoms. At any rate, if binary classification is needed, it must be done at the point of care when all utilities are known, not in a data analysis.
\nWhen are forced choices appropriate? I think that one needs to consider whether the problem is mechanistic or stochastic/probabilistic. Machine learning advocates often want to apply methods made for the former to problems where biologic variation, sampling variability, and measurement errors exist. It may be best to apply classification techniques instead just to high signal:noise ratio situations such as those in which there there is a known gold standard and one can replicate the experiment and get almost the same result each time. An example is pattern recognition - visual, sound, chemical composition, etc. If one creates an optical character recognition algorithm, the algorithm can be trained by exposing it to any number of replicates of attempts to classify an image as the letters A, B, \u2026 The user of such a classifier may not have time to consider whether any of the classifications were \"close calls.\" And the signal:noise ratio is extremely high. In addition, there is a single \"right\" answer for each character. This situation is primarily mechanistic or non-stochastic. Contrast that with forecasting death or disease where two patients with identical known characteristics can easily have different outcomes.
\nWhen close calls are possible, or when there is inherent randomness to the outcomes, probability estimates are called for. One beauty of probabilities is that they are their own error measures. If the probability of disease is 0.1 and the current decision is not to treat the patient, the probability of this being an error is by definition 0.1. A probability of 0.4 may lead the physician to run another lab test or do a biopsy. When the signal:noise ratio is small, classification is usually not a good goal; there one must model tendencies, i.e., probabilities.
\nThe U.S. Weather Service has always phrased rain forecasts as probabilities. I do not want a classification of \"it will rain today.\" There is a slight loss/disutility of carrying an umbrella, and I want to be the one to make the tradeoff.
\nWhether engaging in credit risk scoring, weather forecasting, climate forecasting, marketing, diagnosis a patient's disease, or estimating a patient's prognosis, I do not want to use a classification method. I want risk estimates with credible intervals or confidence intervals. My opinion is that machine learning classifiers are best used in mechanistic high signal:noise ratio situations, and that probability models should be used in most other situations.
\nThis is related to a subtle point that has been lost on many analysts. Complex machine learning algorithms, which allow for complexities such as high-order interactions, require an enormous amount of data unless the signal:noise ratio is high, another reason for reserving some machine learning techniques for such situations. Regression models which capitalize on additivity assumptions (when they are true, and this is approximately true is much of the time) can yield accurate probability models without having massive datasets. And when the outcome variable being predicted has more than two levels, a single regression model fit can be used to obtain all kinds of interesting quantities, e.g., predicted mean, quantiles, exceedance probabilities, and instantaneous hazard rates.
\nA special problem with classifiers illustrates an important issue. Users of machine classifiers know that a highly imbalanced sample with regard to a binary outcome variable Y results in a strange classifier. For example, if the sample has 1000 diseased patients and 1,000,000 non-diseased patients, the best classifier may classify everyone as non-diseased; you will be correct 0.999 of the time. For this reason the odd practice of subsampling the controls is used in an attempt to balance the frequencies and get some variation that will lead to sensible looking classifiers (users of regression models would never exclude good data to get an answer). Then they have to, in some ill-defined way, construct the classifier to make up for biasing the sample. It is simply the case that a classifier trained to a 1/2 prevalence situation will not be applicable to a population with a 1/1000 prevalence. The classifier would have to be re-trained on the new sample, and the patterns detected may change greatly. Logistic regression on the other hand elegantly handles this situation by either (1) having as predictors the variables that made the prevalence so low, or (2) recalibrating the intercept (only) for another dataset with much higher prevalence. Classifiers' extreme dependence on prevalence may be enough to make some researchers always use probability estimators like logistic regression instead. One could go so far as to say that classifiers should not be used at all when there is little variation in the outcome variable (prevalence is near 0 or 1), and that only tendencies (probabilities) should be modeled.
\nOne of the key elements in choosing a method is having a sensitive accuracy scoring rule with the correct statistical properties. Experts in machine classification seldom have the background to understand this enormously important issue, and choosing an improper accuracy score such as proportion classified correctly will result in a bogus model. This is discussed in detail here.
\nThanks to Andrew Howe for suggesting some wording clarifications.
\n@online{harrell2017,\n author = {Harrell, Frank and Harrell, Frank},\n title = {Classification Vs. {Prediction}},\n date = {2017-01-15},\n url = {https://www.fharrell.com/post/classification/},\n doi = {10.59350/17jxr-hf782},\n langid = {en}\n}\nHarrell FE (2017). Ideas for Future Articles. Statistical Thinking. https://doi.org/10.59350/bk0rc-5jv57
\nSuggestions for future articles are welcomed as comments to this entry. Some topics I intend to write about are listed below.
A few weeks ago we had a small discussion at CrossValidated about the pros and cons of matching here. I am sorry that I did not had enough time to elaborate further on the support of matching procedures (in my field researchers do not focus much on a bias-variance tradeoff but they prioritize on minimizing biases. For that reason, they like matching procedures). Now, I have seen that you started a blog recently (congratulations!). I would like to encourage to take up the topic of matching because it is probably interesting for many applied researchers. I think in your 'philosophy', this would belong to the point \"Preserve all the information in the data\". Here, perhaps some input for a blog post. Back then, you wrote:
\nMatching on continuous variables results in an incomplete adjustment because the variables have to be binned.
\nWhat about propensity score matching?
\nMatching throws away good data from observations that would be good matches.
\nExtrapolation bias is only a significant problem if there is a covariate by group interaction, and users of matching methods ignore interactions anyway.
\nHere, you go too far (in my view). You can add interactions, again for example with propensity score matching. Imbens and Rubin (2015) suggest a procedure using quadratic and interaction terms of the covariates.
\nComment: Nice to know this exists but I've never seen a paper that used matching attempt to explore interactions. If you don't want to make regression assumptions that are unverifiable, remove observations outside the overlap region just as with matching.
\nWhich assumptions do you refer to? I think that treating everyone the same (statistically) is also an unverifiable assumption (do you disagree?). What is your opinion about weighted least squares?
\nComment: This is the no-interaction assumption. If you assume additivity then it's more OK to have a no-overlap region, otherwise throw-away non-overlap regions and do a conditional analysis. Not clear on the need for weighting here. In general I like conditioning over weighting.
\nArne Jonas Warnke
Labour Markets, Human Resources and Social Policy
Internet: zew.de
@online{harrell2017,\n author = {Harrell, Frank and Harrell, Frank},\n title = {Ideas for {Future} {Articles}},\n date = {2017-01-16},\n url = {https://www.fharrell.com/post/ideas-articles/},\n doi = {10.59350/bk0rc-5jv57},\n langid = {en}\n}\nHarrell FE (2017). Fundamental Principles of Statistics. Statistical Thinking. https://doi.org/10.59350/njpzb-qcd80
\nThere are many principles involved in the theory and practice of statistics, but here are the ones that guide my practice the most.
\u2026 the statistician must be instinctively and primarily a logician and a scientist in the broader sense, and only secondarily a user of the specialized statistical techniques.
In considering the refinements and modifications of the scientific method which particularly apply to the work of the statistician, the first point to be emphasized is that the statistician is always dealing with probabilities and degrees of uncertainty. He is, in effect, a Sherlock Holmes of figures, who must work mainly, or wholly, from circumstantial evidence.
Malcolm C Rorty: Statistics and the Scientific Method
JASA 26:1-10, 1931
The statistical method is more than an array of techniques. The statistical method is a mode of thought; it is sharpened thinking; it is power.
William Edwards Deming, International Statistical Institute Meeting, Sept 1953
Science is not about falsifying hypotheses. Science is about determining what component of a set of observations are reproducible, and then trying to determine the mechanism or rule to explain those observations\u2026 again, reasoning from data, not by starting with an axiom.
David J Glass
See this post for related thoughts.
\n\n@online{harrell2017,\n author = {Harrell, Frank and Harrell, Frank},\n title = {Fundamental {Principles} of {Statistics}},\n date = {2017-01-18},\n url = {https://www.fharrell.com/post/principles/},\n doi = {10.59350/njpzb-qcd80},\n langid = {en}\n}\nHarrell FE (2017). Split-Sample Model Validation. Statistical Thinking. https://doi.org/10.59350/hqg7h-8p324
\nMethods used to obtain unbiased estimates of future performance of statistical prediction models and classifiers include data splitting and resampling. The two most commonly used resampling methods are cross-validation and bootstrapping. To be as good as the bootstrap, about 100 repeats of 10-fold cross-validation are required.
As discussed in more detail in Section 5.3 of Regression Modeling Strategies Course Notes and the same section of the RMS book, data splitting is an unstable method for validating models or classifiers, especially when the number of subjects is less than about 20,000 (fewer if signal:noise ratio is high). This is because were you to split the data again, develop a new model on the training sample, and test it on the holdout sample, the results are likely to vary significantly. Data splitting requires a significantly larger sample size than resampling to work acceptably well. See also here.
\nThere are also very subtle problems:
\nThere is a less subtle problem but one that is ordinarily not addressed by investigators: unless both the training and test samples are huge, split-sample validation is not nearly as accurate as the bootstrap. See for example the section Studies of Methods Used in the Text here. As shown in a simulation appearing there, bootstrapping is typically more accurate than data splitting and cross-validation that does not use a large number of repeats. This is shown by estimating the \"true\" performance, e.g., the R2 or c-index on an infinitely large dataset (infinite here means 50,000 subjects for practical purposes). The performance of an accuracy estimate is taken as the mean squared error of the estimate against the model's performance in the 50,000 subjects.
\nData are too precious to not be used in model development/parameter estimation. Resampling methods allow the data to be used for both development and validation, and they do a good job in estimating the likely future performance of a model. Data splitting only has an advantage when the test sample is held by another researcher to ensure that the validation is unbiased.
\nMany investigators have been told that they must do an \"external\" validation, and they split the data by time or geographical location. They are sometimes surprised that the model developed in one country or time does not validate in another. They should not be; this is an indirect way of saying there are time or country effects. Far better would be to learn about and estimate time and location effects by including them in a unified model. Then rigorous internal validation using the bootstrap, accounting for time and location all along the way. The end result is a model that is useful for prediction at times and locations that were at least somewhat represented in the original dataset, but without assuming that time and location effects are nil.
\n@online{harrell2017,\n author = {Harrell, Frank and Harrell, Frank},\n title = {Split-Sample {Model} {Validation}},\n date = {2017-01-23},\n url = {https://www.fharrell.com/post/split-val/},\n doi = {10.59350/hqg7h-8p324},\n langid = {en}\n}\nHarrell FE (2017). Clinicians' Misunderstanding of Probabilities Makes Them Like Backwards Probabilities Such As Sensitivity, Specificity, and Type I Error. Statistical Thinking. https://doi.org/10.59350/td3jn-6cf74
\nOptimum decision making in the presence of uncertainty comes from probabilistic thinking. The relevant probabilities are of a predictive nature: P(the unknown given the known). Thresholds are not helpful and are completely dependent on the utility/cost/loss function.
Corollary: Since p-values are P(someone else's data are more extreme than mine if H0 is true) and we don't know whether H0 is true, it is a non-predictive probability that is not useful for decision making.
\nImagine watching a baseball game, seeing the batter get a hit, and hearing the announcer say \"The chance that the batter is left handed is now 0.2!\" No one would care. Baseball fans are interested in the chance that a batter will get a hit conditional on his being right handed (handedness being already known to the fan), the handedness of the pitcher, etc. Unless one is an archaeologist or medical examiner, the interest is in forward probabilities conditional on current and past states. We are interested in the probability of the unknown given the known and the probability of a future event given past and present conditions and events.
Clinicians are people trained in the science and practice of medicine, and most of them are very good at it. They are also very good at many aspects of research. But they are generally not taught probability, and this can limit their research skills. Many excellent clinicians even let their limitations in understanding probability make them believe that their clinical decision making is worse than it actually is. I have taught many clinicians who say \"I need a hard and fast rule so I know how to diagnose or treat patients. I need a hard cutoff on blood pressure, HbA1c, etc. so that I know what to do, and the fact that I either treat or not treat the patient means that I don't want to consider a probability of disease but desire a simple classification rule.\" This makes the clinician try to influence the statistician to use inefficient, arbitrary methods such as categorization, stratification, and matching.
\nIn reality, clinicians do not act that way when treating patients. They are smart enough to know that if a patient has cholesterol just over someone's arbitrary threshold they may not start statin therapy right away if the patient has no other risk factors (e.g., smoking) going against him. They know that sometimes you start a patient on a lower dose and see how she responds, or start one drug and try it for a while and then switch drugs if the efficacy is unacceptable or there is a significant side effect.
\nSo I emphasize the need to understand probabilities when I'm teaching clinicians. A probability is a self-contained summary of the current information, except for the patient's risk aversion and other utilities. Clinicians need to be comfortable with a probability of 0.5 meaning \"we don't know much\" and not requesting a classification of disease/normal that does nothing but cover up the problem. A classification does not account for gray zones or patient and physician utility functions.
\nEven physicians who understand the meaning of a probability are often not understanding conditioning. Conditioning is all important, and conditioning on different things massively changes the meaning of the probabilities being computed. Every physician I've known has been taught probabilistic medical diagnosis by first learning about sensitivity (sens) and specificity (spec). These are probabilities that are in backwards time- and information flow order. How did this happen? Sensitivity, specificity, and receiver operating characteristic curves were developed for radar and radio research in the military. It was a important to receive radio signals from distant aircraft, and to detect an incoming aircraft on radar. The ability to detect something that is really there is definitely important. In the 1950s, virologists appropriated these concepts to measure the performance of viral cultures. Virus needs to be detected when it's present, and not detected when it's not. Sensitivity is the probability of detecting a condition when it is truly present, and specificity is the probability of not detecting it when it is truly absent. One can see how these probabilities would be useful outside of virology and bacteriology when the samples are retrospective, as in a case-control studies. But I believe that clinicians and researchers would be better off if backward probabilities were not taught or were mentioned only to illustrate how not to think about a problem.
\nBut the way medical students are educated, they assume that sens and spec are what you first consider in a prospective cohort of patients! This gives the professor the opportunity of teaching Bayes' rule and requires the use of a supposedly unconditional probability known as prevalence which is actually not very well defined. The students plugs everything into Bayes' rule and fails to notice that several quantities cancel out. The result is the following: the proportion of patients with a positive test who have disease, and the proportion with a negative test who have disease. These are trivially calculated from the cohort data without knowing anything about sens, spec, and Bayes. Instead of computing the obvious, the previous backwards way of thinking harms the student's understanding for years to come and influences those who later engage in clinical and pharmaceutical research to believe that type I error and p-values are directly useful.
\nThe situation in medical diagnosis gets worse when referral bias (also called workup bias) is present. When certain types of patients do not get a final diagnosis, sens and spec are biased. For example, younger women with a negative test may not get the painful procedure that yields the final diagnosis. There are formulas that must be used to correct sens and spec. But wait! When Bayes' rule is used to obtain the probability of disease we needed in the first place, these corrections completely cancel out when the usual correction methods are used! Using forward probabilities in the first place means that one just conditions on age, sex, and result of the initial diagnostic test and no special methods other than (sometimes) logistic regression are required.
\nThere is an analogy to statistical testing. p-values and type I error are affected by sequential testing and a host of other factors, but forward-time probabilities (Bayesian posterior probabilities) are not. Posterior probabilities condition on what is known and does not have to imagine alternate paths to getting to what is known (as do sens and spec when workup bias exists). p-values and type I errors are backwards-information-flow measures, and clinical researchers and regulators come to believe that type I error is the error of interest. They also very frequently misinterpret p-values. Type I error is one minus spec, and power is sens. The posterior probability is exactly analogous to the probability of disease.
\nSens and spec are so pervasive in medicine, bioinformatics, and biomarker research that we don't question how silly they would be in other contexts. Do we dichotomize a response variable so that we can compute the probability that a patient is on treatment B given a \"positive\" response? On the contrary we want to know the full continuous distribution of the response given the assigned treatment. Again this represents forward probabilities.
\nSander Greenland: \"researchers and regulators\u2026 also very frequently misinterpret p-values.\" So do statisticians: \"The p-value is one minus spec, and power is sens.\" NO, that's just wrong. The Type-I error rate (capped by alpha) is 1-spec, and power = sens (so beta = Type-II error rate = 1-sens). These are properties of the test given the design and P is not the first one. P is random, not a property of the test or design; it is a property of the data in relation to the model from which it is computed. That means it varies from data set to data set. In contrast, error rates are calculated across a sequence of data sets.
\nHardly anyone interprets P-values correctly, it seems\u2026but is that that their fault or the fault of statisticians who confuse them with alpha levels or Type-I error rates (test size)?
\nFrank Harrell: Thanks for the correction Sander, although I could imagine a nicer way to state this. I'll correct the analogy in the post from p-value to type I error.
\nSander Greenland: Sorry for the jarring tone, I was reacting to the cottage industry of P-bashing. That industry would be fine except the more I examine each case of supposed P-badness, the more it looks like P is just being misinterpreted - sometimes in obviously wrong ways (as the posterior probability of a point hypothesis) and sometimes in not so obvious ways as in your passage. I should have said that I agree with most everything you said and thought the over-all point was great, so that's why this mistake was so jarring\u2026Well that and I've been battling P-bashers for years now on just these grounds. Not all are like you in terms of seeing and admitting a mistake. Consider the howler \"P-values overstate evidence\" - no they don't; people overinterpret P-values in terms of cutoffs and their meaning. Senn pointed out the confusion here 16 years ago.
\nFH: Thanks very much for that Sander. I agree with you. I have been sloppy in mixing p-values with type I error in a couple of other places so I appreciate the wake-up call.
\nFH: I think you got what I was trying to convey Martin. Diagnostic models require a cohort study or a non-oversampled cross-sectional study. You can also use an oversampled study such as case-control if you can apply a correction to the model's intercept (no other corrections usually needed). Yes you need a nomogram or a prediction calculator to get the baseline risk and then to update it by the (usually continuous) test result. The technology is there now. I agree with your interpretation of the figure, until you know more about action thresholds the clinician/patient use.
\nMartin: Dr.\u00a0Harrell, I read the chapter of your book and I think I'm getting the idea. Please correct me if I'm wrong:
\nWhat you propose is to condition on other clinical variables (which are theoretically a part of the clinical diagnostic process) to \"predict\" the post-test log odds of the disease. A good diagnostic test would add information to a pre-test model (you picture this as the absolute diagnostic yield).
\nAnd that would be the way of knowing which test to choose in a specific clinical setting, instead of sensitivity and specificity (and LRs, I guess).
\nThis model relies on an analytical cohort design. Most of diagnostic studies I've read about are based on a cross-sectional design. I wouldn't know if this can change your proposal, but it doesn't look like it does.
\nMy other concern is the translation into practice. I guess we would need the nomogram or clinical diagnostic rule (mobile app) for it to work. This can be cumbersome in some settings.
\nPS: How can one interpret figure 19.6? The test is most useful with pretest probs < 80%? I'm having a hard time interpreting this figure.
\nFH: Thanks for continuing this discussion because it will help me better understand the clinical issues. I should say that it's not they they are unimportant but that they are too indirect. Chapter 19 of BBR gives my suggestions. Briefly I want a partial effect plot for each predictor, whether it's a test output or a regular baseline variable. For a test output this is the contribution to the log odds of disease holding all the baseline variables constant. As shown in that chapter once you know the regression fit (odds ratio for a binary test) you can show its impact on absolute risk of disease as a function of baseline risk in general (regardless of what gave the patient a high risk, for example). For the rare case where the test is binary, the odds ratio is the product of the likelihood ratio positive and the likelihood ratio negative.
\nMartin: Thank you for your reply, Dr.\u00a0Harrell.
\nMaybe we're talking past each other, and since this is an important topic, I'll try my best to understand your point of view.
\nTo answer your question, in the context of a RCT it would not make any sense.
\nMy question to you is, if sensitivity and specificity are not important, what would be the criteria a physician should have to choose a diagnostic test for a given clinical situation?
\nFH: Nothing you discussed motivates the need for sensitivity and specificity. What I need is the effect of each piece of information, e.g., the effect of age, the effect of having a test result of 17.35 on the log odds of disease. Give me a very specific example where sens and spec were needed and probability of disease or probability of outcome did not do the job.
\nIf you were not diagnosing a patient but were interpreting RCT results would you take time to compute the probability that a patient was on treatment B if she improved her blood pressure?
\nSteve Pitts: I believe I understand your point of view about sn/sp vs probabilities. The main advantage of sensitivity and specificity is the intuitive one: Dr.\u00a0Yerushalmy was onto something with these adjectives. We all know people or pets that are \"sensitive\". It enables medical students to understand intuitively what is actually a pretty complicated subject. The main first order of business is to get them past the TV-show level of understanding, where a positive test means you have disease, and negative means you don't. Tests have properties that are more constant than individual posterior probabilities, whether the test is a clinical finding or an expensive machine.
\nFH: Thanks for the comments Steve. I think it is very appropriate to show people how to move past positive=disease, negative=no disease thinking. I just don't see the role an an indirect way to do this that is based on P(positive given disease) etc. I'd rather go for the jugular: P(disease given all test results and background variables). And beware of assuming that sensitivity and specificity are quite that constant.
\nMartin: Dr.\u00a0Harrell, diagnostic tests have different uses in clinical practice. They can confirm or discard a disease when there's suspicion of disease, discard a disease in an asymptomatic patient (screening), they also have prognostic capabilities, and they can help us decide which therapy is better.
\nNow, we cannot mix them up, these are different questions, although some tests can be both diagnostic and prognostic.
\nSensitivity and specificity help us to choose which test to order according to the clinical context. For example, an EKG could not be used to discard a myocardial infarction and a D-dimer could not be used to confirm pulmonary thromboembolism.
\nRegarding prevalence. It works as a starting point for prior probabilities, which take into account specific patient characteristics. This and the likelihood of the test result gives us the posterior probabilities. Since rarely are we ever 100% sure of the diagnosis, based on some specified (subjective) thresholds we decide to treat, discard, or take another test.
\nSo, I would have to disagree that sensitivity and specificity are not important in medical training.
\nFH: I disagree. The confusion caused by sens and spec is massive and even more clearly they take valuable class and study time away from the real need: how to specify probability models and design studies that will yield the data need to estimate their parameters. The only place I can thing of for sens and spec is when you don't have data, or at least the needed data. Do we ever teach physicians how to compute the probability that a patient is male given his disease status? Why is a test different from the patient's sex?
\nClyde Schechter: The only thing I disagree with here is your implication that clinicians should not be taught about sensitivity and specificity. It really is important to know those concepts, along with prevalence and Bayes' theorem. The reason is that some tests are used in different populations and settings. The same test may be used to mass screen asymptomatic populations, and also as a confirmatory diagnostic test in people with symptoms suggestive of the disease in question. In the former setting the prior probability of disease is typically very low, and in the latter it is usually moderate or even high. The sensitivity and specificity are, under reasonably broad circumstances, the same in both populations as they are generally physico-chemical properties of the test itself. The clinician needs to understand that, and why, the same test result has very different implications in these two uses.
\nIn fact, I think that this is not sufficiently emphasized in medical training. Most of the clinicians I deal with, even those relatively recently trained, are shocked to learn how low the positive predictive value of most screening tests are. But once they have mastered that, they also need to understand why the very same test can be very informative when applied to a patient who is likely to have the disease in question.
\ntherandomtexan: This merits a series of 4 or 5 lectures that tie all these ideas together and get out students past mere sensitivity, specificity, and the PPV. You've just nailed my foot the floor for a couple of days this summer, writing those lectures.
\n\n@online{harrell2017,\n author = {Harrell, Frank and Harrell, Frank},\n title = {Clinicians' {Misunderstanding} of {Probabilities} {Makes}\n {Them} {Like} {Backwards} {Probabilities} {Such} {As} {Sensitivity,}\n {Specificity,} and {Type} {I} {Error}},\n date = {2017-01-25},\n url = {https://www.fharrell.com/post/backwards-probs/},\n doi = {10.59350/td3jn-6cf74},\n langid = {en}\n}\n