Why S-values (surprisals) Instead of P-values?

The other day, a colleague asked me an important question. When the hypothesis under consideration is not the null hypothesis of no effect (e.g., the difference between two mean values is 50 and not 0), a P-value equal to 1 indicates that—assuming chance is the only factor at play—we would obtain a "result" greater than or equal to the observed one 100% of the time in numerous future equivalent experiments.

So, how can chance - assuming it operates alone - always generate a greater "result" than the observed experimental one?

The apparent problem is easily solved by noting that the so-called "result" (or, better, the "statistical result") does not refer to the magnitude of the effect under examination but to the value of the test statistic. If we fix the hypothesis "difference = Δ = 50" and observe a difference of 50, then the test statistic (i.e., the statistical result) turns out to be 0! And what is the probability of obtaining a test statistic as or more extreme than 0 assuming chance is operating alone? Obviously, 100% (every real number satisfies such a request). That's the meaning of p = 1.



Surprisal, I love u <3

If we adopt surprisals, this problem does not arise: Indeed, the S-value represents the number s of consecutive heads we would need to obtain—by fairly tossing a coin s times—to match the statistical information entailed by the result. In this case, the researcher is perfectly aware that the coin toss is just a point of comparison and has nothing to do with the investigated scientific phenomenon. Specifically, the S-value tells us how surprised we should feel when observing an experimental outcome compared to the prediction of the hypothesis under consideration. For instance, s = 4 implies that we should be as surprised as when we get 4 consecutive heads in 4 fair coin tosses. In the case of p = 1, the corresponding s = 0 (since p = (1/2)^s, i.e., s = -log_2 p) tells us that we should not feel surprised at the observed result compared to the fixed hypothesis (as they are in perfect agreement).

No, the S-value is not a hippie-naive version of the P-value

P-values present numerous interpretive difficulties beyond the one described above. For example, two pairs of probabilities (90%, 95%) and (5%, 10%) might seem equally "distant" from each other since 95-90=10-5=5. Yet, the first two events are almost equally probable (in fact, 0.95/0.90 = 1.1) while the second two are not (0.10/0.05 = 2, i.e., one is twice as likely as the other). Nonetheless, even the probability ratio has its problems. For instance, an event with a probability of 0.04 (4%) has an expected frequency 4 times higher than an event with a probability of 0.01 (1%); However, when compared to the number of consecutive heads when fairly flipping a coin, the difference in "rarity" between these two is equal to the rarity of the event "getting 2 consecutive heads when fairly flipping a coin 2 times" (since 1/2·1/2 = 1/4 or, equivalently, log_2 (0.04/0.01) = 2). In other words, the difference between the two events under consideration is generally not substantial in probabilistic terms. Therefore, S-values describe statistical information in a more balanced way than P-values. As a 'side' note, remember that all this is conditional on the validity of the statistical model (which needs to be checked)!

And what is information?

Sander Greenland, Valentin Amrhein, and Blake McShane [thank you for your patience and your help] have helped me a lot in understanding the essential importance of this concept in statistics. However, although (I believe and hope) everything is clear to me from an operational standpoint, the definition of 'information' still eludes me. In this regard, I believe there is no universally valid way to quantify information as this is intrinsically tied to our perception (decoding of content). In this specific context, we could say that the fact that statistical information is 'log-scaled' is the mathematical way of saying that the communicability of its content generally improves due to this transformation (although there are also solid mathematical reasons). For some practical examples of why this is the case, you can refer to our very recent article, section "A less innovative but still useful approach" [again, thank you Sander, thank you Valentin, and thank you Blake] [1].


References

1. Rovetta, A., Mansournia, M. A., & Vitale, A. (2024). For a proper use of frequentist inferential statistics in public health. Global Epidemiology, 100151. https://doi.org/10.1016/j.gloepi.2024.100151

Comments

Post a Comment

Popular posts from this blog

How to Interpret (Frequentist) Statistical Estimates in Medical Research

Image
[This is the English translation of an article originally published on SIAF Community, the Italian scientific platform for forensic and insurance medicine] Statistical methods are used in medical research to estimate the effects of treatments or health conditions in populations. For example, when testing the effectiveness of a new treatment that has shown very positive results in preclinical animal studies, researchers draw from the target clinical population a small group of patients - called a statistical sample . The ideal goal is to administer the treatment to the patients in the sample to estimate its effectiveness in the entire population. The process by which the effect observed in the sample is “transported” to the wider population is called inference . However, even when the best inferential methods available are used, a perfect transport is factually impossible . The reasons range from the many uncertainties involved in conducting a clinical study to the variability of cond...
Read more

Reification: “The data speak for themselves” only if we confuse mathematics with reality

Image
[This is the English translation of an  article  originally published on SIAF Community, the Italian scientific platform for forensic and insurance medicine] Suppose we flip a coin and ask what the probability of getting heads is. Our experience and what we learned in school suggest that the answer is 50%, since there is one “favorable” case (heads) out of two possible outcomes (heads or tails). This is a very common form of reification : confusing a theoretical abstraction - in this case, the probabilistic model that assigns a 50% chance to heads - with an empirical property of the real world, as if that 50% were an intrinsic characteristic of the coin or of the physical act of flipping it. Contrary to what one might think, this almost automatic answer already assumes a very specific statistical model, based on at least two main assumptions about the process that generates the outcomes (where an outcome is obtaining 'heads' or 'tails'). Assumption of the competent man...
Read more

On the various definitions of P-value

Reading one of the many wonderful insights by Professor Andrew Gelman on the Columbia University blog (see https://statmodeling.stat.columbia.edu/2023/04/14/4-different-meanings-of-p-value-and-how-my-thinking-has-changed ), I have developed a series of thoughts that I believe - based on my current knowledge and abilities - might be potentially useful. Original comments, part 1 Andrew Gelman: << Definition 1. p-value(y) = Pr(T(y_rep) >= T(y) | H), where H is a “hypothesis,” a generative probability model, y is the observed data, y_rep are future data under the model, and T is a “test statistic,” some pre-specified specified function of data. [...] >> April 14, 2023 9:14 AM Christian Hennig: << By the way, on misinterpretation that bugs me and seems to come up often is the idea that there is a “true” unobserved p-value potentially different from the observed one in case the model doesn’t hold. Not so. The p-value measures the relation between the data and a specifie...
Read more