Statistical Thoughts

The other day, a colleague asked me an important question. When the hypothesis under consideration is not the null hypothesis of no effect (e.g., the difference between two mean values is 50 and not 0), a P-value equal to 1 indicates that—assuming chance is the only factor at play—we would obtain a "result" greater than or equal to the observed one 100% of the time in numerous future equivalent experiments. So, how can chance - assuming it operates alone - always generate a greater "result" than the observed experimental one? The apparent problem is easily solved by noting that the so-called "result" (or, better, the "statistical result") does not refer to the magnitude of the effect under examination but to the value of the test statistic. If we fix the hypothesis "difference = Δ = 50" and observe a difference of 50, then the test statistic (i.e., the statistical result) turns out to be 0! And what is the probability of obtaining a t...

Reading one of the many wonderful insights by Professor Andrew Gelman on the Columbia University blog (see https://statmodeling.stat.columbia.edu/2023/04/14/4-different-meanings-of-p-value-and-how-my-thinking-has-changed ), I have developed a series of thoughts that I believe - based on my current knowledge and abilities - might be potentially useful. Original comments, part 1 Andrew Gelman: << Definition 1. p-value(y) = Pr(T(y_rep) >= T(y) | H), where H is a “hypothesis,” a generative probability model, y is the observed data, y_rep are future data under the model, and T is a “test statistic,” some pre-specified specified function of data. [...] >> April 14, 2023 9:14 AM Christian Hennig: << By the way, on misinterpretation that bugs me and seems to come up often is the idea that there is a “true” unobserved p-value potentially different from the observed one in case the model doesn’t hold. Not so. The p-value measures the relation between the data and a specifie...