Greetings everyone,
I’ve been thinking a lot about how I can provide more value in this newsletter, grow the audience, and educate on things I am interested in. I think covering more science-related topics is a good way to do so.
Now more than ever, there is a big emphasis on science and data as standards of truth and a compass for what is right. This entails belief in the scientific process, and more specifically, the scientific research industrial-complex. Scientific studies inform our understanding and our decisions. Thus, it is important to have the ability to understand what we are reading, critique it, and acknowledge its limitations.
Scientific research can be complex, and is often simplified for the masses. But the devil is in the details — anything touched by the hands of humans is liable to bias, tampering, and honest mistake. Thus, the results section of a paper should not be interpreted without sufficient background and assessment of the methodology.
Today, we will talk about one of the most cherished elements of quantitative research — the p-value.
There are two ways to conquer and enslave a nation. One is by the sword, and the other is by debt.
—John Adams (1735-1826)
✏️ The P-roblem with p-values
P<0.05. This is considered the threshold between something no better than what we’ve already got, and something that should be funded, supported, and brought to the market and to the people. However, the p-value is merely the result of a statistical test. It is not in itself a statement of the value of one treatment over another. Understanding what the p-value is is critical to interpreting scientific research.
In short, the p-value is the probability of observing one’s data when the null hypothesis is true (in other words, no difference actually exists and the observed data is misleading). The p-value has limitations: in particular, with respect to the information it can provide and the influence of inputs. For example, a tiny difference between two treatments can lead to a statistically significant difference if there is a large enough sample size. This is because, as the sample size increases, the precision of the estimates increases. With enough of a sample size, even a 0.1% difference in risk reduction can have a p-value less than 0.05. Conversely, if you have a study with a small sample size, a statistically significant difference may not be observed despite one treatment showing a large difference in estimated effect size vs. the comparator. Long story short, the p-value does not take into account effect size/magnitude. This is particularly important in clinical and health policy decision-making, as spending 10x more money to fund a new treatment that provides minuscule benefit over the existing treatment is not very cost-effective.
The importance of obtaining statistically significant results can lead researchers to go ‘fishing’ for p-values. Researchers can be tempted to perform multiple experiments or tests in order to get significant results that look better in journal publications. This can introduce ethical problems into the scientific research process, as studies can be designed or conducted in a way that promotes a given result.
There’s a lot more to say about p-values and I am no expert on the topic. But as someone who works in health research, I have accepted that p-values are not even close to the full story, and should be taken with their fair grain of salt. They must be interpreted alongside the characteristics and methodological design of the study, the magnitude of difference between groups, whether this difference is clinically relevant, and the potential sources of bias in the study.
💡 Food for Thought
Palma non sine pulvere — no reward without effort
🔗 Sunday Best
Researchers & Founders
A blogpost by Sam Altman, CEO of OpenAI
I spent many years working with founders and now I work with researchers.
Although there are always individual exceptions, on average it’s surprising to me how different the best people in these groups are (including in some qualities that I had assumed were present in great people everywhere, like very high levels of self-belief).
An Introvert’s Field Guide to Friendship
Maria Popova reflecting on Henry David Thoreau and his words on the challenges and rewards of the art of connection.
And yet this openhearted longing is itself the only real raw material of friendship — only by surrendering to it, with all the vulnerability this demands of us, do we become receptive to the longing of others, the mutual yearning for connection that is shared heartbeat of humanity.
The Wisdom of Lamentation
On silence and speech in the Book of Job, by Stephen Gregg
Our suffering can always be a kind of speech, whether aloud or not, and this is what is so beautiful about human beings. When we encounter those who suffer, we do not just try to teach them a way out or simply console them, but we love them, want to hear from them, want them to share with us the pain of their souls, because that pain is a uniquely human treasure and the sign of the heart’s capacity for feeling expansion, a defiance of all spiritual dullness and numbness.
A work of art is really above all an adventure of the mind.
—Eugene Ionesco (1909-1994)
Bon dimanche,
AT