When a Psilocybin Result Changes After Statistical Adjustment

A positive trial result is rarely a single number produced by one unavoidable calculation. Researchers must decide how to account for starting symptom severity, repeated measurements, missed visits, and participants who depart from the protocol. These decisions can be reasonable and still change the estimate. The useful question is not whether statistics were used, but whether the conclusion survives plausible alternatives.

This article examines that narrow part of evidence appraisal: robustness. It complements the site's overview of the depression trial literature, rather than rating psilocybin as a whole. A robust result is one that does not depend entirely on a fragile analytical choice. Robustness does not remove other limitations, such as imperfect masking or selective enrollment.

Start with the analysis population

A randomized trial assigns participants to groups before outcomes are known. An analysis that preserves those assignments protects an important advantage of randomization. But papers can report several populations: everyone randomized, everyone treated, and people meeting additional protocol requirements. Those groups answer related but different questions. Restricting an analysis after randomization may make treatment delivery look cleaner while weakening the protection against selection bias.

The participant-flow diagram and analysis descriptions in the Raison trial report distinguish efficacy and safety populations and explain protocol-related exclusions. This is a practical place to learn the vocabulary. Readers should identify which participants contributed to the primary estimate before interpreting its size. A safety analysis organized around treatment actually received can be appropriate even when efficacy is analyzed by assignment. [1]

Adjustment is not automatically manipulation

Suppose one randomized group starts with slightly more severe depression. Adjusting for baseline severity can improve precision and account for that imbalance. Multicenter studies may also account for study location. The key distinction is whether these choices were specified before investigators inspected the outcomes, rather than selected afterward because they produced a more appealing result. Adjustment itself is not evidence of either misconduct or reliability.

Raison and colleagues describe a repeated-measures model with baseline and other specified factors. Such models use information from multiple visits, but they still depend on assumptions about the data-generating process. Their output is an estimated group difference under that model, not a direct observation applying identically to each participant. A technical appendix is therefore part of the evidence, not merely optional statistical decoration. [1]

Missing observations are a substantive issue

Someone may miss a visit because they improved and returned to work, because they became unwell, or because travel became impossible. Those explanations imply different relationships between missingness and outcome. A model that treats missing observations as explainable by recorded information may perform poorly if unrecorded deterioration is the main reason people disappear. Reporting only the number missing does not settle that concern.

Ask how many measurements are missing in each group, when they became missing, and what reasons were recorded. Then ask whether the researchers tested a less favorable scenario. If a conclusion holds only when every missing participant is assumed to resemble the observed participants, confidence should be qualified. Conversely, a small amount of missing data with consistent sensitivity analyses is more reassuring than an unexplained complete-case result.

Separate the planned test from exploratory findings

The Goodwin trial assigned 233 adults with treatment-resistant depression to three dose groups and specified a depression-scale outcome at three weeks. Its headline finding must be anchored to that comparison and time point. Additional analyses of sustained response, subgroups, or other visits can help explain the result, but they should not silently replace the original question. [2]

A common mistake is to search a paper for whichever outcome gives the smallest probability value. With many outcomes and many possible subgroup divisions, an apparently striking result can occur by chance. Prespecification and multiplicity procedures help distinguish a planned confirmatory test from a hypothesis worth testing later. Neither a sophisticated model nor an impressive graph changes an exploratory analysis into an independent replication.

Read uncertainty alongside the estimate

A confidence interval shows the precision of an estimate under the analysis assumptions. A narrow interval is not protection against systematic bias, and a wide interval should discourage precise claims about how much better one intervention is. Statistical significance addresses compatibility with a particular null hypothesis; it does not by itself establish a clinically important benefit or applicability to an individual reader.

For a concrete reading exercise, write down the primary group difference, its interval, the analysis population, and the missing-data approach. Next, record whether the direction and practical size remain similar in the sensitivity analyses. Finally, list design limitations that no statistical adjustment resolves. The site's guide to trial representativeness explains why even a stable estimate may apply to a narrow population.

A fair conclusion can be conditional

It is reasonable to say that a trial supports improvement under its specified analysis while uncertainty remains about alternative assumptions. That is more informative than calling every adjusted result unreliable or treating a significant result as unassailable. Statistical appraisal works best when it explains exactly what would need to change for the conclusion to weaken. Readers can then distinguish genuine fragility from ordinary, transparently reported analytical choices.

Disclosure: It provides research education, not medical advice or a recommendation to use psilocybin.

Editorial disclosure: Generated pictures are illustrative, not study participants. Consult a qualified professional for personal care.