Methodology

5. Honest limitations

We publish our limitations because hidden weaknesses are more dangerous than visible ones:

• Evidentiary base (not the corpus size): The corpus holds 3,669 data points across the 15 published side effects, but only 2612 of them state a rate, and after our eligibility rules 1961 rates from 89 distinct studies are allowed to support a published estimate. Excluded: 150 seed unverified, 66 spontaneous report share, 14 aggregator result, 421 placebo comparator arm. Rates are collapsed to one entry per source per effect (614 entries in total); since 2026-09-08 the site-wide count is keyed by study URL, so one registry record or paper counts once however many effects it reports. Many effects therefore rest on a single source — which is stated next to each estimate. (Correction 2026-08-31: the April-2026 seeding pass turned out to have also written rows restating our own static literature table under real trial/study URLs that no extraction ever read — rows a URL-based screen cannot catch. All 150 seed rows are now labelled and excluded; the eligible base roughly halved as a result, and five effects fell back to a labelled literature figure. Base figures published before this date should not be cited.)
• FAERS is not incidence: FAERS shares are reported separately and never averaged into an incidence estimate. Note what the denominator is: our collector sums the counts of a drug's 30 most-reported reaction terms and divides each term's count by that sum, so the published share is a share of term-mentions — not of reports and not of patients (a report naming three reactions counts under all three, and any term outside the top 30 is in neither the numerator nor the denominator).
• Reviews do not count as extra studies: A systematic review or network meta-analysis that restates rates from trials we already pool must not carry its own pooling weight — otherwise the review's already-pooled sample size (thousands to tens of thousands of participants across many trials) would outweigh the individual registry records it draws from. Any source whose title or abstract describes itself as a systematic review or (network) meta-analysis is therefore capped to a pooling weight of 1, the same floor a single community post gets. Checked 2026-09-18: every source this rule currently affects is itself a self-described review (0 primary trials incorrectly capped); this is text detection, not a guarantee, so it stays an acknowledged residual risk.
• Populations and counting windows are mixed: A pooled clinical rate combines trials of every indication among the trials we pool — obesity, type 2 diabetes, cardiovascular outcomes, and type 1 diabetes on background insulin — and each trial counts an event at least once within its own window, which runs from days to years. The window changes the number: short Phase 1 studies supply many rows but little weight, and most of the weight comes from trials of a year or longer, so read a pooled rate as a share over roughly one to several years. The indication and window mix behind each estimate is published per effect in /api/data (indicationMix, windowMix); we re-weight by neither, because no evidence base for such a weighting exists. Read the weight, not the row count: seven cardiovascular-outcomes trials (participants with type 2 diabetes and established cardiovascular disease or high cardiovascular risk or, in SELECT, established cardiovascular disease with overweight or obesity and without diabetes) carry 63% of nausea's pooled weight on 2 October 2026, obesity trials 15%.
• Below 5%, trials go missing: Most of the registry trials we pool list a non-serious adverse event only when it exceeded 5% in at least one arm. A trial in which a term stayed at or under 5% in every arm contributes no row for it. For effects pooled near or below 5%, the base can therefore mostly include only trials where the term was listed: the count of trials is a floor on how many trials saw the effect, and the pooled rate is likely higher than it would be across all trials. Serious adverse-event tables carry no threshold. Rows per threshold are published per effect in /api/data (thresholdMix).
• Demographic gaps: Female bias (81% of the 151 data points that specify a sex), minimal ethnic diversity in sources. Only 151 of 3,884 data points state a sex at all.
• Modifier sources: Every modifier currently applied was hand-coded at the 2026-04-12 seed and no per-modifier citation is recorded for any of them, so treat each odds ratio as an expert-coded prior rather than a sourced estimate — the predictor now prints the modifiers it applied, and their provenance, beside every adjusted figure. Zero have so far been replaced by empirically estimated values, because no effect reaches the threshold of 10 citable stated rates. Target architecture: inverse-variance meta-analytic framework with formal study quality assessment (Cochrane Risk of Bias).
• No interaction terms: Modifiers are applied independently. Interactions (e.g., sex × age) are not modeled — we cap cumulative log-odds shifts to limit stacking bias.
• Calibration testing: Not yet formally validated against independent outcome data. Planned once n ≥ 500 per effect.
• LLM extraction: Imperfect. Gold-standard manual audit is planned on a sample of 50 sources per effect.
• Journey predictor: Since 2026-09-12 the weight-trajectory model takes its total loss from the ClinicalTrials.gov arm matching the selected molecule, dose tier and diabetes status (semaglutide: STEP; tirzepatide: NCT04184622/NCT04657003; liraglutide: NCT01272219/NCT01272232, with lower tiers scaled via NCT00422058); the shape of the curve and every age, sex, exercise and BMI modifier remain calibrated on semaglutide STEP data for all molecules. The muscle-preservation and after-stopping models use STEP trial constants with expert-coded modifiers throughout. None of the three is empirically validated.
• Not causal: These are population-average conditional risks, not individual causal predictions.