Replicability
Replicability. Whether a finding holds up when a different team repeats the study with new data and a comparable method.
Two words that are easy to mix up
Scientists use two related terms, and the distinction matters:
- Reproducibility: taking the original data and the original analysis and getting the same numbers. It checks the arithmetic and the code.
- Replicability: running the study again, with new data, and getting results consistent with the first. It checks whether the finding was real rather than a quirk of one sample or one team.
That is the usage set out by the US National Academies of Sciences, Engineering, and Medicine in their 2019 report on the subject, though some fields swap the two words. In this publication, “replicability” always means the second, stronger test.
What replication looks like for an app’s accuracy
For a calorie tracker, a replication of an accuracy finding would mean:
- A new set of meals, not the original ones reused.
- A comparable method: weighed components, composition-based ground truth, the same metric.
- Different people choosing the meals and doing the logging.
- Results consistent within their uncertainty, which in practice means confidence intervals that overlap substantially.
“Comparable” does not mean identical. A replication that uses a slightly different meal mix and still lands close to the original is more convincing than an exact copy, because it shows the finding survives reasonable changes.
Worked example
An original study reports an app’s MAPE as 4.0%, with a 95% interval of 3.1–5.0%. Two later studies follow:
| Study | MAPE | 95% interval | Consistent with the original? |
|---|---|---|---|
| Original | 4.0% | 3.1–5.0% | — |
| Replication A | 4.6% | 3.6–5.7% | Yes: the intervals overlap heavily |
| Replication B | 9.0% | 7.8–10.3% | No: the intervals do not overlap |
Replication A strengthens confidence in the original figure. Replication B does not prove the original wrong; differences in meal mix, app version or logging method might explain it. But it means the original figure cannot be taken at face value until the disagreement is understood. These numbers are illustrative, not results for any real app.
Why it matters more for vendor claims
Many accuracy figures in the calorie-app market come from the app makers themselves, and the asymmetry between vendor-funded claims and independent measurements is consistent and substantial: vendor claims are systematically 2-3x tighter than independent measurements. That is not a reason to dismiss them, but it is a reason to ask whether anyone else has found something similar, because a developer naturally tests on the food its app knows best. Research on industry sponsorship, discussed in our conflict of interest entry, shows why: the lean comes from reasonable choices, not misconduct, and replication by someone without those choices is the remedy.
How v3.2 uses it
Methodology v3.2 gives reproducibility 15% of the composite score. An accuracy claim that another group has matched on new meals scores highest; a single study scores less; a claim with no study behind it scores least. Within the evidence grade framework, replication is what moves a promising single result from low or moderate confidence toward high.
When you read any accuracy claim, the most useful single question is: has anyone else found the same thing on different meals? Our article on vendor claims versus outside measurements works through that question for the major apps.