Validation Study
Validation Study. A study that checks a measuring tool's readings against a trusted reference to find out how close, and how consistent, the tool is.
The question a validation study answers
Every measuring tool, from a bathroom scale to a food-frequency questionnaire, has to earn trust. A validation study is how it does so: you measure the same things with the tool and with a reference you trust more, and you describe how far apart they are. For a calorie-tracking app, the tool is the app and the reference is a meal whose true calories are known.
What every validation study needs
- A reference standard. The ground truth the tool is compared against, and an account of how accurate that reference is.
- A sample. What was measured, how many, and whether it resembles what real users measure.
- A protocol. Exactly how the tool was used, by whom, and when, so the study can be repeated.
- Metrics. How the differences are summarised, such as MAPE, MAE or limits of agreement, ideally with confidence intervals.
- A disclosure. Who funded the work and who chose the sample.
Leave any of these out and a reader cannot judge the result.
Absolute and relative validity
Nutrition researchers distinguish two kinds of validation. Absolute validity compares a tool with the truth, for example an app’s estimate against weighed meals. Relative validity compares one imperfect method with another that is considered better, for example an app’s nutrient totals against a research database for the same recorded diet.
Most published studies of consumer calorie apps are of the relative kind. Griffiths and colleagues in 2018 and Evenepoel and colleagues in 2020, for instance, compared app nutrient totals with research-grade databases for diets that had been recorded or recalled. Studies like these tell you how good an app’s food records are; they say less about the error a consumer experiences when choosing entries and judging portions. Our evidence map sorts the published record by kind.
Worked example: a mini validation you can run at home
You can get a rough sense of how well an app works for your own food with a kitchen scale and an evening.
- Pick ten meals you eat often, including a couple of mixed dishes.
- Weigh every component before cooking or serving and work out each meal’s calories from the package labels or a composition database. This is your reference.
- Log each meal in the app the way you normally would, without looking at your reference first.
- For each meal, divide the difference by the reference value. Average the ten results.
If your average comes out at 12%, the app is typically about 12% off on your meals. Ten meals is far too few for a firm conclusion, and your reference has its own small errors, but it will show you where an app struggles with the food you actually eat. Repeat with another app on the same ten meals and you have a small comparison of your own.
How to tell a strong study from a weak one
A strong validation uses many meals of mixed difficulty, a well-documented reference, a protocol fixed in advance, intervals around every figure, and authors without a stake in the outcome, or at least an open disclosure of one. A weak one uses a handful of easy foods, reports a single number, and is run by the people selling the product. Both may report the same percentage. Only one of them tells you much. Across apps where both kinds have been published, vendor-funded claims are systematically 2-3x tighter than independent measurements. The replicability entry explains what happens next: whether a second team, on new meals, finds the same thing.