Evidence Grade
Evidence Grade. A rating of how much confidence a body of evidence deserves, from high to very low, based on design, bias, consistency, directness and precision.
Confidence, not truth
An evidence grade does not say whether a claim is true. It says how confident you can be in it given the evidence available. A claim graded “low” may well be correct; it simply rests on evidence that could easily be overturned by the next study.
The most widely used system is GRADE, set out by the GRADE Working Group in the BMJ in 2004 and since adopted by guideline bodies around the world. It uses four levels:
- High: further research is very unlikely to change our confidence in the estimate.
- Moderate: further research is likely to have an important effect on our confidence and may change the estimate.
- Low: further research is very likely to change the estimate.
- Very low: any estimate is very uncertain.
What moves a grade up or down
GRADE looks at the type of study first and then adjusts. The reasons to rate evidence down are:
- Risk of bias: flaws in how the study was designed or run, including who controlled it.
- Inconsistency: different studies of the same question disagree.
- Indirectness: the study answers a slightly different question from the one you have.
- Imprecision: too few observations, or a wide confidence interval.
- Publication bias: reason to think unfavourable results went unpublished.
Adapting it to calorie apps
GRADE was built for treatment questions, “does this drug work?”, rather than measurement questions, “how close is this app to the truth?”. Our framework keeps its structure and changes the content of each domain:
| GRADE domain | What it means for a calorie-app study |
|---|---|
| Study design | A weighed-meal benchmark is the strongest design; a database comparison is indirect for consumer use |
| Risk of bias | Funding, author ties and who chose the meals; see conflict of interest |
| Inconsistency | Whether other groups’ results agree, the replicability question |
| Indirectness | Meals unlike real diets, researcher-entered data, a different app version |
| Imprecision | Number of meals and the width of the interval around the error figure |
| Reporting | Whether the protocol, meal list and metric are described well enough to rerun |
The overall grade is capped by the weakest important domain: a beautifully designed study with a tiny sample cannot be “high”.
Worked example
Take a hypothetical accuracy study of one calorie app: 40 weighed meals, run by the app’s developer, meals chosen by the developer’s team, MAPE reported without a confidence interval, no other study of the same app.
- Design: weighed-meal benchmark, so it starts high.
- Risk of bias: developer-run with developer-chosen meals. Down one level, to moderate.
- Imprecision: 40 meals and no interval. Down one more, to low.
- Inconsistency: nothing to compare with yet. No change, but nothing to support it either.
Result: low. The figure may be right, but a second study could easily move it. If an outside group later ran 150 meals and reported a similar figure with a tight interval, the combined evidence would rise toward moderate or high.
Why grades matter to an ordinary reader
When two apps quote similar accuracy figures, the grade behind each figure is what separates them. A percentage on a product page with no study behind it would be very low; the same percentage from a large, well-reported benchmark is a different kind of claim. Our validation study entry explains what a study needs to include for a grade to be possible at all, and our evidence map applies the approach to the published record.