Surprising Findings: NutriGrading the evidence - Does it stack up?
As a new PEN Evidence Analyst, there is always more to learn about the how to critically review evidence. I was recently updating a practice question when I came across an assessment tool to evaluate the quality of evidence of meta-analyses that I was not familiar with called NutriGrade. While I was aware of GRADE, I was surprised to read about NutriGrade and was intrigued to learn more!
NutriGrade is an adaptation of the GRADE system modified for the field of nutrition research (1). While many of the assessment criteria between the two tools are the same, NutriGrade has developed criteria intended to better judge the unique features and outcomes of nutrition studies not addressed in GRADE such as the: methodological challenges inherent in nutrition study designs (e.g. lack of blinding, having a true placebo and crossover bias), dietary assessment methods and tools and their validation (e.g. dietary records, food frequency questionnaires), diet-associated surrogate biomarkers and funding bias. Cohort study designs, which NutriGrade developers felt were undervalued and underscored in GRADE, are weighted higher. After a lot of testing and refining, the developers ended up with a seven-item tool for RCTs and an eight-item tool for cohort studies that could assess and score the quality of evidence from meta-analyses. Items in NutriGrade include risk of bias, study quality and study limitations, precision, heterogeneity, directness, publication bias, funding bias, study design, effect size and dose-response. For each item, well-defined criteria were developed.
Another unique feature of NutriGrade is that a 10-point scoring system to quantify and grade the quality of evidence was developed, which is then further divided into four categories of confidence similar to GRADE (1). A grading score of 8-10, reflects a “high” level of confidence; 6 to 7.99 suggests a “moderate” level, 4 to 5.99 is a low confidence and 0 to 3.99 reflects a very low level of confidence in the certainty of the evidence. Internal validity was tested and excellent agreement and reliability between independent raters and their overall NutriGrade scores was found when 30 meta-analyses were compared.
In terms of reliability, when GRADE and NutriGrade were compared, good reliability between the two tools for RCTs was found, but NutriGrade ranked the same cohort studies higher, which is not surprising given the additional weight that NutriGrade gives to cohort studies (1,2). Since publication of this work, it is interesting to note that GRADE has integrated an approach that permits better comparison of evidence from RCTs and nonrandomized studies (3).
My investigation of these two approaches brought me to my second surprising finding. While the name implies that NutriGrade is like a cousin to GRADE, it has not received the same “Gold Star” recognition. It has not been endorsed by GRADE and there is hefty academic discourse on the merit and need for NutriGrade. While the developers suggest that NutriGrade better assesses the unique feature of nutrition-based studies (1,4), the proponents of GRADE argue that it has been used successfully to do so (2). As I continue my learning in the field of evidence-based research, I will be curious to learn how this debate proceeds and what the future of NutriGrade will be.
Written by:
Christine Mehling MSc, RD
PEN Evidence Analyst
References
- Schwingshackl L, Knüppel S, Schwedhelm C, Hoffmann G, Missbach B, Stelmach-Mardas M, ert al. Perspective: NutriGrade: A Scoring System to Assess and Judge the Meta-Evidence of Randomized Controlled Trials and Cohort Studies in Nutrition Research. Adv Nutr. 2016 Nov 15;7(6):994-1004. doi: 10.3945/an.116.013052. Erratum in: Adv Nutr. 2017 Mar 15;8(2):399. PMID: 28140319; PMCID: PMC5105044. Abstract available from https://pubmed.ncbi.nlm.nih.gov/28140319/
- Meerpohl JJ, Naude CE, Garner P, Mustafa RA, Schünemann HJ. Comment on "Perspective: NutriGrade: A Scoring System to Assess and Judge the Meta-Evidence of Randomized Controlled Trials and Cohort Studies in Nutrition Research". Adv Nutr. 2017 Sep 15;8(5):789-790. doi: 10.3945/an.117.016188. PMID: 28916579; PMCID: PMC5593107. Abstract available from: https://pubmed.ncbi.nlm.nih.gov/28916579/
- Schünemann HJ, Cuello C, Akl EA, Mustafa RA, Meerpohl JJ, Thayer K, et al; GRADE Working Group. GRADE guidelines: 18. How ROBINS-I and other tools to assess risk of bias in nonrandomized studies should be used to rate the certainty of a body of evidence. J Clin Epidemiol. 2019 Jul;111:105-14. doi: 10.1016/j.jclinepi.2018.01.012. Epub 2018 Feb 9. PMID: 29432858; PMCID: PMC6692166. Abstract available from: https://pubmed.ncbi.nlm.nih.gov/29432858/
- Ali Mohsenpour M, Fallah-Moshkani R, Ghiasvand R, Khosravi-Boroujeni H, Mehdi Ahmadi S, Brauer P, et al. Adherence to Dietary Approaches to Stop Hypertension (DASH)-Style Diet and the Risk of Cancer: A Systematic Review and Meta-Analysis of Cohort Studies. J Am Coll Nutr. 2019 Aug;38(6):513-25. doi: 10.1080/07315724.2018.1554460. Epub 2019 May 29. PMID: 31140934. Abstract available from: https://pubmed.ncbi.nlm.nih.gov/31140934/