So, you all saw that headline about red meat being unequivocally Bad For You, right? The headline was super scary – “All red meat is bad for you, new study says” according to the Los Angeles Times.
Very scary, very definite-sounding.
I tried to avoid looking at the study all day, because I knew it would make me angry. I suspected shenanigans from the get-go. Why? Because pretty much nothing in science is ever that unequivocal. Science is, allegedly, the study of material reality – when it’s done well. And material reality is incredibly complex and nuanced and unsure. Bets must always be hedged when observing it.
So I knew that, at the very least, the mainstream media reporting of this study — which might have been a very good study — was oversimplified and sensationalized, as any juicy news story is.
Finally, late last night, I gave into temptation and clicked on the LA Times link, which led me to the abstract of the study. And before I even got into the full-text, by looking at the numbers and methodology reported in the abstract, I was…deeply embarrassed.
I’m a nobody — I have a measly undergrad in applied science, and I have never been the best at math. And I was so dreadfully embarrassed for the researchers whose names were emblazoned on this study, for the journal who published it, and for the Harvard School of Public Health’s nutrition department. (My condolences to the rest of the Harvard School of Public Health.)
Why? Because it is an incredibly bad study – worse than I, a notorious crank with a streak of bias a mile wide, could have ever possibly hoped for in my wildest, crankiest dreams.
This study used a ridiculous methodology to determine how often people ate red meat – they used a food frequency questionnaire, something I was taught in school should never be used alone to assess a person’s actual food intake. It’s simply not precise enough – it is only a rough guesstimate, and it is vulnerable to faulty memory, to misunderstanding food amounts, and to embarrassment or shame.
Even if you gave a person a food frequency questionnaire every single day for the duration of the study, it would not be great data.
Sadly our friends, The Absent-Minded Scientists, didn’t even go to that much trouble. They based their analysis on food frequency questionnaires that were only updated once every four years.
I’m going to give you a second to let that sink in.
Once every four years.
Every four years, subjects in the study were given a piece of paper with a bunch of check boxes on it, next to a long list of foods, and asked to check off how often they ate each food over the course of the past four years. Totally accurate, I’m sure — accurate enough to pinpoint with reasonable certainty the type of food each person ate every single day for the 24 years of the study.
Oh wait a second, no. Not at all.
Then, based on this rigorous assessment of each subject’s diet, the researchers managed to find an association between people who reported eating a serving of red meat every single day…
I’ll give you another second.
A serving of red meat Every Single Day…
[Interlude: this is not to say that no one, anywhere eats red meat every single day for 24 years. Plenty of people around the world probably do. But one may assume they are not the vast majority of the population from which this study's subjects were drawn -- and even if they were, the results of the data from this study would give them no reason to worry.]
…was associated with a 20% higher risk of dying during the study.
That actually still sounds pretty scary, doesn’t it?
I mean, nobody wants to entertain a 20% higher risk of dropping dead at any moment. Death is the worst bad thing that could ever happen to you, and if avoiding a couple of hamburgers can stave it off, why the hell not?
Yes, it does sound scary — until you point out that the average risk of each person in the study dying, in the first place, was actually very low.
Per person, per year of the study, the risk of dying was less than 1%.
For the people who (allegedly) ate red meat every day, the risk of dying was…also less than 1%.
I’ll give you another second.
The risk of each person dying, per year of the study, was less than 1% — whether or not they ate red meat.
In fact, the risk of people in this study dying was quite a bit lower than the risk of the average person of the same age in the general US population (for year 1994, right in the middle of the study period.)
The subjects would have been between the ages of 44 and 89 years old by the middle of the study, and the risk of an average USian (of roughly this age group) dying in the middle year of the study was 2.5%. (I’m just grabbing rough numbers to make a point here – please don’t mistake for Actual Science.)
Which tells us something important – and probably not that Being In A Study Reduces Your Risk of Death by 67%!!! – it tells us that not only did the people in this study who allegedly ate red meat every single day for 24 years have a lower risk of death than the average person, it tells us that the people in the study don’t represent average people.
You could very well say that not being a predominantly white health professional or nurse is associated with an increased risk of death. Investigating why that is might be a pretty interesting question, no?
But, sadly, it’s not as newsy as saying that red meat will kill you.
Last layer of the onion: this study was not a clinical trial, which means it can only draw correlations between things – it cannot prove causation.
So even if the dietary assessment strategy were sound, even if the population of the study did represent the average person, and even if the difference in risk of death between meat eaters and not-so-much meat eaters was very large, it would only signal the need to do a more rigorous study to get to the bottom of the association, and to find out whether it’s likely that eating more red meat makes you die faster. Nothing more.
Results of the study aside, if a wholly unimpressive person like myself can read the abstract and see some pretty big problems in this study, then certainly a doctor, or a biochemist, or a dietitian, or an epidemiologist, or anyone who has taken a statistics course most definitely can and will. Which should be rather embarrassing for its authors.
Which leads me to think that I, and all the people out there with real scientific training, are not the target audience for this piece of scholarship.
So who is the target audience? Who might take it at face value?
It would have to be someone who wouldn’t look at the abstract, let alone the full text of the study. It would have to be someone, if they did look at the abstract, whose first instinct wouldn’t be to pull out a calculator and do some arithmetic. Someone like, say, a hurried journalist who reads the press release, hears what seems a plausible and common-sense conclusion, and leaves it at that.
Someone like a person who leads a real life, in a complex world with a bunch of information flying at them from all sides, and who counts on the news to at least resemble the truth — and who doesn’t have time to fuss with a bunch of statistics. Who furthermore, in an honest world, wouldn’t have to.
I’m convinced that this study, embarrassment that it is, was not written to gain respect from people who would read the full text, calculator in hand, as part of their day job. It wasn’t written for other scientists, doctors or, actually, for anyone like the health professionals who were actually in the study.
It was written for the sake of headlines. It was hand-crafted for newspapers, which we count on to deliver at least a Reader’s Digest version of the truth. Last, and most importantly, it was written for someone like a person who leads a busy, complex life, and who doesn’t always carry a calculator.
Someone who might never know that the data is bad, and the headline misleading. Maybe someone like your friends who are Facebooking it, and your mom who’ll now think twice about buying dad’s favourite beef jerky. Maybe even someone a lot like you.
This study didn’t prove that red meat was bad, because it didn’t prove anything at all — except that predominantly white health professionals die less often than the average person, and that while the nutrition department at the Harvard School of Public Health can easily drum up media attention, they are not easily embarrassed.
The foundation of nutrition as we know it remains variety. And variety, as we know it, can still include red meat if you like it and are inclined to eat it. I personally wouldn’t suggest eating it every single day for 24 years, but even if you wanted to, this study wouldn’t give you a single good reason to worry.
I’d be more concerned if I were not a predominantly white health professional, since their risk of death is so much lower than average — probably because they have the luxury of not being terrorized by studies like this.
ETA: Some commenters have pointed to some other critiques of this study, which you may want to take into consideration: