[Misleading] First misinformation susceptibility test finds 'very online' Gen Z and millennials are most vulnerable to fake news

Hey all, thanks for reporting this to bring some extra attention to it. I'm going to leave this article up, as it is not exactly misinformation or anything otherwise antithetical to being shared on this community, but I do want to note that there are four different sources here:

• There's the original study which designed the misinformation susceptibility test; the ArXiv link was already provided, but in case anyone would like a look the study was indeed peer reviewed and published (as open access) in the journal Behavior Research Methods. As with all science, when reading the paper it's important to recognize exactly what it is the authors were even trying to do, taking into account that they're likely using field-specific jargon. I'm not a researcher in the social sciences so I'm unqualified to have too strong an opinion, but from what I can tell they did achieve what they were trying to with this study. There are likely valid critiques to be made here, but as has already been pointed out in our comments many aspects of this test were thought out and deliberately chosen, e.g. the choice to use only headlines in the test (as opposed to, e.g., headlines along with sources or pictures). One important thing to note about this study is that it is currently only validated in the US. The researchers themselves have made it clear in the paper that results based on the current set of questions likely cannot be compared between countries.

• There's the survey hosted on streamlit. This is being run by several authors on the original paper, but it is unclear exactly what they're going to do with the data. The survey makes reference to the published paper so the data from this survey doesn't seem like it was used in constructing the original paper (and indeed the original paper discusses several different versions of the test as well as a longitudinal study of participants). Again, taken for what it is I think it's fine. In fact I think that the fact that this survey has been made available is why this has generated so much discussion and (warranted) skepticism. Being able to test yourself on a typical survey gives a feel for what is and isn't actually being measured. I consider this a pretty good piece of science communication / outreach, if nothing else.

• There is the poll by YouGov. This is separate from the original study. The researchers seem to be aware of it, but as far as I can tell weren't directly involved in running the poll, analyzing the data, or writing the article about it. This is not inherently a bad poll, but I do think it's worth noting that it is not a peer reviewed study. We have little visibility into how they conducted their data analysis here, for one thing. From what I can tell without knowing how they actually did their analysis the data here looks fine, but (this not being a scientific paper) some of the text surrounding the data is a bit misleading. EDIT: Actually it looks like they've shared their full dataset including how they broke categories down for analysis, it's available here. Seeing this doesn't much change my overall impression of the survey other than to agree with Panteleimon that the demographic representation here is not very well balanced, especially once you start trying to take the intersections of multiple categories. Doing that, some of their data points are going to have much lower statistical significance than other. My main concern is that some of the text surrounding the data is kinda misleading. For example, in one spot they write, "Older adults perform better than younger adults when it comes to the Misinformation Susceptibility Test," which (if their data and analysis can be believed) is true. However nearby they write, "Younger Americans are less skilled than older adults at identifying real from fake news," which is a different claim and as far as I can tell isn't well supported by their data. To see the difference, note that when identifying real vs fake news a reader has more to go on than just a headline. MIST doesn't test the ability to incorporate all of that context, that's just not what it was designed to do.

• Finally, there's the linked phys.org article. This is the part that seems most objectionable to me. The headline is misleading in the same way I just discussed, and the text of the article does a bad job of making it clear that the YouGov poll is different from the original study. The distinction is mentioned in one paragraph, but the rest of the article blends quotes from the researchers with YouGov polling results, strongly implying that the YouGov poll was run by these researchers (again, it wasn't). It's a bit unfortunate that this is what was linked here, since I think it's the least useful of these four sources, but it's also not surprising since this kind of pop-sci reporting will always be much more visible than the research it's based on. (And to be clear, I feel I could have easily linked this article myself, I probably wouldn't have even noticed the conflation of different sources if this hadn't generated so many comments and even a report; just a good reminder to keep our skeptic hats on when we're dealing with secondary sources.)

Finally, I'd just like to say I'm pretty impressed by the level of skepticism, critical thinking, and analysis you all have already done in the comments. I think that this indicates a pretty healthy relationship to science communication. (If anything folks are maybe erring a bit on the side of too skeptical, but I blame the phys-org article for that, since it mixed all the sources together.)

May I be honest? The study is awful. It has two big methodological flaws that stain completely its outcome.

The first one is the absence of either an "I don't know" answer, or a sliding scale for sureness of your answer. In large part, misinformation is a result of lack of scepticism - that is, failure at saying "I don't know this". And odds are that you're more likely to spread discourses that you're sure about, be them misinformation or actual information.

The second flaw is over-reliance on geographically relevant information. Compare for example the following three questions:

1. Morocco’s King Appoints Committee Chief to Fight Poverty and Inequality
2. International Relations Experts and US Public Agree: America Is Less Respected Globally
3. Attitudes Toward EU Are Largely Positive, Both Within Europe and Outside It

The likelihood of someone living in Morocco, USA and the EU to be misinformed about #1, #2 and #3 respectively is far lower than the odds of someone living elsewhere. And more than that: due to the first methodological flaw, the study isn't handling the difference between "misinformed" (someone who gets it wrong) and "uninformed" (someone who doesn't believe anything in this regard).

(For me, who don't live in any of those three: the questions regarding EU are a bit easier to know about, but the other two? Might as well toss a coin.)

Got 20/20, was rewarded with a message, “You're more resilient to misinformation than 100% of the US population!” and looked for the Fake button because that is a mathematical impossibility.

I feel like a lot of people are missing the point when it comes to the MIST. I just very briefly skimmed the paper.

Misinformation susceptibility is being vulnerable to information that is incorrect

• @ach@feddit.de @GataZapata@kbin.social It seems that the authors are looking to create a standardised measure of "misinformation susceptibility" that other researchers can employ in their studies so that these studies can be comparable, (the authors say that ad-hoc measures employed by other studies are not comparable).
• @lvxferre@lemmy.ml the reason a binary scale was chosen over a likert-type scale was because
  1. It's less ambiguous to participants
  2. It's easier for researchers to implement in their studies
  3. The results produced are of a similar 'quality' to the likert scale version
• If the test doesn't include pictures, a source name, and a lede sentence and produces similar results to a test which does, then the simpler test is superior (think about the participants here). The MIST shows high concurrent validity with existing measures and states a high level of predictive validity (although I'd have to read deeper to talk about the specifics)

It's funny how the post about a misinformation test was riddled with misinformation because no one bothered to read the paper before letting their mouth run. Now, I don't doubt that your brilliant minds can overrule a measure produced with years of research and hundreds of participants off the top of your head, but even if what I've said may be contradicted with a deeper analysis of the paper, shouldn't it be the baseline?

I'm not sure this is a good study. I mean I scored 85% so woohoo but you just get headlines to go off. The art of noticing disinformation is in reading articles and making inferences on them. Questions like "vaccines contain harmful chemicals" are obvious red flags but there are some that are a reasonable-sounding headline but I'd imagine the article itself would fall apart on first reading. I know half the problem is people don't read articles but this is a very simplistic survey.

As a terminally online millennial, I was scared for a second, but I did okay on the test. Then again, I'm 40 and barely even qualify as 'millennial', and not at all as 'young'.

I found the language of the questions was glaringly obvious. What do you think?

That is such a lazy study it's pitiful and it does in no way test your ability to discern the veracity of news, so even the full marks I got are useless.

First of all, if you generate fake headlines either test someone's general knowledge or critical thinking, don't conflate the two. Secondly, it's the latter that actually matters the most, so if you build your knowledge based on headlines, you're already close to the fake news group.

that test is bs, first I might be gullible and the second round 17/20, the study is the fake news

I would cheat on this test because I cheat in real life. I've been humbled enough times not to put total faith in my initial impression and would rather have more evidence than whatever I happen to be aware of at the moment to determine whether a claim is true.

18/20, wonder which ones I got wrong.

I imagine a main goal is to create a sensation of being overwhelmed, which in turn can easily make one apathetic, cynical.

Weird. The only people I know that continually and aggressively bring up very obvious misinformation are the 50+ people in my life.

I got 18/20, but also THEY DONT HAVE AN OPTION FOR AUSTRALIA! What kind of survey has Austria and Azerbaijan but not Australia. Seriously. And you Americans love a binary scale of political preference. At least it wasn’t a required question.

I recall reading something about fake news and propaganda some decades ago. Can't recall the source book but it goes like this:

If one person tells you something absolutely outrageous you won't believe it. If a second person tells you the same story you will stop and wonder. If a third person, preferably someone you respect, tells you the same you will have no doubts about the story at all.

I have no idea how true this is but if two more people tell you the same thing...

17/20. I got 90% in detecting fake news, but 80% in identifying real news...

Hooo boy. This article is wildly misrepresenting both the study and it's findings.

1. The study did not set out to test ability to judge real/fake news across demographic differences. The study itself was primarily looking to determine the validity of their test.
2. Because of this, their validation sample is wildly different from the sample observed in the online "game" version. As in, the original sample vetted participants, and also removed any who failed an "attention check", neither of which were present in the second test.
3. Demographics on the portion actually looking at age differences are... let's say biased. There are far more young participants, with only ~10% over 50. The vast majority (almost 90%!) were college educated. And the sample trended liberal to a significant degree.
4. All the above suggests that the demographic most typically considered "bad" at spotting fake news (conservative boomers who didn't go to college) was massively underrepresented in the study. Which makes sense given that participation in that portion relies on largely unvetted volunteers to sign up to test their ability to spot fake news.

Most critically, the study itself does not claim that differences between these demographics are representative. That portion is looking at differences in the sample pool before/after the test, to examine its potential for "training" people to spot fake news (this had mixed results, which they acknowledge). This article, ironically, is spreading misinformation about the study itself, and doing the researchers and its readers a great disservice.

I got 19/20, my girlfriend got 15/20. We both think the test design is not super good - only the headlines lead to guessing some times, where parts of article might have painted a clearer picture