Which is to say that in and around my field (biochemistry), I’m pretty good at sort of “vibe checking”. In practice, this is just a subconscious version of checking that a paper is published in a legit journal, and having a sense for what kind of topics, and language is common. This isn’t useful advice though, because I acquired this skill gradually over many years.
I find it tricky in fields where I am out of element, because I am the kind of person who likes to vet information. Your question about how to identify work as peer reviewed seems simple, but is deceptively complex. The trick is in the word “peer” — who counts as a peer is where the nuance comes in. Going to reputable journals can help, but even prestigious journals aren’t exempt from publishing bullshit (and there are so many junk journals that keeping up even within one field can be hard). There are multiple levels of “peer”, and each is context dependent. For example, the bullshit detector that I’ve developed as a biochemist is most accurate and efficient within my own field, somewhat useful within science more generally, slightly useful in completely unrelated academic fields. I find the trick is in situating myself relative to the thing I’m evaluating, so I can gauge how effective my bullshit detector will be. That’s probably more about reflecting on what I know (and think I know) than it is about the piece of material I’m evaluating.
In most scenarios though, I’m not within a field where my background gives me much help, so that’s where I get lazy and have to rely on things like people’s credentials. One litmus test is to check whether the person actually has a background in what they’re talking about, e.g. if a physicist is chatting shit about biology, or a bioinformatician criticising anthropology, consider what they’re saying with extra caution. That doesn’t mean discount anyone who isn’t staying in their lane, just that it might be worthwhile looking into the topic further (and seeing who else is saying what they are, and what experts from the field are saying too).
As I get deeper into my academic career, I’ve found I’m increasingly checking a person’s credentials to get a vibe check. Like, if they’re at a university, what department are they under? Because a biochemist who is under a physics department is going to have a different angle than one from the medical research side, for example. Seeing where they have worked helps a lot.
But honestly a big part of it is that I have built up loose networks of trust. For example, I’m no statistician, but someone I respect irl referenced a blog of Andrew Gelman’s, which I now consider myself s fan of (https://statmodeling.stat.columbia.edu/). Then from that blog, I ended up becoming a fan of this blog, which tends to be about sociology. Trusting these places doesn’t mean I take them at face value for anything they say, but having that baseline of trust there acts as a sort of first pass filter in areas I’m less familiar with, a place to start if I want to learn about a perspective that I know the rough origin of.
In the context of news, I might start to see a news outlet as trustworthy if I read something good of theirs, like this piece on 3M by ProPublica, which makes me trust other stuff they publish more.
Ultimately though, all of these are just heuristics — imperfect shortcuts for a world that’s too complex for straightforward rules. I’m acutely aware of how little spare brain space I have to check most things, so I have to get lazy and rely on shortcuts like this. In some areas, I’m lucky to have friends I can ask for their opinion, but for most things, I have to accept that I can’t fact check things thoroughly enough to feel comfortable, which means having to try holding a lot of information at arms length and not taking it as fact. That too, takes effort.
However, I got a hell of a lot smarter when I allowed myself to be more uncertain about things, which means sometimes saying “I don’t know what to make of that”, or “I think [thing] might be the case, but I don’t remember where I heard that, so I’m unsure”, or just straight up “I don’t know”. Be wary of simple and neat answers, and get used to sitting with uncertainty (especially in modern science research).
To some extent, I don’t.
Which is to say that in and around my field (biochemistry), I’m pretty good at sort of “vibe checking”. In practice, this is just a subconscious version of checking that a paper is published in a legit journal, and having a sense for what kind of topics, and language is common. This isn’t useful advice though, because I acquired this skill gradually over many years.
I find it tricky in fields where I am out of element, because I am the kind of person who likes to vet information. Your question about how to identify work as peer reviewed seems simple, but is deceptively complex. The trick is in the word “peer” — who counts as a peer is where the nuance comes in. Going to reputable journals can help, but even prestigious journals aren’t exempt from publishing bullshit (and there are so many junk journals that keeping up even within one field can be hard). There are multiple levels of “peer”, and each is context dependent. For example, the bullshit detector that I’ve developed as a biochemist is most accurate and efficient within my own field, somewhat useful within science more generally, slightly useful in completely unrelated academic fields. I find the trick is in situating myself relative to the thing I’m evaluating, so I can gauge how effective my bullshit detector will be. That’s probably more about reflecting on what I know (and think I know) than it is about the piece of material I’m evaluating.
In most scenarios though, I’m not within a field where my background gives me much help, so that’s where I get lazy and have to rely on things like people’s credentials. One litmus test is to check whether the person actually has a background in what they’re talking about, e.g. if a physicist is chatting shit about biology, or a bioinformatician criticising anthropology, consider what they’re saying with extra caution. That doesn’t mean discount anyone who isn’t staying in their lane, just that it might be worthwhile looking into the topic further (and seeing who else is saying what they are, and what experts from the field are saying too).
As I get deeper into my academic career, I’ve found I’m increasingly checking a person’s credentials to get a vibe check. Like, if they’re at a university, what department are they under? Because a biochemist who is under a physics department is going to have a different angle than one from the medical research side, for example. Seeing where they have worked helps a lot.
But honestly a big part of it is that I have built up loose networks of trust. For example, I’m no statistician, but someone I respect irl referenced a blog of Andrew Gelman’s, which I now consider myself s fan of (https://statmodeling.stat.columbia.edu/). Then from that blog, I ended up becoming a fan of this blog, which tends to be about sociology. Trusting these places doesn’t mean I take them at face value for anything they say, but having that baseline of trust there acts as a sort of first pass filter in areas I’m less familiar with, a place to start if I want to learn about a perspective that I know the rough origin of.
In the context of news, I might start to see a news outlet as trustworthy if I read something good of theirs, like this piece on 3M by ProPublica, which makes me trust other stuff they publish more.
Ultimately though, all of these are just heuristics — imperfect shortcuts for a world that’s too complex for straightforward rules. I’m acutely aware of how little spare brain space I have to check most things, so I have to get lazy and rely on shortcuts like this. In some areas, I’m lucky to have friends I can ask for their opinion, but for most things, I have to accept that I can’t fact check things thoroughly enough to feel comfortable, which means having to try holding a lot of information at arms length and not taking it as fact. That too, takes effort.
However, I got a hell of a lot smarter when I allowed myself to be more uncertain about things, which means sometimes saying “I don’t know what to make of that”, or “I think [thing] might be the case, but I don’t remember where I heard that, so I’m unsure”, or just straight up “I don’t know”. Be wary of simple and neat answers, and get used to sitting with uncertainty (especially in modern science research).
Thanks for the extensive response! I appreciate the perspective, particularly the nuances on peer review, and the grounded conclusion.