Putting “Peer Review” and “Scientific Consensus” in Perspective

As I was cleaning up my Inbox this evening and going through some old saved e-mails, I came across an article passed on to me by MK that was very worth passing on to all of you.  It was sent to me last May, so it isn’t the most recent story off the presses, but the content is still worth a read.

Before linking to it I should qualify myself a bit…

I love science.  (I know: some of you out there interpret sentences like that made by religious people to mean, in reality, “I want you to think I love science but really I hate science.”  I don’t know how to convince you otherwise.  We don’t have any plans for a Child #5 in the works, so to speak, but if we did and I named him/her “Science” would that help?)

I am one of those who believes that repetition of results is important.  I believe that, in general, one strong and carefully controlled study that repeats the results of a previous study is worth far more than a large number of anecdotes and “personal stories” about how something “just works” (regardless of what a Kevin Trudeau may say in his profit machines books).  I am skeptical of many (though not all) “alternative therapies” for some illnesses or conditions for the very reason that if they did, indeed, work with the accuracy and efficacy that their loudest proponents boast of, their results should be repeatable under reasonable controls.

I believe that a peer-reviewed paper should be given more of a benefit of doubt than an article published only on some fellow’s website (or, for that matter, something said in an infomercial).  I believe that the practice of science is, generally, self-correcting if given enough time (which may, in some cases, need to be a great deal of time).

But I also believe in the fact that the practice of science is very much a human endeavor.  And as such, it is very much subject to the passions and pitfalls of all other human endeavors.  It is subject to groupthink, pride, the insidious but unavoidable tyranny of assumptions, and the Jeremiah 17:9 nature of the human heart.

And I think that these human faults are brought out more frequently and more intensely when the subject is a big one (evolution, human origins, global warming, et al.).

So, while I do love science, I do not worship in its cathedrals as Carl Sagan taught me to when I was a child.  (As much as I adored St. Carl, I long ago learned to take his statue off the dashboard of my car.)

For instance, while I spoke very positively above of the role of peer-reviewed journals, I have learned that they are not quite what they are understood to be by the public.  That is, they are not the completely fair-minded arbiters of quality and truthfulness that the uninitiated might think they are.

As products of the very human activity of the practice of science, they very much bear the fingerprints of an oh-so-faulty humanity.

Which leads me to my link, for which you have had to suffer an interminably drawn-out introduction (my apologies!).  The article is “Peer Review, Publication in Top Journals, Scientific Consensus, and So Forth” by Robert Higgs.  While I gather that the motivation for the article is global warming and climatology, the points Professor Higgs makes provide for an excellent commentary on science (and medicine) publications and “pronouncements” in general, and he speaks with the kind of authority that comes from a great deal of experience.

It isn’t an exciting read (though I found it thrilling!), but neither is it a long one, and it does an excellent job of putting science journals and publications in a perspective that the “layman” rarely gets to see.  While I have reasons to take scientific proclamations with a grain of salt (or a pinch–perhaps a pound) in addition to those he mentions, those he chooses to elucidate are powerful enough to dispel a good bit of the illusion of “default trustworthiness” that surrounds the phrases “peer review” or “scientific consensus”.  And I recommend it here for those who might find such an article educational.

14 thoughts on “Putting “Peer Review” and “Scientific Consensus” in Perspective

  1. Oh, to be sure, that any one (or handful of) peer-review studies should be taken with a grain of salt. No doubt there.

    But you have to acknowledge that science (and it’s applied version, technology) are vastly more successful than its alternatives, including religion and superstition, as a way of explaining and understanding the natural world.

  2. I forgot all about this article. I think this line just about sums up scientific review, which is partly why it works, and partly why it fails: “Science is an odd undertaking: everybody strives to make the next breakthrough, yet when someone does, he is often greeted as if he were carrying the ebola virus.”

    I also like the line near the end: “Climate scientists are the best qualified people to talk about climate science, but they have no qualifications to talk about public policy, law, or individual values, rates of time preference, and degrees of risk aversion.” And he says ditto to Congress and the UN, stick to what you know. Scientists: deliver the facts… thoroughly. Then serve as counselors to policy-makers rather than handing out your own solutions. Too many times, problem solvers think they can solve any problem in any area. Insight, possibly. Expert, no. Cue Richard Dawkins the biologist, please…

  3. Howdy, Dan, and thanks so much for writing in.

    I have to acknowledge that? Um, no I don’t. Superstition, absolutely. Religion, well, maybe all of them but mine. 🙂

    Seriously, you make it sound like a competition, when, of course, it isn’t. God never intended to tell us in Scripture how the organelles of the cell do their job, so is science “more successful” because it does? In cases like that, it’s like saying Hank Aaron was more successful at hitting homeruns than President Lincoln. Or like saying, “Wow, this apple tastes more apple-ish than that orange does.” Sure, you’re saying something in each of those cases, but not really much. Perhaps you were trying to imply more than I am reading in your statement — if so, I invite you to elaborate further.

    Now, that said, science is not nearly as good at “explaining and understanding the natural world” as many scientists claim it is, and sadly it is too often the most vocal of that wonderful tribe who tend to make the most outrageous claims.

    And (that said)², I am not of the Stephen Jay Gould variety of humans who thinks (or in his case, thought, per Psalm 146:4 & Ecclesiastes 9:10) that science and religion involve two completely separate spheres of authority and thus cannot come into conflict. Rather, while I disagree with Richard Dawkins on many things I do agree with him that both fields of study, science and religion, are making truth claims that can at times (at least theoretically) come into conflict.

    I could say more, but I have chewed on your bait enough to enjoy the flavor, and now I have to floss. 🙂

    Again, please elaborate if I have missed something you are trying to say.

    Have a great day, and thanks again for writing.

    Regards,
    Wallace Smith

  4. Howdy, Mike —

    Those are both great quotes. There are actually a good number of good quotes in that article (like the one on the “incestuous” nature of the research community), but I knew that if I began picking and choosing I would have been up all night (or, at that point, all morning).

    Thanks for the comment, as well as for sending the article in the first place!

    Regards,
    Wallace Smith

  5. That was a very interesting article. It will be good information to remember this next year as I am researching grad schools. Finding a well-funded research project to work on through grad school is one of the most important things to do, I’ve been told.

    Furthermore, it addresses a problem that will likely never be resolved by people. How do we know who’s opinion to trust on a given matter? Currently, many people decide based on how many PhDs are on each side. One could make the argument that this is bad. It leaves science open to the dangers of faction. It ensures a slower response time to the ideas of revolutionaries. One could also argue that this slow response time is beneficial. In political science, slow response time is sometimes argued as a form of a check against faction. I am very intrigued by this concept and have been reading/discussing it more. At the end of the day, it’s good to analyze where research money comes from and what it’s going to. A large shift of governmental funding to basic scientific research during WW2 led to some of the most revolutionary finds of the time. Calculus was created completely in a vacuum from any funding. I don’t know what sort of balance is best to get money where it is most beneficial, but I know there is a LOT of money being thrown around. It would be nice to know that people are concerned with putting it in the right place. Like my pocket.

  6. Well, as physical human beings, we do have the ability to accumulate material knowledge. Look at the amazing progress in technology that humankind has made in the past few centuries.

    As physical human beings, we are still subject to the flaws of human nature, however. Which includes biases, preconceived conclusions, and downright pettiness at times.

    There’s no competition between science and religion. The Bible simply provides spiritual knowledge not available to physical human beings. Think of the scientific progress humankind would make, if the problems of human nature did not exist.

  7. Hi Mr. Smith,

    Perhaps it would help if you explained that you love natural science, but not the metaphysical combination of naturalism (“nature is all there is”) and scientism (“the scientific method is the only path to truth”) that is either too often confused with natural science or else is believed to be necessary to natural science.

    The joke all the time is that naturalism and scientism are self-defeating propositions. They are meaningless because neither can be tested in terms of their own rules. They are simply ways of closing one’s mind to other possibilities, and frankly to where the evidence from natural science leads (to supernaturalism [“yeah, nature exists, but so does a transcendent First Cause to nature”] and to true science coupled with true religion).

    People who come up with the false “science vs. religion” dichotomy overlook the fact that they too have a religion, or something that serves the same function, in their thinking (and your comment about “St. Carl” illustrates this). There is no such thing as metaphysical neutrality; we all operate from articles of faith. The only question is which articles of faith lead to the simplest and most complete explanation of the facts assembled through the scientific method. That’s why I believe…well, what you and I believe.

  8. Well, the good news is as long as we remain curious and keep digging, the truth about nature will either be shown to be true or false over time. We will either come up with a way to demonstrate String Theory right or wrong, or it will remain just a cool idea. We will one day prove dark matter and energy, or refine the law of gravity or find some completely different way to describe it all. Proof finds a way. Yes, there is still the Flat Earth Society running around out there, but nobody takes them seriously. The fact remains, truth is truth is truth (modus ponens style).

    Likewise, human reasoning about God will one day be settled by His appearing. “What’s in that giant booming cloud that’s approaching?!? Quick, let’s quit blowing each other up and fight it instead! Aaaaah, my eyes!” We just have to be VERY patient and accept what’s currently available while making our own educated decisions on what’s closest to the truth based on what we’ve experienced and already understand.

  9. Dear Mr. Smith,

    As as Christian and scientist, I felt compelled to reply to your reflection. Probably I misunderstood some of your statements, but it seems that you are trying to somehow de-construct ‘peer review’. It is true that often lobbies are behind specific publication (or lack of them), but it cannot be generalized that scientists are not objective in their reports.

    In what religion and science concerns, I believe that it could be wise to hear some of the online lectures by NT Wright, the Lord Bishop of Durham, who clearly describes differences in the scientific method according to the field. E.g. History is all about single events, that would never be repeated as we do with other kinds of laboratory or survey based research.

    One of the dangers within Christianity is to simply dismiss automatically whatever has science in the label. Often, we Christians give very poor answers or are perceived as fanatics simply by our approach to science. I work in several projects aiming at providing evidence for specific policies. However, to our dismay, despite the evidence based recommendations, governments and policy makers prefer to abide to the lobbies than to evidence. According to my understanding, neither the Church nor the governments hear the scientific community enough to actually act accordingly.

    Regards,

  10. Greetings, refreshingfromheaven, and thanks for your thoughtful response.

    Indeed, I was not trying to de-construct peer review. The article to which I linked provides perspective on peer review that is worth noting, however, and I don’t think you have said anything to discredit the good points he has made in that article. Also, I do not mean to generalize that scientists are not objective in their reports. I do mean to generalize, however, that (1) scientists are not infrequently less objective than they believe themselves to be, and (2) they are not immune from the tyranny of assumptions or the many other challenges that plague all other human endeavors.

    Your comments about Wright’s observations would have more impact if science did not concern itself with unrepeatable, single events (the creation of life, the beginning of the universe). However it does so concern itself – and rightly so, I believe. There is no reason why scientists should not explore such avenues, and I encourage them to do so! I just wish that such explorations were salted with more humility and less seasoned with dogmatism. Or at least that there would be more public recognition that dogmatism plays a role in their endeavors at all – a fact borne out by history, but hardly admitted.

    By the way, even though it is not directly related to my post, I want to comment on one of your later remarks. I believe like you do that governments and policymakers are VERY beholden to the lobby machinery in our system. It really is a blight on the process. However, I also believe that scientists do not generally have the policymaking expertise that they tend to credit themselves with. They should stick to doing their best to finding and reporting the facts as best they can and leave the policymaking to others, and those scientists who fancy themselves frustrated policymakers could do with a healthy dose of humility.

    Determining a fact and deciding what to do about it are two fairly separate and distinct acts that require two equally different sets of skills and expertise. Just because scientists are experts at the one does not necessarily imply even a functional competence at the other.

    Perhaps “neither the Church nor the governments hear the scientific community enough,” as you suggest. (I am thankful that my church does not generally fit such a description, which is one reason I am drawn to it.) Yet even if it were so, in my experience many in the scientific community are so smugly convinced of their lack of need to listen to others and so blind to the possibility of fundamental error that I should think that they would find “the Church” and “governments” appropriate, like-minded company.

    If this response has seemed harsh, it is not meant to be and I apologize. Thanks so much for taking the time to write so clearly, and I’m sure you have given those who read these comments something to think about.

    Warm regards,
    Wallace Smith

  11. Howdy, again, Mike.

    You say something important there (believe it or not!). It’s easy to get all in a tither because some historians say that the Hittites were a fictional people, when — given enough time — the truth comes out (as it did with the Hittites).

    Science revolutionaries are often dismissed and lumped with the “other nuts”, but if they’re right history will validate them (even if it is a century or so later!).

    Such human endeavors as these are generally self-correcting, however slowly (which isn’t always a bad thing, like the article says). And if too slowly on some accounts, we know there is the Christ-correcting that’s a’coming. 🙂

  12. I received a comment in which someone wrote:

    “It seems as though you are just discussing the problems with acting under a given paradigm, a problem that people will always face.”

    My comment:

    On one hand, I would not say that I am only discussing that, yet on the other hand, if I understand what you are saying then it is, indeed, included in what I am discussing. And while people will, indeed, always face the problem of acting within a given paradigm, worldview, set of assumptions, etc. the validity of that paradigm is not always discussed like it should be. That, or the possibility that the paradigm should be questioned is dismissed out of hand.

    And while paradigms/worldviews/assumptions differ from person to person, they are not all equally valid. Yet they all affect how the conclusions are formulated and understood.

    [Also, my apologies: I couldn’t allow your comment to post because your avatar was inappropriate for this blog. Sorry about that, and I hope you’ll understand. I do appreciate your taking the time to write, however — thanks!]

  13. I have to disagree with “refreshingfromheaven” regarding his oblique suggestion that most scientists are objective in their reports. They might believe they are being objective, but they do have biases which they are frequently unaware of.

    I have seen this time again with university professors that I know. The scientific method of hypothesis and experiment leads them down the road of proving a preconceived conclusion. They decide on the answer, then look only at the evidence which supports their desired results. They do not test or consider potential fallacies in their theory. If they do, it’s a cursory effort with no meaning.

    The worst are those who simply stack statistical correlations, then develp conclusions based on correlations. A statistical correlation should be a beginning point for asking questions, not the conclusion.

    The secular view that scientists are neutral observors is simply false.

  14. Apropos to a lot of comments so far:

    I got back not long ago from an extended trip to Jerusalem, in which I attended two academic conferences. One was a conference on music in the Ancient Near East and eastern Mediterranean, one a symposium on the Talpiot Tomb — see my Yahoo blog:

    http://360.yahoo.com/johanan_rakkav

    I am on record (literally: on tape in the Q&A session) as reminding those present at the Talpiot Tomb symposium that there are two kinds of scholars: those who let their data shape their metaphysics, and those who let their metaphysics shape their data. You can tell the difference (I noted) in that those in the latter group overemphasize part of the data and underemphasize or suppress the rest. (I did not say “distort”, I believe, although I saw that happening before my eyes too in some cases.) It’s a flaw in human nature that we all have to watch out for (I concluded).

    I think it’s as good a sign as any that the God of the Bible exists (and is reliable) that the first group of scholars — the strict empiricists — end up making an increasingly good case for biblical faith as they proceed. Whereas the latter group, willy-nilly, show themselves trying to evade biblical faith by manipulating the facts to their own advantage. The tragicomic joke is that they don’t or won’t see what they themselves are doing — their own illogic and whatever motivates it blinds them.

    Steve points to a flaw in secular reasoning: that “scientists are neutral observers”. Indeed. There are no such things as neutral observers, either physically or metaphysically. There are only those who seek the simplest and most complete explanation of the facts they observe (and let that explanation change their metaphysical viewpoint, if necessary) and those who don’t. That is as true in religion as it is in science. I can understand someone who doesn’t believe the Bible because he doesn’t have enough facts to draw a conclusion that will satisfy him. I can’t understand someone who doesn’t believe the Bible simply because he’s looking for a pretext not to do so. Yes, that too is part of human nature, including my own; but really, shouldn’t we humans know better by now? (Rhetorical question.)

What are you thinking?

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s