Well, I thought I was done writing on this for a while, but I thought I would post once more about the “math” behind the “Jesus Tomb” show. I’m prompted by an article in last Friday’s Wall Street Journal (March 9, 2007) titled “Odds of ‘Lost Tomb’ Being Jesus’ Family Rest on Assumptions” by Carl Bialik (the WSJ’s “Numbers Guy”). [Click here to read — I do not think subscription is required.]
I appreciate that the WSJ article brings out some things about mathematical work that often aren’t mentioned — chief among them: how dependent the mathematician is on the assumptions he makes or, to a great extent in this case, those given to him. And I appreciate, as well, that Professor Andrey Feuerverger’s work is being seen in a better light (which means that he, himself, is hopefully being seen in a better light) than that cast by the horrible presentation of his work in the documentary.
For example, one of the key assumptions in the calculation is that the “Mariamene e Mara” tomb is Mary Magdalene. Without that key assumption, according to Professor Feuerverger as quoted in the article, the mathematical results would be “statistically not significant.” The assumption that “Yose” is so rare nickname for Joseph that it should be treated distinctly from the more common “Yosef” is also a key presupposition.
This is one of the places in which I thought the “docudrama” went south, and did so in a fairly contrived way on the backs of the filmmakers’ biases. (Please note: I am not pretending to be unbiased myself!) I recall the narrating voice of Mr. Jacobovici pointing out the crucial role identification of one of the ossuaries with Mary Magdalene played in the calculation. Then, with an air of “Da Vinci Code” suspense, we see laid out the “case” for this identification. Sadly, at this point in which honest scholarship should have played a key role, it was ignored and left to watch from offstage as Jacobovici, Tabor and company perform their “magic” and make the identification of “Mariamene e Mara” with Mary Magdalene seem an “air tight” conclusion.
Thankfully, many more responsible scholars have stepped forward to fill the gap, and anyone who has kept up with the matter has been able to learn what is not told on the TV program: the identification of “Mariamene e Mara” with Mary Magdalene is far from air tight. In fact, even the translation of “Mariamene e Mara” has been debated, with a number of possibilities seeming as probable as “Mary the Master,” if not more probable — regardless of how confident Dr. Tabor sounds when he expresses his opinion.
But I’ve gotten a little off track. In the article, Ivo Dinov, assistant professor of statistics at UCLA says, “I wouldn’t be comfortable coming up with a number like this, because the general audience will not understand that it is very, very subjective.” And that is exactly right. When I was working as an actuary (a mathematician for an insurance company), the audience’s interpretation of my conclusions always had to be foremost in my mind. Often this lead to frustration between my department and, say, the marketing department, when they would want to use the numbers to say one thing, but we would have to rein them in because it was not an accurate use of the numbers. Or, the numbers we would publish had to be explicitly qualified by an accompanying statement of the key assumptions behind them so that those assumptions could be considered by others for their reasonableness. If others are going to be making decisions based on your numbers, then you want them to have all the info that they need to make a responsible decision. Anyone deciding on the “truth” of Mr. Jacobovici’s proposal about this tomb based on “1-in-600” odds without understanding the elaborate set of assumptions made to get to that figure has made a horribly ill-informed decision.
In the WSJ article, Professor Feuerverger seems to recognize the deficiency in how his work was presented on the show when he says, “There is a mismatch between how the media works and how academia works. Obviously it would have been a whole lot better if I had completed the paper [before the documentary aired].” The “paper” in question would be his documentation of how he arrived at his result and the nature of his assumptions, which would have been reviewed and critiqued by his mathematical peers and relevant experts. That’s the way that real science works, though none of it was on display on the Discovery Channel that evening.
I really do hope that Prof. Feuerverger is given a break by those who critique his calculation. He seems (from what I’ve read) to be a decent fellow whose assistance to some filmmakers is earning him more notoriety than he would ever wish for. As he says towards the article’s end: “When doing the calculation, I was naively unaware of the extent to which the filmmakers might be depending on the ultimate result of it.” I hate to speak for him, but I would think that he probably regards this as his one mistake in this matter.
I would, however, offer that he has made one other. One of the warnings given to us as actuaries by the Society of Actuaries was to beware the temptation to work with assumed confidence and competence outside your usual bounds of experience. It seems to me that an error related to this warning has come into play here.
The Wall Street Journal article briefly references another critique of the calculation on the website of Scientific American, which I suspect is the article here. As reported in the SA article, emphasis mine:
“I did permit the number one in 600 to be used in the film-I’m prepared to stand behind that but on the understanding that these numbers were calculated based on assumptions that I was asked to use,” says Feuerverger. “These assumptions don’t seem unreasonable to me, but I have to remember that I’m not a biblical scholar.”
Essentially, the calculation is incredibly assumption dependent. That is the nature of the beast in this sort of work. Prof. Feuerverger was asked to use certain assumptions, and he checked them against his own sense of reasonableness, yet he admits that he lacks the expertise to accurately decide how reasonable they truly are. In most truly scientific publications he could communicate that uncertainty, but it doesn’t make for “compelling television,” which — as Mr. Jacobovici said in the hour long “discussion” that aired after the show — was the filmmaker’s goal. And how certain should we be about the assumptions behind the “1-in-600” calculation? As Scientific American states:
…the calculations made by Feuerverger and others rest on premises that must be decided by historians and archaeologists, who are still far from agreement on even the basics of the Talpiot tomb.
However sincerely or innocently done, the filmmakers’ misuse of Prof. Feuerverger’s calculation — using it to add an unearned sense of solidity to what is, by its very nature, a house of cards — is regrettable. And the anxiety that the professor admits he has gone through since the show aired is also regrettable. But he says in the WSJ article that the distress will make him a stronger human being and a stronger statistician, and I hope this is the case. And, I hope those who look into how mathematics was used in this film come away more educated about the role mathematics plays in such things. The presence of mathematical results brings an air of authority to one’s conclusions, but hopefully we will learn that it should not be an authority that goes unquestioned or unchallenged.