What’s a credible source?

Note: This post should in no way, shape, or form be construed as a criticism of Pundita, one of my favorite bloggers and correspondents. This post are my musings on the path I took after reading one of her many fine posts.

---

I tend to follow links. You can learn a lot by doing that and find yourself in places you might not otherwise discover. I also read footnotes. I can think of at least one book that can be read for the footnotes alone: Sir Richard Francis Burton’s remarkable translation of the Thousand Nights and a Night.

In a recent post, “Once upon a midnight dreary quoth the Raven, ‘The vodka is good but the meat is rotten.’”, Pundita linked to an article from The Epoch Times. The article purports to be a transcript of a speech given by Chinese Minister of Defense Chi Haotian. I found the speech completely horrifying. A mixture of mythology, fantasy, poppycock, racism, nationalism, strategic planning, and monstrosity. Here’s a snippet:

Comrade Mao Zedong taught us that we must have a resolute and correct political orientation. What is our key, correct orientation? It is to solve the issue of America.

This appears to be shocking, but the logic is actually very simple.

Comrade He Xin put forward a very fundamental judgment that is very reasonable. He asserted in his report to the Party Central Committee: The renaissance of China is in fundamental conflict with the western strategic interest, and therefore will inevitably be obstructed by the western countries doing everything they can. So, only by breaking the blockade formed by the western countries headed by the United States can China grow and move towards the world!

Would the United States allow us to go out to gain new living space? First, if the United States is firm in blocking us, it is hard for us to do anything significant to Taiwan and some other countries! Second, even if we could snatch some land from Taiwan, Vietnam, India, or even Japan, how much more living space can we get? Very trivial! Only countries like the United States, Canada and Australia have the vast land to serve our need for mass colonization.

[…]

To resolve the issue of America we must be able to transcend conventions and restrictions. In history, when a country defeated another country or occupied another country, it could not kill all the people in the conquered land, because back then you could not kill people effectively with sabers or long spears, or even with rifles or machine guns. Therefore, it was impossible to gain a stretch of land without keeping the people on that land. However, if we conquered America in this fashion, we would not be able to make many people migrate there.

Only by using special means to “clean up” America will we be able to lead the Chinese people there. This is the only choice left for us. This is not a matter of whether we are willing to do it or not. What kind of special means is there available for us to “clean up” America? Conventional weapons such as fighters, canons, missiles and battleships won’t do; neither will highly destructive weapons such as nuclear weapons. We are not as foolish as to want to perish together with America by using nuclear weapons, despite the fact that we have been exclaiming that we will have the Taiwan issue resolved at whatever cost. Only by using non-destructive weapons that can kill many people will we be able to reserve America for ourselves. There has been rapid development of modern biological technology, and new bio weapons have been invented one after another. Of course we have not been idle; in the past years we have seized the opportunity to master weapons of this kind. We are capable of achieving our purpose of “cleaning up” America all of a sudden. When Comrade Xiaoping was still with us, the Party Central Committee had the perspicacity to make the right decision not to develop aircraft carrier groups and focus instead on developing lethal weapons that can eliminate mass populations of the enemy country.

You may read the whole thing, for what it is worth.

My initial reaction to the article was anger. My second reaction was curiosity. Could this possibly be true? I read everything I could put my hands on on Chi Haotian particularly transcripts of speeches. I checked the source of the speech. I read everything I could find about the source of the translation and the source of the original.

My Mandarin isn’t good enough to read the Chinese language version of the speech (or any of the general’s speeches). It’s pretty difficult to determine authoricity based on translations.

The Epoch Times is a questionable source. The Times is an anti-Chinese Communist Party newspaper and has been characterized as having a pro-Falun Gong bias.

The source of the Chinese language version of the speech was Boxun.com. Boxun (“Abundant News”) is a US-based Chinese language dissident news site. Here’s what EastSouthWestNorth has to say about it (from the unlinkable South China Morning Post):

Boxun’s founder, who goes by the pseudonym of Wei Shi and describes himself as a businessman, said from the US that he could not verify the web-posted stories from Qinghai that Boxun had run. Nor could he vouch for the alleged whistleblower’s credentials. All Boxun’s non-secondary source reports are posted anonymously. But he said he hoped that by putting the stories in the public domain, somebody would prove them true or false.

There is no second independent source for the speech.

So let’s recap. We have a speech that was allegedly given by an actual Chinese official. It’s not possible to establish the authoricity of the speech by textual analysis. There’s only a single source for the speech and the publishers of both the English language and Chinese versions of the speech may have axes to grind. To further complicate things it’s not possible to establish the reliability of several of the critics of these sources e.g. Wikipedia and EastSouthWestNorth. The final critic, the South China Morning News, is a legitimate newspaper and, presumably, can be considered to have at least a little reliability. Unfortunately, the quotation attributed to the SCMP isn’t actually linked and can’t be verified.

It seems reasonable to treat all of this with at least a grain of salt.

That brought me to a larger question: how do you evaluate the credibility of sources, generally? I fired off a number of emails to relatives and friends who do this sort of thing for a living: reference librarians, journalists, etc.

The sad truth is that there is no gold standard for establishing the credibility of a source. Journalism, particularly, is seat-of-the-pants on the issue. I have, however, compiled quite a nice little library of tips for establishing the credibility of sources on the Internet and generally. Here are a few that you might find useful:

University of Vermont bibliography on evaluating web information	A great site with lots of resources (online and otherwise) for evaluating credibility. Forms, checklists, guidelines, etc.
University of California, Berkeley Library Guide to Evaluating web resources	An excellent, detailed, highly explanatory set of tips for evaluation.
Cornell University Library Guide to Critically Analyzing Information Sources	A fantastic, detailed authoritative guide to the subject.

It appears to me that it should be possible to at least make a start at a formal tool for measuring the credibility of sources using the approaches suggested in these resources and coming up with a quantitative measure of reliability. I’m thinking of something using Likert scales, that would save prior results in a database, and so on. Ideas?

UPDATE: One thing I’d meant to mention but somehow lost track of in formulating the post is that simply because a source isn’t credible or reliable doesn’t necessarily mean it’s wrong. That’s called the Genetic Fallacy. There are other considerations for measuring the truth or falsehood of a proposition than the source of the assertion.