A beleaguered Michael Brown said Friday he doesn't know why he was removed from his onsite command of Hurricane Katrina relief efforts, but he does know the first thing he'll do when he returns to Washington.''I'm going to go home and walk my dog and hug my wife, and maybe get a good Mexican meal and a stiff margarita and a full night's sleep,'' Brown told The Associated Press. ''And then I'm going to go right back to FEMA and continue to do all I can to help these victims.''
Heh. Improbable Research pointedly recalls the 2000 IgNobel Prize in Psychology, awarded to David Dunning and Justin Kreuger for their fascinating paper “Unskilled and Unaware of It: How Difficulties in Recognizing One's Own Incompetence Lead to Inflated Self-Assessments” [pdf]. Dunning and Kreuger subjected several dozen undergraduates to written tests on humor, grammar, and logic; they also asked the students to rate their own abilities relative to their peers and to predict their test scores. Not surprisingly, everybody thought that their abilities were above average. Students who preformed best slightly underestimated their ability; students who preformed worst grossly overestimated theirs.

Of course, the Lake Wobegon Effect is no surprise to anyone who has taught freshmen or non-majors, dealt with incompetent (but invariably "experienced") teachers or administrators, argued with intelligent design "experts", or read a newspaper in the last five years. The hardest thing for many people to learn, especially in a subject that they've never seriously encountered before, is that they don't know what's going on, that their opinions are not facts, that their intuition is not proof. This is especially frustrating in math and CS theory classes, where the students have the tools to check whether their answers are correct, if only they'd think to try them. It's almost impossible to actually learn anything if you don't realize that you have something to learn. The first step, as they say, is to admit that you have a problem.
That's one of the reasons for my "I don't know" policy—an answer of "I don't know" on any homework or exam question is worth 25% partial credit. A blank response doesn't count; to get the partial credit, you must explicitly acknowledge your ignorance. (The other reason, of course, is that it cuts way back on random nonsense maybe-I'll-get-pity-credit-for-stumbling-on-the-right-keywords answers, which makes grading much easier.)
Dunning and Kreuger's conclusion suggests a few possible causes for the Lake Wobegon effect. (References removed.)
One puzzling aspect of our results is how the incompetent fail, through life experience, to learn that they are unskilled. This is not a new puzzle. Sullivan, in 1953, marveled at "the failure of learning which has left their capacity for fantastic, self-centered delusions so utterly unaffected by a life-long history of educative events." With that observation in mind, it is striking that our student participants overestimated their standing on academically oriented tests as familiar to them as grammar and logical reasoning. Although our analysis suggests that incompetent individuals are unable to spot their poor performances themselves, one would have thought negative feedback would have been inevitable at some point in their academic career. So why had they not learned?One reason is that people seldom receive negative feedback about their skills and abilities from others in everyday life. Even young children are familiar with the notion that "if you do not have something nice to say, don't say anything at all." Second, the bungled robbery attempt of McArthur Wheeler not withstanding, some tasks and settings preclude people from receiving self-correcting information that would reveal the suboptimal nature of their decisions. Third, even if people receive negative feedback, they still must come to an accurate understanding of why that failure has occurred. The problem with failure is that it is subject to more attributional ambiguity than success. For success to occur, many things must go right: The person must be skilled, apply effort, and perhaps be a bit lucky. For failure to occur, the lack of any one of these components is sufficient. Because of this, even if people receive feedback that points to a lack of skill, they may attribute it to some other factor.
Can you say social promotion? Grade inflation? Sure, I knew you could.
This is one of the reasons I find the current administration so scary. If the media is to be believed, George II surrounds himself with yes-men. He doesn't tolerate criticism in his environment, no matter how tangential. Conservative voters are drawn to his strong convictions, but those convictions never seemed to be tempered by honest criticism. He doesn't admit that he makes mistakes. He apparently doesn't believe that he might be wrong.
Is it any wonder, then, that Bush's hand-picked FEMA director did such a bad job? Given this environment in Washington, is anyone surprised that Brown doesn't know why he was removed from his post?
Oooh, I'm going to have to blog about something related to this. I know that ignorant people's ignorance often accounted for their ignorance about their ignorance, but I'm still amused that the line of best fit for perceived ability is practically horizontal. Wow.
Regarding giving marks for "I don't know" - I've done something similar. If a student answers a question incorrectly, and can identify that their answer is wrong, and explain how they know that it's wrong, I'll give at least half credit (sometimes as much full credit minus one mark). So, an answer of the form, "Hmm, the price of a widget that maximizes profit can't possibly be $5, because then the profit is $100, whereas a price of $4 gives a profit of $120" will get a fair bit of credit.
I think that four students have taken advantage of this in my entire teaching career.
Posted by: Moebius Stripper | September 11, 2005 at 08:31 PM
I like the "This must be wrong" idea!
Students take full advantage of my "I don't know" rule, at least once they realize that I'm serious. I've actually had a student take an algorithms final exam in two minutes—just enough time to write his name and "I don't know" seven times. He still failed, but it was close.
It occasionally backfires, though. The last time I taught an undergrad class, a few of them criticized the policy because "it discourages us from even trying". (I told them they were missing the point.) Several times that semester, students asked me during an exam "Is this answer worth more than 'I don't know'?" (I told them I didn't know.)
Posted by: JeffE | September 12, 2005 at 09:53 AM
Yeah, I can imagine that students would probe for loopholes to the "I don't know" rule. And I don't like to discourage students from trying - what I want to discourage is students not thinking about what they're doing.
Also, the following has not been my experience - The hardest thing for many people to learn, especially in a subject that they've never seriously encountered before, is that they don't know what's going on, that their opinions are not facts, that their intuition is not proof. No, I had a lot of precalc students last year who were well aware that they didn't know anything - they freely admitted that they were never good at math, that they just weren't math people, that they didn't "get" [fractions | graphs | word problems | numbers], what have you. The problem is that many of those same students were highly indignant when their marks reflected that - they're trying, therefore, don't they deserve A's? It wasn't that they didn't realize how little they knew. It was that they thought that their grade should be correlated more strongly with the grade they wanted, than with their level of understanding.
(Aside - I am tutoring a grade 12 student, who, two months ago, couldn't do early elementary school-level math. No joke: he couldn't add single digit numbers without a calculator. So, forget any level of algebra. I've been meeting with him 6-8 h/week, and he's now halfway though his class. His mark? An A-minus. To his credit, he's working really hard - but this kid is not doing A-minus-level grade 12 math. Suddenly, the "I am no good at math / give me an A" complaint makes a lot more sense, in a nonsense sort of way.)
Posted by: Moebius Stripper | September 12, 2005 at 04:36 PM
Apart from the excellent discussion, I really enjoyed the Lake Wobegon reference. I miss NPR ... but then again WILL does stream online.
(Mitch, are you reading this? If you had paid membership dues to WILL, you would have felt less guilty listening to them in Germany.) :-))
Posted by: Shripad Thite | September 14, 2005 at 06:58 AM
i agree with möbius...the flat graph of perceived ability is fascinating. i mean, i think i'd know if i had done well on a test on logic, for example, and i'd probably rate my ability on that one pretty high. same for grammar due to, i don't know, my degree in linguistics. not sure what i'd do about humor, although i'm not sure how you can take a test about that.
but i've also said that you get more points on my tests for a wrong answer if you tell me why you know it's wrong. as in, "the area of that lune can't possibly be 6 minus 3 pi."
i also try to be as forthright as i can as diplomatically as i can when a student just doesn't get it. but that gets me into trouble with parents sometimes. my best technique: when a kid doesn't get something recognizably simple in a conversation with me i let them struggle with it for a long time until they just have to admit they don't know the answer.
Posted by: Polymath | September 15, 2005 at 07:04 AM
What I find most interesting on the graph is how accurately realistic the B-students are! The 3rd quartile X 60 percentile folks just nailed it - the match between perceived ability and real ability is uncanny!
Posted by: coturnix | September 17, 2005 at 11:32 PM
"One reason is that people seldom receive negative feedback about their skills and abilities from others in everyday life."
I wonder if the Mayor of New Orleans and the Governor of Louisiana know how badly they have bungled *their* jobs with respect to Katrina? When events happened mostly as predicted in advance, they immediately began blaming the federal government for the problems, and everyone else seems to have joined in. Maybe some should spread a bit more honest feedback their ways.
Posted by: Anonymous, Please | September 19, 2005 at 02:14 PM
Wheres the pancakes?
Posted by: God | April 12, 2007 at 12:30 PM