I am currently preparing a paper for publication, and the last author sent it out to a bunch of people for comments. A common complain was a discrepancy between the run times of the same algorithm in two different parts of the paper. I ran a number of algorithms 12 times each, and then later on in the paper I picked the fastest and re-ran it another 36 times; the average time taken for the fastest algorithm in the second set of runs was significantly slower than in the first. Two different people asked me to fix this, but it isn’t a mistake, it is of course regression to the mean.
Anyway, this inspired me to post a very interesting anecdote from the economist Daniel Kahneman, writing about punishment, praise and regression to the mean:
I had the most satisfying Eureka experience of my career while attempting to teach flight instructors that praise is more effective than punishment for promoting skill-learning. When I had finished my enthusiastic speech, one of the most seasoned instructors in the audience raised his hand and made his own short speech, which began by conceding that positive reinforcement might be good for the birds, but went on to deny that it was optimal for flight cadets. He said, “On many occasions I have praised flight cadets for clean execution of some aerobatic maneuver, and in general when they try it again, they do worse. On the other hand, I have often screamed at cadets for bad execution, and in general they do better the next time. So please don’t tell us that reinforcement works and punishment does not, because the opposite is the case.” This was a joyous moment, in which I understood an important truth about the world: because we tend to reward others when they do well and punish them when they do badly, and because there is regression to the mean, it is part of the human condition that we are statistically punished for rewarding others and rewarded for punishing them. I immediately arranged a demonstration in which each participant tossed two coins at a target behind his back, without any feedback. We measured the distances from the target and could see that those who had done best the first time had mostly deteriorated on their second try, and vice versa. But I knew that this demonstration would not undo the effects of lifelong exposure to a perverse contingency.