Selection Bias and the Perils of Benchmarking


Review of Research Paper by: Shubhasheesh Bhattacharya, Faculty, IBS Pune
Research Paper:  "Selection Bias and the Perils of Benchmarking"
Author: Lerker Denrell
Research Paper Published in: HBR, April 2005
 


Shubhasheesh Bhattacharya
Faculty Member
ICFAI Business School
Plot No 5, Equity Tower, Sanghvi Nagar Road, Aundh
Pune-411 007
 


Review of Research Paper

The author is making a very valid point in this Research Paper. He says that when researchers conduct study of successful business companies, on certain issues, they tend to commit a blunder : They make generalization about the attributes of the successful companies and the top entrepreneurs. They tend to ignore the fact that even the unsuccessful companies also had such attributes which are possessed by the successful companies. Most often, the attributes of the unsuccessful companies could not be studies because they simply vanished from the market and did not exist any more.

So, the research findings and generalization taking into account of only successful companies, will be highly misleading, wrong and will not be representative sample of the whole universe; if we do not consider the study of the unsuccessful companies. 

For example, if we want to examine the effective leadership traits of managers in successful companies, and we find that they were Visionary or Humble. So researcher may generalize that effective managers in successful companies are visionary or humble. But the researcher must also consider to study of the leadership traits of managers who failed to be promoted and were fired. We may find that perhaps their styles of leadership were equally same visionary or humble.

This tendency on the part of the researcher of including only the successful companies, successful entrepreneurs in the study sample, without giving due consideration to the unsuccessful companies & unsuccessful entrepreneurs, is called Selection Bias.

Also, the selection bias is a difficult trap to avoid. Any sample of current managers will contain more successes than failures, because the failing managers are expected to be out of the company, if the company's internal selection system works properly.

Similarly, poorly performing firms tend to fail and disappear, and so any sample of existing companies will consist largely of successful ones.

Biased data or incomplete data, may lead to the mistake of overvaluing risky business practices. Suppose, researcher wants to find out the relationship between the performance of the companies & the risky business practice ( say for example, using cross-functional teams), and takes into account all the companies which have implemented this risky business practice. The average trend line will show that engaging in the risky practice, somewhat reduces the performance of the organization.

Now, suppose the researcher conducts the same study only after many of the worst performing companies had gone out of the business or acquired by other firms. In that case, we would observe more of successes and less of failures associated with the risky business practices. Hence, the average trend line will show that engaging in the risky practices, increases the performance of the organization.

The author advises that we should get all possible data on failure, in order to guard against selection bias. Also, the economists and statisticians have developed a number of tools to correct for selection bias. These tools are grounded in certain assumptions, which may be more or less realistic depending on the context.

Successes seem to be more inspirational. However, the managers looking for high performance, are more likely to achieve their goals if they give due weightage to the stories of unsuccessful companies as they do to successful ones.
 


Shubhasheesh Bhattacharya
Faculty Member
ICFAI Business School
Plot No 5, Equity Tower, Sanghvi Nagar Road, Aundh
Pune-411 007
 

Source: E-mail June 29, 2005

 

Back to Articles 1-99 / 100 onwards / Faculty Column Main Page

 

Important Note :
Site Best Viewed in Internet
Explorer in 1024x768 pixels
Browser text size: Medium