Telephone Surveys Versus Other Surveys

Many clients prefer the collection of quantitative survey data by telephone. They prefer a live interviewer call the staff member at the work location. After introducing the Organizational Survey, the interviewer would ask the staff member when and where would be appropriate to call to conduct the survey. This may result in the staff member asking to be called at home or in the office at another time. A related technique is Interactive Voice Response (IVR). Survey respondents call a number and use the touch tone keys to answer questions. They can be directed by a computer generated voice, or computer controlled human voice recording.

These are excellent techniques. They have been used for decades in market and consumer research, are very effective in political polling, and are being used with greater frequency (albeit a low frequency) with Organizational Surveys. There is much to recommend about telephone survey techniques but some of the advantages are more apparent than real.

For example, it is assumed that telephone surveys produce results that are equivalent to other data collection methods, they provide results in a shorter period of time, and are less expensive than other administration methods for Organizational Surveys. These assumptions are not necessarily true.

The use of Interactive Voice Response (IVR), and live interviewer telephone surveys is not recommended. With all its benefits as a proven technique, telephone surveys have some significant problems when it comes to Organizational Surveys or employee attitude surveys. Telephone surveys are not necessarily the fastest for the reporting of results. Also, they are not as cost-effective as other methods of data collection. The most significant problem, however, is that attitude survey results from telephone surveys do not compare with results for the same questionnaire that are collected by other means.

Published research over the past decade and our own experience lead us to conclude that IVR phone survey results cannot be compared to survey results from hard copy and online surveys. Research and our own experience demonstrate that results from paper and pencil surveys, PC-based administration, e-mail administration, network administration, internet administration, and scannable fax-back forms are largely comparable to one another.

IVR phone surveys or live operator telephone interviews would be acceptable if all data were collected in this manner and no comparisons were made to results from other modes of administration. Some research suggests that phone surveys produce more favorable results than other methods. However, there is no consistency to the differences, thus attenuating the utility of such comparisons. Significant experience of one of our senior consultants suggests no consistency at all. An enormous amount of time, effort, and money went into a project to collect national norms with a telephone survey. It was abandoned when it became apparent that comparing the data to paper surveys was an impossibility..

It is not uncommon to see results vary for telephone surveys vs. paper surveys by 15 to 20 percentage points. The variation also seems to be inconsistent. If the results were always 15 points higher or 15 points lower, then adjustments would enable comparisons to be made. Telephone surveys do not have a consistent effect on these differences.

If your Organizational Survey used a telephone survey to collect its data, it could never compare the results with any confidence to other survey data collected under different techniques. Our Vice President, William E. Dodd, Ph.D., had first-hand experience with this problem while working for a major survey consulting organization in New York City. Recently published research, as well as anecdotal reports, confirm this phenomenon.

If your organization chooses to use a telephone survey, we recommend that you not compare the results with data gathered with other techniques, and not change the telephone survey in the future. Even if your organization so chooses, it will be almost impossible not to make the comparisons, and not make decisions based on unreliable comparisons.