Human Resources: Understanding Salary Survey Data

Salary Surveys Involve A Lot of Math, But There is More than Math is Involved
Rows and columns of numbers, precisely arrayed on a page. Means and medians, percentiles and standard deviations. Charts and graphs.

Obviously, there´s plenty of math involved in the creation of a compensation survey, and we need to have a good comprehension of statistical concepts in order to understand and apply the data from a compensation survey. But what about some other concepts: what about intuition? prejudice? pride? embarrassment? Could issues such as these be involved in any way in a compensation survey? What about hope? faith? Or fear?

The fact is that if you know about the complex interplay of factors that combine to create a compensation survey, and if you stop and think about the complex interplay of factors that go on within your own organization - and within your own mind - in the interpretation and application of the data, you´ll recognize that to understand a compensation survey involves more than just math and statistics. There´s a lot going on behind those cold, official-looking numbers on the page. The more you know about all the things going on there, the better equipped you are to assess those numbers, and make good business decisions with them.

To the Newcomer Surveys Seem to Give the Answers in Black and White
An inexperienced compensation professional - someone like, oh, let´s say, me, more years ago than we need dwell on - might be inclined to believe that a compensation survey provided authoritative, precise facts. A compensation survey is where one would go to find answers. How much should we pay our Senior IT Engineer? Simple: we just open up our compensation survey binder, and see? There it is, in black and white. The survey has answered our question.

As we compensation professionals begin to gain more practical experience - we like to call it "gathering wisdom," while our kids tend to choose other terms ("gathering waistline?") to describe what´s happening to us - we begin to realize that a compensation survey may not necessarily provide simple answers for us. Instead, employing a compensation survey tends to stimulate a lengthening series of questions in our minds, such as: what organizations participated in this survey? How did they match to this job? How does my organization´s job match to this job? When was the survey data effective? What geographical regions does the data represent? And only after doing our best to find the answers to such questions, and then answering a few more, such as: what economic assumptions should I make in deciding how to "age" the data? Should I be aggressive or cautious in applying them? (Here´s where we deploy our crystal ball: from murky shadows, we must distinguish future conditions!) What are the internal equity implications that will flow from this situation? Then, and only then, would we sadder-but-wiser compensation professionals be ready to say, "This is how much we should pay our Senior IT Engineer."

What It Is Like as a Producer of Compensation Surveys
And then there are a few of us who, after a long career in industry as a consumer of compensation survey data, make a change and go to work for a compensation survey firm, and become a producer of compensation survey data. We like to call such a career shift a "broadening move," while others may elect to employ terms such as "having a midlife crisis," or "throwing it all away," or simply, "going over to The Dark Side." Whatever. Those of us who have done such a thing soon realize that we may have thought we understood compensation surveys before, but we really didn´t. We quickly learn that the production of compensation surveys is one of those kinds of things in life -- you know, like making sausage, or crafting bills in Congress: once you know how it´s done, you´ll never look at it the same way again.

But while ignorance may be bliss, knowledge is power, and in business, you´re almost always better off being smart instead of happy. So, please allow me to share a few of the behind-the-scenes facts of life about how compensation surveys come into this world. I trust you´ll pardon me if I shatter any illusions you may hold about the influence of anything other than objective scientific methodology.

Who Participates
First off, there´s the issue of which organizations participate in the survey. Certainly, there´s plenty of logical strategy involved: companies and/or public sector organizations participate based on their common status as labor market competitors. But the precise issue of why this particular organization participated, and why that particular one didn´t, is often far less logical, and far more political. Or even personal. It often comes down to something like: the Compensation Director from Company A used to work for Company B, and they´re all best pals, but the C&B Manager for Company C can´t stand either of them, and refuses to participate in any survey they´re involved in. None of this may be of any interest to you, but the fact that Company C isn´t in the survey may be of great concern to your company. Here´s my suggestion: never assume that the participant list for a given survey is logical or complete. Just because a survey purports to be covering the major insurance companies in Milwaukee, doesn´t mean they´re all in there. Scrutinize the participant list carefully. If it´s a survey you anticipate having an ongoing need for, proactively engage yourself with the vendor and the other participants, and the survey Steering Committee or Advisory Board, if there is one. Assert your influence, offer your assistance, and in general, do whatever you can to ensure that the participants you need in the survey, are in there. Believe me, an enlightened survey vendor will welcome your help with open arms; it´s almost always the case that an invitation from a peer in a participating organization will be more successful in recruiting a new participant than will a "sales call" from the vendor.

Job Matching Issues
Next, there´s the issue of job matching. Even though you may be meticulous in matching your organization´s jobs to the survey jobs - and more importantly, in not matching your organization´s jobs to the survey jobs, if they don´t fit - you would be ill-advised to assume that your counterpart at every other participating organization is doing the same. As a survey vendor facilitating job-matching efforts, I´ve frankly been amazed sometimes at how wide of the mark are some participants´ attempts at job matching. "But our job title is Associate Program Manager, just like in the survey, so that´s a match!" they will say, and we must reply, "Well, yes, but you´ll notice that the survey job description requires five years of related experience, and your people are right out of school. That, and your job is in the Marketing function, and the survey job is in Technical Support." "Oh," they will reply. (Shameful truth: often these conversations are the source of great hilarity between us survey folks, when no one else can hear us.) There are cases in which participants are embarrassed to admit that they don´t have any understanding of the content of the jobs they´re matching, or in which they´re insulted when we conclude that their job doesn´t match up to the qualifications outlined in the survey job description. Job matching is definitely a task that requires the vendor to exercise strong communication and interpersonal skills, along with content knowledge.

A well-conducted survey will involve extensive job matching efforts, ideally face-to-face in a participants´ meeting, or at least via telephone with the survey vendor. My suggestion here: insist upon understanding the job matching procedure for every survey you use. Be especially wary of a survey to which you submit your data, and never hear a word back from the vendor until your output report (or the invoice!) is sent to you. If no verification was provided for your job matches, how can you assume it was done for the other participants?

Data Errors
And finally, most critically of all, is the issue of the survey data itself. Oh, but this one provides a never-ending source of mirth to survey people. (Yes, you´re right, we survey people don´t get out much.) Beyond the basic, obvious errors in format - say, the pay range maximum being submitted in the pay range minimum column, and vice-versa - or completeness ("Your cover sheet says you´ve submitted 57 records to the Buyer family, but your data sheet has only 12." "Oops.") - more pervasive, and usually harder to detect, are the simply erroneous data records. There´s just no substitute for the painstaking effort on the part of the survey vendor, carefully scrutinizing every data input record, line-by-line, to be able to discover these little beauties. ("Okay, you´ve submitted 24 Administrative Assistant II´s, all making between $2800 and $4700 per month. And one making $7400. Are you sure that´s correct?") It bears repeating: be very wary of any survey in which you receive no calls back from the vendor, asking to verify data. Maybe your data was absolutely perfect, with nothing questionable at all. Maybe. Or maybe the survey vendor just didn´t take the time to scrutinize and verify it. Do you think every participant´s data submission was perfect? Do you want to make that assumption?

As a survey vendor, the most frustrating situation of all is taking the time and effort to discover what appear to be errors, and contacting the participant to verify, and getting no response. Maybe I shouldn´t have been surprised to learn how often this happens, but I was: it happens a lot. Here is where your survey vendor is truly called upon to exercise independent judgment. When we´ve left five voicemail messages and sent three emails to this company, and gotten no reply, and there´s data in their file that looks strange, and we´re up against the production deadline: what do we do? Well, we have to decide what to do, and our choices range from (a) leaving the data as is, (b) deleting the records in question, or - here´s where the judgment really comes in - (c) modifying the records in question, based on the circumstances and our knowledge and experience, to what we think the correct data is. Most definitely, it´s at this point that survey production crosses the line from science, and becomes an art.

These are very tough decisions for survey vendors. Which choice do you want your vendor to make? It depends, doesn´t it, on a variety of factors. Certainly, a good rule of thumb for vendors to follow is, "When in doubt, leave it out," and so choice "b" will typically be applied. But what if the figures in question aren´t obviously erroneous, just moderately unusual, and the vendor is just looking for confirmation. Do you want those records thrown out of your survey database, or might choice "a" be prudent on such an occasion? Or what if the figures in question are these: of 25 Sales Representative data points submitted by a company, 23 have annual base pay submitted as $50,000, while one is submitted as $5,000, and another is submitted as $500,000. Isn´t it highly likely that those final two are just typos? Might this be the rare instance in which choice "c" ought to be applied? Would you consider this an example of foolhardy mind-reading, or sound deductive reasoning?

Like it or not, every vendor conducting every survey is faced with these kinds of decisions. The confidence you can have in how well such decisions are made is a function of the knowledge, experience, and demonstrated capacity for judgment your vendor has. How well do you know your survey vendor? As a general rule, you should never place any more trust and confidence in survey data than you have in the survey vendor. Get to know your survey vendors. Run scenarios past them, and ask them how they would handle the situation. Ask other people you know about them. The more you know about their experiences and inclinations, the better you´ll be able to make your own independent judgments about the data they produce.

Especially in today´s environment, with sources of information all around us, on the internet as well as through more traditional channels, it´s easy to be fooled into believing that a particular set of data is authoritative: valid, complete, and current. After all, it sure looks official, arrayed there so neatly and attractively on the page, or on the computer screen. But don´t be fooled. Ask pertinent questions about it: who supplied it? How were the jobs matched? How was the data validated? Knowing that in the best compensation surveys, a great deal of behind-the-scenes effort must be expended to make the data usable, you should be very skeptical if the answers you receive are vague.

Conclusion
No compensation survey is perfect. As with any product of human effort, surveys can´t be expected to be perfect: they´re as fraught with error and foolishness as anything else. But neither is it true to say that all surveys are equally flawed. Some are better than others (or to put it less kindly, some are worse than others). As a compensation professional, it´s important for you to assess each survey on a basis of more than just the numbers; you should assess the whole survey process, with all of the complex human behavioral elements involved. The more you know about surveys, the less impressed you may be with the science - but the better you´ll be able to make use of the results.