I filled out the annual Customer Service Survey, dropped it in the mail, and forgot about it … until one day a nervous Field Service Engineer and a Concerned Corporate Suit showed up outside my office.
This was back in the day when the numerous DEC (later bought by Compaq, then by HP) minicomputers we had ranged in size from dorm refrigerator to 6-foot high, 10-foot long cabinet (not counting the storage systems). The unit of repair was usually a “module” — a circuit board that plugged into a backplane, or a disk HDA, that sort of thing.
We had service contracts with 4-hour (business day) response time. At the time, DEC had a real Field Service office in Jackson and a logistics hub in Memphis. Typically, when a service call was required, a FSE from the local office would show up with one or more “kits” containing related modules, and swap out the module most likely to be failing. Generally the first one or two modules tried would solve the problem.
If the local office didn’t have the right module (or the module from the kit was DOA), they would get it (or a whole kit) sent overnight from Memphis. In one or two extreme cases, they had drivers leave from Jackson and Memphis, meet halfway, and return.
The local guys were great. Also, once they realized I knew what I was doing, they’d trust me enough to just drop off parts, let me fix stuff overnight, and pick up the bad parts the next day. Good for me (no daytime downtime for a 1000 users) and good for them (more time for customers that needed it). More on this later.
However, the logistics guys in Memphis had had a really bad year. Way too many DOA parts in kits, or out of stock parts which they had to get shipped in cross-country.
The infamous Customer Survey covered all aspects of support: local office services, logistics, software support, etc. It was a fairly long mark-sense (pencil in the circles) form, broken into those various categories. I gave the local Field Service office high marks (probably all 9’s and 10’s), but zinged Logistics as they so well deserved.
The Concerned Corporate Suit
Why DEC decided it was important enough to send a Concerned Corporate Suit (CCS) to visit me personally, I have no idea. Perhaps they had cloned a bunch of ’em and needed to give them something to do.
In any case, DEC was well known for two things: first, generally excellent engineering; second, being so bad at marketing that they couldn’t give away ice water to someone dying of thirst in the desert. I understood why much better after this meeting, assuming marketing and probably most management (or at least the policy makers) were CCS’s.
Anyway, the CCS had come to explain to me the ramifications of my survey responses.
Responsibility Without Authority
Back to the Survey … now you’d think that this would make the local guys look good and that someone in Memphis would get his ass chewed a bit. But you see, that’s not how the survey worked. The FSE had a good reason to look nervous when he showed up at my office: the local office staff’s raises, promotions, and even continuing employment depended on the overall survey score. Despite the fact that the local guys provided excellent service, and had absolutely no control over the performance of the jokers at the logistics hub, their livelihood would be impacted by shoddy performance in Memphis — if you were honest on the survey.
Point #1 — responsibility without authority is stupid
Know What You’re Measuring … and Why
When you measure something, you expect variations in the results. Furthermore, you have to select the appropriate range of values so that you can see that variation. For example, if you’ve ever used a DMM, you have to select the correct range — trying to measure the voltage of your wall outlet using the 0-10 volt range is simply gonna peg the meter (or burn it out).
Satisfaction surveys are highly subjective and I would guess extremely variable, even if designed well and used appropriately. To have anything worth analyzing, you’d need to have some form of curve, maybe a bell curve, maybe not, but at least some shape that indicates that you’re picking up a sort of consensus with some outliers tapering the ends.
The Concerned Suit explained to me that Corporate expected all employees / centers to be rated 10, or (he implied) they were considered failures. Well, that’s easy to do if you set your criteria so low that only an earthworm or chicken would rate below 10. But if your 1-10 scale goes from “can drink a glass of water without drooling unless distracted” to “walks on water while chewing gum and juggling chainsaws”, then expecting a “10” is a little outrageous.
But that was just my own naivete. The Survey wasn’t meant to be an instrument to evaluate and suggest improvements to customer service. It was actually a club to browbeat employees and a marketing tool (“We’ve been rated 10 by our customers for N years in a row!”).
Point #2 — if you expect to peg the meter, you’re not measuring anything
Gaming the System
I liked working overnight, and preferred to have downtime scheduled then because it inconvenienced fewer users (and limited the number of people interrupting me with “Is it fixed yet?” (see all these parts on the floor; do you think it’s fixed yet?) and other inane questions).
We briefly tried DEC’s Remote Diagnostic Console, which tied into DEC’s network and would proactively report potential and real problems to Field Service. Which was fine, except that it logged an “official” service call. Which meant that FS had to respond within four hours. Which meant that if we didn’t shut down and “repair” whatever was bad when they showed up (which we wouldn’t do arbitrarily, because we had our own service responsibilities to our users), they got zinged for taking too long to complete the service call.
So, the game was to notice any potential problem, informally call the FS guy and say I needed module XYZ123 for (let’s say) Wednesday night, and he’d drop off the part and “open” the call on Wednesday. If it was more difficult (that is, FS would have to do the diagnosis / work), I’d schedule downtime, and wait until then to log the call. Of course, for rare it’s-really-broken-and-we’re-down stuff, I’d log the call immediately.
In general, people will do a lot of whatever you’re measuring them with. Lines of code? Your programs are suddenly much longer. Number of calls answered? That single extended problem turns into multiple “minor” calls answered and closed right away. Ditto bug fixes, parts replaced, widgets produced, units sold, ….
Point #3 — the people you evaluate will game the system you use to evaluate them; choose your criteria wisely
My discussion with the Concerned Corporate Suit covered points one and two pretty directly, and the idea of point three in general without ratting out FS. However, he was adamant that they expected 10’s.
Me: So you want me to lie on the survey, rather than giving you useful information?
Me: I see. I will no longer waste my time on your surveys, since you won’t address the problems I’ve indicated but will penalize the local office instead. If you want them filled out, send them to my boss.
They kept sending them to me, and I kept throwing them in the trash.