23

After seeing this question today on variation of the FizzBuzz interview question, as well as seeing other questions related to it around, I have to ask: is there any formal research on how effective this is as an interview technique? There are plenty of folks who say that it filters out some crazy-sounding percentage of candidates (95%+ in some cases) but how good is it for finding good programmers/developers?

There has been some research that shows that soft skills might be better indicators than hard skills for choosing a good candidate for a job role. As well, there might be some intermediate reasons for so many candidates failing this test (nervousness, thinking "it's a trick", etc).

Based on the comments so far, it looks like the idea behind FizzBuzz questions is to determine whether or not the person has good (enough) coding skills to be further considered. I'm not totally convinced FizzBuzz actually does this. Also, remember that FizzBuzz filters a very large percentage of candidates very early in the hiring process, which could mean that significant numbers of good candidates are being removed early on because they can't solve silly problems fast enough.

Also, I'll point out that looking at FizzBuzz effectiveness is something that can completely be done in a quantitative manner. This is unlike looking at "fit" or "passion" which are highly variable and highly unscientific (but still quite important for new hiring decisions).

Community
  • 1
joshin4colours
  • 3,708
  • 1
  • 27
  • 37

3 Answers3

34

The purpose of fizzbuzz isn't to find good programmers, it's to find a certain class of bad programmers, which is people who can't implement a simple algorithm. Your question is sort of like asking how many Nascar champions are identified during their driver license exams. There's a lot of middle ground between someone you definitely don't want to hire and someone you definitely do.

There are other classes of "bad" programmers that require different methods of identification, like those who aren't proficient in a given language or have behavioral issues. However, "fizzbuzz-bad" programmers are almost universally undesired, which is why the test is so frequently recommended. Whether a programmer who is overly territorial or one who needs to transfer skills to a new language is "good" or "bad" depends much more on the individual employer's needs and preferences.

Karl Bielefeldt
  • 148,830
14

It's been a long time since I read up on job interview research, (so I can't site specifics, for which I apologize) but the general rule is that people hire on the basis of soft skills but there is no correlation between that and performance. Testing for specific skills can work, for personality does not (according to measurable data).

Almost everyone believes their special interview techniques work, the data says you should probably not bother interviewing (the "tell me about yourself" part of the interview, as opposed to objective testing) since candidates selected by interviewers are not any better but interviewers do show biases towards extroverts, people demographically similar to the interviewer, tall people, attractive people, etc. (Arguably you should hire people who do well on tests and bad on the rest of the interview, since they are apparently just as good on the job but may find it harder to leave your company).

I kind of expect to get flamed for this, usually anyone I tell this to says "but we use method X and it really works for us", but unless you

  1. hire some people who don't pass your interview process,

  2. don't tell their coworkers who passed and who didn't, and

  3. compare performance of those who passed with those who didn't, using previously agreed on objective criteria

you really have very little idea whether your process works. Unfortunately the effectiveness of interviewing is inherently difficult to evaluate.

If you are wondering, experiments do things like ask people to evaluate candidates based on interview transcripts, from candidates who have actually already been hired (among many other techniques). Ideally experimenters would have people do interviews, pick candidates, erase the memory of the interviewer, randomly select who to hire from among the candidates, and compare the candidates performance using accurate, objective criteria versus the ratings given by the interviewers, but that experiment is hard to pull off.

So, though I haven't seen FizzBuzz specific research, it is the kind of thing which can show effectiveness in scientific testing. If it screens out 95% of candidates then I'd be curious as to why - it was invented to be so absurdly easy that only an incompetent programmer could fail to pass it, and it's hard to see how to dumb down FizzBuzz. I wonder what percentage pass the "Hello world" test? The "make the computer do something, anything" test?

psr
  • 12,866
5

FizzBuzz isn't about identifying good programmers, it's about eliminating bad ones/fakers. If a candidate can't pass the FizzBuzz test then they don't know how to program and shouldn't be applying for a programming position.

nicholaides
  • 793