Tricky logic puzzles - Are they really useful in assessing programming skills?

Question

In the last interview I attended, I was asked to solve a puzzle where I was expected to measure exactly blah liters of water given two buckets with capacities - blah and blah liters respectively. I was unable to solve the puzzle in given time (~5 minutes).

The interviewer was a bit disappointed and said a programmer has got to have "these" skills. I didn't get what skills he was talking about.

I have always felt strange about these kind of puzzles that are normally asked at programming job interviews. I do not understand what, if at all, is the connection between such puzzles and programming. Exactly what skills do interviewers intend to assess with such puzzles?

Answer 1

Some people ask them in an attempt to gauge your ability and approach to solving problems. Personally, I don't think that such puzzles provide an accurate indicator. In the "real world", you have more than five minutes to figure out if your dealing with a bin packing vs a back pack problem, for instance. Initially, it's sometimes easy to misunderstand the problem at hand until you're in the middle of applying the wrong solution. That happens to people with 1, 5, 10 or even 20 years of experience.

The best interview 'puzzles' are the ones where you sit down at a computer to solve a problem in the domain in which you claim expertise. I also dislike the "Well, a programmer should be able to ..." thinking because it doesn't take into consideration that people get anxious when hit with something unexpected in a setting that is already stressful. Sure, you could solve that if you had time to think about it.. and perhaps you could solve it faster if you realized that your life would be over if you didn't. Do you want to work somewhere where your life will be over if you can't solve problems in five minutes? Will you get fired if you can't?

Should all great programmers also be champion sudoku solvers? I'm sure that plenty are, but it's not like some kind of prerequisite for competency.

I'm not saying that you should not be tested on how you approach problems, but the tests should be fun and invite the 'best' that the applicant has to give, given their area of expertise. Proving that you are as smart as a character that Bruce Willis portrays seems kind of pointless, considering that producers spent a pretty sum to get that scene just right.

In other words, if you detect that you're being interviewed by someone who has little comprehension over what you'll actually be doing, excuse yourself to go to the restroom and never return.

Answer 2

Microsoft began using these questions in the early 1980s. As Microsoft became notably successful, other companies began to copy them, but a couple of key points got lost in translation.

At the time, Microsoft was trying to fill a lot of technical but non-programming positions: technical writers, testers, phone support, etc. These were not common jobs back in the day, and folks with actual experience in these areas were hard to find. As an alternative Microsoft thought they could hire really smart, clever, flexible people, and train them in the job. Since these folks didn't have a programming background, asking them programming questions in the interview was pointless. The riddles were chosen to try and pin-point folks who were clever and had exceptionally good analytical skills. Programmers were generally given white board programming problems, though they might also be asked riddles over lunch or dinner.

These questions were never meant to be pass-fail. They were intended to be the start of a conversation about how you would tackle the problem, and how you thought about problems you'd never seen before. The only sure way to "fail", was to refuse to try to solve the problem. At the time this was a novel strategy, and you couldn't just look up the questions on Google.

Edit:

Sometime after writing this answer I read The Computer Boys Take Over, a history of institutional computing in the 1950s and 1960s. Apparently the practice of asking brain teasers and riddles of candidates for programming jobs goes all the way back to the 1950s. The US was trying to computerize its air defense system and IBM estimated that they would need several thousand programmers to do the work. The response was shock and consternation: there were only a couple dozens of "professional programmers" in the entire world. Several approaches were tried: abstract programming aptitude tests, recruiting mathematicians as programmers, recruiting chess players and crossword puzzle solvers, and screening applicants with riddles and brain teasers.

They did eventually succeed in recruiting enough programmers to complete the project, but the conclusion was that none of the screening methods was any better than chance at identifying the recruits who went on to be notably successful as programmers.

Answer 3

Are they useful? No, not really. They were once so common at Microsoft they even got called "Microsoft questions", and there have been books written about them, this one is actually quite a good read..

There are 2 problems with them. Firstly, if the applicant does research (and reads the book) they'll know them anyway and second, even if they can solve them how does that show that the'll be a good dev/test/PM.

For these reasons they are seldom asked anymore at Microsoft. It is far better to ask coding questions, or problem solving questions that do not require a "trick" answer. In other words you need to ask questions that allow you to explore the skills and behaviour of the applicant as they try and tackle the problem - as an interviewer I want them to ask questions, come up with solutions and then back track when they figure out a problem, maybe not even find a solution in the time they have but at least go about it in a sensible way. That reflects real life work. I have never had to measure 3 pints using 2 buckets and a chicken (or whatever the question was).

That said, I was asked a couple of trick questions in my time, and I now regard myself as an expert in transporting chickens and foxes in small boats and calculating the lifetime of a fly that lives on a train. I've never had to use this information, but who knows, maybe one day....

Answer 4

You might want to read the book How Would You Move Mount Fuji?. It goes into the reasoning that many folks use riddles in interviewing, and my opinion is that it is a combination of cargo cult behavior ("Microsoft does it, and if we want to be as successful as they are, then we better do what they do") and fraternity hazing ("by gosh!, I had to answer those questions and you better believe the next guy is going to have to answer them!").

The history of these questions as an interviewing practice started with William Shockley in the 1950s. They were a reasonably common Silicon Valley sort of interview question that interviewers asked because other interviewers were doing it (and maybe they knew something that this interviewer didn't know?). Shockley intended them as an intelligence test, and the question with the 2 buckets was on one of the original Stanford Binet IQ test back in 1916.

Quite possibly, the people doing the interviewing actually want to see how you look for answers, so they'll pose impossible to calculate questions, such as how many gas pumps are there in your city. These sorts of problems are Fermi Problems. Two interesting blog posts from Jeff on this subject are The Hardest Interview Puzzle Question Ever and How Good an Estimator are You? Part III.

Personally, I have a low opinion of these sorts of questions as they generally are used by interviewers who don't know what they are doing, nor how to look for developers. Unless you are going to work for a company that makes puzzles/riddles, they belong on the dustheap of history along with "what is your greatest weakness" (answer the truth to this and you end your interview in a bad way) or "why are manhole covers round" (not all of them are).

Answer 5

Other's have provided answers that I have upvoted as a matter of must. The reason I write another answer is because what I want to say will probably not fit in a comment, and because something has to be said about how a good programming job interview can be like.

In the first good interview I remember, we talked, a lot, without a hurry. First for an hour, on the phone, about object oriented design and the pros and cons of implementing it in C++. Then, on site, I spoke with several people about their software development practices, integration, testing, version control, and configuration management, about teams and responsibilities, about technology and about design. It was a whole-day interview that included lunch with the folks that interviewed me. In hindsight, it was all about if I would productively fit in what they were already doing.

Ever since, the good interviews have all been long, one to two-hour conversations about software development. There have been no problem-solving questions, no puzzles, and no coding challenges.

If I were to interview someone for a programming job today, I would proceed in the likes. I'd request opinions about a breadth of topics, and leave depth aside:

1. Which are your programming language preferences? Why?
2. How to approach exception handling?
3. Aren't the benefits of layered design a myth?
4. Isn't continuous integration a burden to efficiency?
5. Whomever wrote a piece of code should own it, Right?
6. What do you do to get into "flow".
7. How should reported defects be included in a project plan?
8. ...

Those are questions with more than one answer, and they are all about topics about which a software developer should have an informed opinion. I wholeheartedly agree with the answers that mention previous real problems experienced as a conversation topic (not as questions).

The more scientific studies about effective software development since Peopleware say that the best programmers are those who understand the dynamics of software development, even if they don't have the highest IQs. I'd rather take a rookie that is eager to learn than someone with n years of experience that boil down to 1 year of experience repeated n times. My personal bias is towards candidates that tend to think outside the box, and at the same time know how to fit into the current (my) box.

Answer 6

They can be useful in assessing problem solving skills, which is of course one of the key aspects of programming.

As an interviewer of many people over the years, I don't usually ask gotcha type questions quite like the one you seem to describe, but I may well ask something and ask "how would you solve...".

My expectations in that case are to hear you articulate your approach to the problem. What other data would you try to gather? How would you test your hypothesizes, etc.

Answer 7

These are just voodoo hiring practices. Other people ask these questions so they feel like they are supposed to. They know that not answering the question is "bad" and answering is "good", but they can't tell you why beyond non-answers like "a developer needs these skills". They are a waste of time and an indicator that the interviewer is not a competent interviewer.

Answer 8

That's the old-skool rationalle that you have to have basic logic skills; anything else can be taught. But that's not entirely true. Reading Boolean logic, conditions and loops, is not the same as being able to solve a logic puzzle.

That said, in the days of procedural languages, it was probably true that someone who could solve these problems would have a higher propensity to be able to apply any problem in terms of switches. But to my mind, OO/Functional programming requires an engineering mindset, which is quite different (though not contradictory).

Personally, I'm not sure I'd want a job with a company who still thought logic was more important than practical programming skills.

Disclaimer: I'm very good at logic puzzles and probably wouldn't have got my start in this line of work without this rationalle.

Answer 9

The interviewer must have been referring to problem solving and logic skills, which is part of the everyday work of a programmer. When given a problem, you need to be able to analyze it, subdivide it, and write a solution for it by using the most optimal approach.

You could argue how well a puzzle like that represents your ability to do this. I don't see the merit of asking a puzzle question instead of just asking a real life programming problem.

Answer 10

Two points:

1. Programming is mainly different from puzzle solving. It is perfrectly explained by Steve McConnell at "Code Complete":

   What? You don’t have to be superintelligent? No, you don’t. Nobody is really smart enough to program computers. Fully understanding an average program requires an almost limitless capacity to absorb details and an equal capacity to comprehend them all at the same time. The way you focus your intelligence is more important than how much intelligence you have. As Chapter 5 mentioned, at the 1972 Turing Award Lecture, Edsger Dijkstra delivered a paper titled “The Humble Programmer.” He argued that most of programming is an attempt to compensate for the strictly limited size of our skulls. The people who are best at programming are the people who realize how small their brains are. They are humble. The people who are the worst at programming are the people who refuse to accept the fact that their brains aren’t equal to the task. Their egos keep them from being great programmers. The more you learn to compensate for your small brain, the better a programmer you’ll be. The more humble you are, the faster you’ll improve.

2. Such puzzles can be useful during interview, but Only if the interviewer looks on the Process, not result itself.

But ideally the puzzles should be more complicated and programming related (like small 2-hours project), in my opinion. The thing is that interviewers are people and doesn't have perfect "interview skills".

Answer 11

Programming is not about writing lines of code, it is about solving problems for and from other people (customer, user, etc).

It happens that for programmers the solution takes the form of a program.

So it is why it is important to have problem solving capabilities and why it is tested.

That being said, I am not sure that solving tricky puzzle is the best way to assess someone.

Answer 12

Puzzles in interviews fall into two categories: "logical puzzles" (like the one you were asked) and "think differently" category. The think differently category (I not sure are they also called lateral puzzles?) are usually this type: How many leaves are there in that tree? or How many tailors are present in your city?

I am fine with "Logical puzzles" because they have one or maybe two solutions at most and can be arrive at by straightforward logic. And I believe that logical puzzles are good to an extent because the process needed to solve them is very much the way a coder needs to think in real life.

The "think differently" kind bugs me no end because they force you to make assumptions, and then, make some calculations based on the assumptions. Simply put, if your interviewer agrees with your logic but not with your assumptions, or vice-versa, you've lost. There is too much room for the interviewer to disagree with your solution.

When I take interviews I don't ask logical puzzles. Reason: Most candidates even those with 3-4 years of experience, fail or give up when I ask them to code simple textbook problems such as Fibonacci series or palindromes.

The problem with puzzles, either way is that the not-so-good programmers know that just by looking up solutions to such common puzzles on the net they can impress interviewers. Very few people will be honest enough to tell that they already know the solution.

Answer 13

There are a couple of different ways to examine such problems:

1. Knowing a previous solution. In the movie... Die Hard with a Vengeance ... explain this to me...? being an example of knowing a solution for the case where the blahs are 4,3 and 5 respectively. Some people will be able to quickly tap their internal knowledge of a past solution and adapt it if needed. This is usually what I believe an interviewer would expect which may or may not be a good idea.

2. Creative improvisation skills. This would be the case if you don't know a previous solution or even recognize the problem as being something that one could model as a Diophantine equation. Thus the question is how quickly can you use what is given and find a solution to the problem in a creative manner along with explaining why what you have is a valid solution to the problem.

Either could be what gets one past the question in a satisfactory manner though in each case there is also a bit of a test of one's communication skills as one could also try a reply of, "Is this really relevant to the position that I'm applying? When were these skills last used?" that may lead to an interesting dialogue if the interviewer will open up about what exactly they are wanting to see that maybe an alternative approach could be seen as more effective here.

Answer 14

It's not a particularly tricky problem. Only three steps are required, and there are only two choices at each step. I would be surprised if any of my colleagues were unable to solve it in very short order. We don't present such problems in interviews, but I think it is reasonable to ask such questions. They are certainly more useful than detailed questions about syntax or libraries.

OTOH, I think that programming problems are more useful.

Answer 15

You have to remember that there is no way to know with absolute certainty that someone will be good at a job. Especially a CS job since many challenges that the job might have in store can't be predicted.

So the potential employer must guess on your future performance.

Degrees, recommendations, and GPAs can be obtained with time/effort and social engineering, work experience can be embellished and/or false, and standardized tests are frankly too basic to be overly indicative of ability. So the resume may give some indication of effort/commitment levels in your history, but none of it will speak to your actual ability to solve hard problems that come up in the world of computer science. I can't think of a better way to predict that kind of ability than a few good logic/mathy/CSy puzzles.

Remember it's a guessing game, and the reality is that all things equal any of us would rather hire someone able to solve those puzzles than one that can't.

Yes you could study interview puzzles, but I think you'll find yourself burned if the puzzle given doesn't match the ones you study... and so long as you don't memorize the puzzles and their solutions, maybe studying the puzzles themselves will make you a smarter person in a real way, like any real education should.