How relevant are Brainbench scores when evaluating candidates?

Question

I've seen many companies using certification services such as Brainbench when evaluating candidates. Most times they use it as a secondary screen prior to interview or as a validation to choose between candidates.

What is your experience with Brainbench scores? Did you try the tests yourself, and if so do you feel the score is meaningful enough to be used as part of a hiring process?

---

Difficult choice.

Consensus seems to be that BB cert are not very good as a certification. The biggest argument was around the fact that some of the questions are too precise to form a good evaluation. This view can probably be tempered somewhat but still, to hold someone's future solely on the results of this evaluation would be irresponsible.

That said, I still think it is possible to use them properly to gain additional objective knowledge on a candidate's level of expertise provided the test is done in a controlled environment ensuring that all taking it stand on equal footing.

Thus I went with the answer that best reflected this view keeping in mind that it is still just an hour long 50ish multiple choice question to evaluate skills and knowledge that take years to acquire. To be taken with a grain of salt !

In short, the tests have value but whether or not they are worth the money is another debate.

Thanks all for your time.

Answer 1

I took BrainBench tests for C# and Java some years ago and thought they were completely useless. I scored well, but the questions were largely about details of syntax and the class library. Incompetent programmers can memorize this information and pass these tests. Competent programmers learn the features of the language (e.g. C# has delegates, Java has anonymous classes), and the general contents of the standard library, and look up the details as needed. In my opinion, these tests measure exactly the wrong thing. If you want to hire a programmer, you need to see them program.

Answer 2

I think BB certs make good sense as part of a hiring process - provided you can guarantee that the guy passing these is indeed the one you're interviewing (at BB it is easy to screw up identity and make someone pass the test for you).

• My first encounter with BB was at interview when they gave me OO concepts test. Since it was at their office, and since the BB session was established at their account, this procedure left no chance to trick exam results. Based on my further experience with BB I believe this is the only reliable way to certify a candidate.
   
  I did not receive my results back then because these were delivered to guys who interviewed me, but anyway exam questions and overall BB approach got me curious enough to give it more try...

I tried all the free exams that made me interested, 5 or 6 or maybe 7 (it was few years ago, I don't remember how many) - Java SE and EE, RDBMS, business writing, typing.

• Scores I received felt quite relevant: I got Master's in areas where I felt my knowledge was solid, average scores where I was not so good, and could not certify at exam for typing which has proven that I am not a professional. Also, Java SE score at BB felt pretty consistent with one I've got at SCJP - which in turn felt pretty accurately reflecting my strengths and weaknesses.

My overall impression of BB is that as interviewer I would feel comfortable using these certifications as part of candidate screening - provided that these were taken in a way that guarantees that candidate identity was not screwed.

Answer 3

Honestly, from the few people I've interviewed with Brainbench "qualifications", I would say that the 20-minute multiple-choice test that I built myself, on eSkills, was a more effective filter. And that wasn't much of a filter at all, it just saved me phone-interviewing people who were going to embarrass themselves.

The only way that I use Brainbench as a filter is to filter out CVs where people try to pass it off as a professional qualification like an MCPD. That seems dishonest to me.

Answer 4

I'm not a big fan of these tests because they more of a trivia contest than an actual test of programming ability. Some people simply don't perform well on these tests for various reasons not related to their programming ability. I tend to have trouble with them myself due to my vision problems and my typical programming thinking process.

I also think they're biased toward recent college grads who'll have trivia they learned in class not that long ago more clearly in mind and are used to taking tests on a regular basis than someone who graduated 10+ years ago or who is largely self-taught.

I would consider using such a test for a junior level developer position but I wouldn't necessarily use it as the sole determination of who gets an interview or not. I probably wouldn't use it for a mid-level or senior developer position. For these candidates, a phone or face-to-face interview where they are given an open-ended opportunity to discuss their previous projects and programming knowledge are more productive. I think asking such a programmer why they chose X over Y in developing a project is more revealing than their ability to regurgitate 40 multiple choice questions on programming that may not be relevant to the work you need them to do.

Answer 5

Brainbench is a good way for a non-technical person (e.g. recruiter) to provide a first screen before dropping a resume on a technical manager's desk. They shouldn't substitute for a true tech screen however.

Also, I've taken a few brainbench certs and found many of the questions are too mired in the minutiae of the framework. I passed nonetheless but don't think the questions asked really qualify a candidate as being proficient with development.

Answer 6

I took the Brainbench Java and XML tests in 2003. I thought they were kind of hokey. But in a lot of job interviews, the homebrewed 20 multiple choice questions tests are even worse. They're invariably composed and edited in Word, and the auto-capitalization "feature" makes "string" into "String" and "boolean" into "Boolean". The latter can really make a difference in Java code, and turning "int" into "Int" can render C code where you're supposed to pick out the error into a confusing maze.

As an interviewee, I take those tests with a grain of salt. The goofier the test is the less I want to work there. So: testers beware. You're giving away information, and unless you've formatted extraordinarily carefully, your candidate may find out something undesirable about your company.

Answer 7

I had to take one not long ago and I think this type of test is more for weeding out candidates than to see whether you want to hire them or not.

I had 30 questions, 3 minutes for each and the test encouraged the use of references, like books or the Internet. Most of the questions were too specific and I would have not been able to answer them by myself, but with the use of references, I did very well. The test was great for deciding whether a candidate is great at quickly researching what they need to answer a multiple choice question, but nothing more.

I would definitely call someone in for an interview if they did well on the test. Would I reject someone who did really badly (below 60%?) Probably! Would I reject someone if they did mediocre? (70-85%) Probably not. So I guess it does have some weight in the decision making process, but not much.

Answer 8

I have also took one Brainbench exam regarding J2EE. As a first impression it appeared to me not much related to actual programming. This is like knowing everything about J2EE/Java which in reality is impossible and one cannot claim ever in life.
As per my experience a developer merely touches one aspect/side of any programming language like as my example, I have mainly worked on applications that have a front end in the face of HTML forms and using some framework example Struts/Hibernate I manipulate that information from/to some database.
I guess so far I have touched just one dimension/aspect/corner of java and I don't think that I ever would encompass all features of the ever growing Java.
Point is, the core concepts regarding language semantics and in particular OOP constructs should be very clear to any experienced developer.
If he/she knows what to implement where or which is needed where, then he can be a successful guy. No one has the memory to cover everything related to any language. In fact you just need to strengthen your foundation concepts. Once they are clear, rest is research and development (RnD) work. It all boils down to the fact, how much effort you contribute towards problem resolution using an initial set of tools and then working your way gradually towards the more sophisticated ones.

Answer 9

I've worked in to the Information Technology fields for 20+ years. I started when there were not "any" IT Certifications. I actually have a few BrainBench certifications as well as other "Vendor Neutral" certifications, and I use them on my resume or on web sites. My Information Technology experience spans not only security, networking and hardware and software but also programming. My real world experience is what companies look for first, followed by the certifications that I have. The tests are comparable to other IT vendors, and are worth taking, no matter what your current skill set is.

Answer 10

Working as a technical lead at NASA on the Space Station, it became necessary for me to become certified in IT Security as well as Solaris Sys Admin. NASA required a Brainbench certification at that time, and paid for the three certs I was required to take. This in and of itself speaks volumes as to the opinion NASA has for Brainbench. Since then, I have lost my certs after 3 years, and now NASA does not pay for certifications, and even my employer will not pay for them, so since I have to pay for it out of pocket, I find their prices reasonable. Note that most certification sites do not meet the criteria for reimbursement from employers, since they do not issue CEUs, and do not have traditional accreditation. My employer will pay for prepping for the exam, but not for the exam itself. The yearly subscription at Brainbench is a great deal, if you need multiple certifications ($199 normally, but I got it on-sale for $99.50). Their single certification costs is $49. By comparison, if you want an MSCE for instance, that will cost $72 if you use the outfit recommended by Microsoft. Brainbench requires an extra $15 for a nice printed certificate, so the total price is normally about $65 with certificate. The subscription allows you to take as many certs within one year as you want, but you still have to pay the $15 for the certificate. The tests generally are difficult, and certainly demonstrate the proficiency of the test taker.