how to convince other we should move to hadoop?

Question

Everything I've read about Hadoop seems like exactly the technology we need to make our enterprise more scalable. We have terabytes of raw data that is in non-relational form (text files of some kind). We're quickly approaching the upper limits of what our centralized file server can handle and everyone is aware of this. Most people on the tech team, especially the more junior members of the tech team are all in favor of moving from the central file system to HDFS. The problem is, there is one key (most senior, etc.) member of the team who is resisting this change and every time Hadoop comes up, he tells us that we could simply add another file server and be in the clear.

So, my question (and yes, it's really subjective, but I need more help with this than any of my other questions) is what steps can we take to get upper management to move forward with Hadoop despite the hesitation of one member of the team?

Answer 1

Why are you in favor of this move?

• Does it make life easier for the team?
• Does it help users in managing, accessing files?

Unless you have a technical person who is a decision maker chances are they don't care about who favours what. They care about:

• ROI (Return on Investment) and how long will it take to get there

Would moving to this new system require re-organization or restructuring things? This can be very expensive.

Being a programmer I understand that everyone (except one) may be in favor of something, but the business side of me is saying there needs to be justification for it.

Create a case for moving to the new system. Some good things to include would be:

• What will the ROI be?
• How long will it take to switch everything over? (take your guesstimate and multiple it by 3 to get a realistic number - seriously)
• Provide real world cases and samples where the new system improved, helped, made life easier, etc for other companies.
• What will it cost to implement the solution? (be sure to include costs for hardware, software, programming, project management, etc)
• Compare the differences between your proposed solution and others (ie. just adding a new server). What are the cost differences? What are the long term effects? How long will an additional file server hold them over for?

Basically, to sum it up, take the time to do the research, show facts and figures, present your case.

N.B. Do not make one of your reasons "b/c the team is in favor of it", b/c it will not fly!

Answer 2

what steps can we take to get upper management to move forward with Hadoop despite the hesitation of one member of the team?

Solve the problem.

You can't "convince" people of anything with argumentation or science or even facts established outside the organization.

You need to actually solve an actual problem using Hadoop.

Once they have two side-by-side solutions (Hadoop and whatever you're using now) they can compare cost and complexity and performance.

It's not sufficient to say that Hadoop is faster or uses less storage. You must show how much faster and how much less storage.

Even then people will argue that you picked a particularly easy problem to solve.

Proof of concept is the only thing that works.

You can't often get permission to do a proof of concept. In a few companies that actually value innovation, you might get permission. In the rest of companies (which say they value innovation, but actually don't) you have to break the rules and do it on your own.

Even when you have a total solution in Hadoop and can show how much faster, simpler and how much less storage, you will still have to sell this to everyone. Prepare yourself for years of work. Patience is the most important part of making this kind of change.

If you're very, very lucky, others will see the advantages quickly.

If you're not lucky, others will resist actively.

[I was in a meeting just yesterday where the switch from Oracle to MySQL was actively resisted by some developers who were able to lift up one feature in Oracle that's missing in MySQL and claim that we could not work around that single feature. Sigh. Everything can be worked around. However, the resisters have raised a barrier in the form of another proof-of-concept on another application that uses that Oracle feature. What will be next? I'm sure there will be another barrier.]