Have you ever been developing something and just came to a point where you think that this is rubbish, the design is bad and although I will lose time it will be better to just start all over again?
What should you consider before making this step? I know it can be drastic in some cases, is it best to just totally ignore what you did before, or take some of the best bits from it? Some real life examples would be great.
although I will lose time it will be better to just start all over again?
No.
I have discarded bad software without a second thought.
I rarely think I will "lose time" by discarding rubbish software.
It's rubbish. Discard it. It doesn't matter how precious it may have seemed because some effort has been invested in it. If it's rubbish, then all further effort is simply creating more cost not creating new value.
What should you consider before making this step?
The revised design.
How to preserve the data.
I know it can be drastic in some cases,
False. Rubbish is rubbish. Discard it early and often. It's not drastic when the software is rubbish.
is it best to just totally ignore what you did before, or take some of the best bits from it?
That's imponderable. It depends on specifics of what you did before and how bad it was.
Our current web site uses a trailing application_# naming scheme. The number is the major version number. When we make a database change, we -- essentially -- discard the entire previous version of the application. It's tied to the database pretty tightly.
We don't "incrementally" change the database. We discard the previous version and move forward.
Two of the modules is on major version 3. Two previous discards of rubbish software.
One of the modules is on major version 2. The previous had to be discarded as unworkably bad.
These are discards of the previous version and migration of the data to new tables with a different structure (hence the major version number).
We do it all the time. The quicker you get rid of rubbish software the more effective you become.
I do not think it is wise. Joel thinks so too. Mature code is brittle and tend to have many hacks. But usually mature code tend to be correct too. It is foolish to discard correctness in favor of design purity. Design purity means nothing to the end user, correct software means a lot.
If you feel the design is rubbish, identify the most rubbish part and mend it. To put it in another way, start mending broken windows instead of building a brand new house. The advantage of fixing bit by bit, is that you always have a working system. You have test cases (hopefully) to rely upon to verify correctness. It may take longer than building it from scratch, but you will always have correct software to ship.
This happens all the time.
A criticism of the Waterfall Model is formulated as “Many of the [system's] details only become known to us as we progress in the [system's] implementation. Some of the things that we learn invalidate our design and we must backtrack.”
You should be careful though to know this effect from just being frustrated that you don't reach your goal as soon/easily as you've hoped.
I have been in this field for over thirty years, and I still believe that planning to throw one away is still a solid software engineering practice. Eighty-percent of a useful system's lifetime is spent in maintenance mode. Brittle systems are more expensive to maintain than non-brittle systems. However, one should also be mindful of another Brooksism; namely, "The Second-System Effect."
First thing I normally do in such a situation - take a deep breath and have a break.
Have some fun with friends, go to gym, visit a concert - anything to free my brains from programming. Then look at the project next day. If it's still bad, start it all over.
But don't remove old repositories. I find some handy tricks even in my earliest and most noobish code. And of course it will remind you in future how much you learned.
I think this happens very often, and there are ways to beat this. That's precisely the reason why prototypes exist. And beating them is easier too, you only need to get a bit organized and a little disciplined.
The way to look at it is, programming is fundamentally all about problem solving. So you must start out with a clearly defined problem on paper. No ambiguous definitions allowed. Once you do that you must decide what are the probable solutions to it on paper. Keep planning minimal but sufficient.
After this begin with the first set of features, release them, test them. Take feedback, start with the next set of features. Every time something goes wrong you know where to correct it. I think this is what we call as agile, albeit there is more to agile.
Starting a large project as whole going half way and then discovering things going wrong is pretty much expected.
I believe in the "third implementation" law. First implementation is always a prototype-quality. Second round of rebuilding from scratch addresses the most basic issues found in a prototype and finally shows fundamental design flaws. That flaws are addressed by a third reimplementation-from-scratch. It should not necessarily be your own re-implementation, a competing system may address your issues as well.
If it is so bad that fixing bugs and adding new features is unacceptably slow, you have to do something. Otherwise, don't leap into action yet.
Either way, it's worth making a plan.
You have to start out with declaring what you are trying to achieve and what is an acceptable level of success. Otherwise rewriting might descend into an endless cycle of refinement of increasingly unimportant bits of code.
The next step is to sketch up a design that fits the previously established criteria.
Then you have to try to figure out how to break down the changes into smaller steps so that you don't have to stop everything for a year while you're busy rewriting everything from scratch. If the original system's modular enough and the modules are at least laid out correctly, you can almost certainly find a way.
Refine your design based on the result of the previous step and check if it still meets the requirements.
If the situation is really bad, you can start to implement it, if it isn't, file it away as a plan for the future.
If it's something with a bad design or implementation, it will make you lose more time than if you do it properly.
Every small change that you try to do will take longer than needed because of bad dependencies, isolation, coupling, etc.
If you do it from scratch, you do things faster, better, easier, your morale will go up and your productivity will go up too!
You have to think about one thing first: rebuild it as a whole or a subpart of the system? The latter is easier, faster and less risky. After one part, you choose another one to improve and keep going until you've got something good! The former might produce better/faster results but you also increase the chances of getting stuck somewhere and not being able to do it all, wasting a big amount of time!
When I was doing my master thesis I completely rebuilt my application three times - which wasn't the main point of the thesis - but these changes were totally worthy!
Each time I did it all over again, I learned something new, I improved something, I realized something could be done better the next time, etc.
It took me a lot of time (specially because I was doing these improvements by myself, under my responsibility - if things would have gone wrong, it would have been my problem and my time lost).
If the code was written properly with tested units that are known to work, you can keep those parts safe in the knowledge that they do work, and use it when you need it.
It may be possible to decouple it enough to extract the bits you need as usable.
Throwing everything away is usually a bad idea. Rewriting parts of the system may be a good idea. Refactoring is often a good idea.
The way I see it, the question is not whether the code is good or not. Code you wrote more than a week ago is always a terrible mess because you have since learned something new which you wish you had used.
The question is whether the tech debt you've accumulated is making it hard to get work done. If your system is hard to run or changes are harder than they should be, that means you probably need to fix something. But unless things are difficult across the board, don't rewrite everything. Just rewrite the parts that are causing you pain.
