How to rebuild a large legacy system

Nice and catchy title, isn't it? First a disclaimer: I've never got the chance to try this in real life.

I was having lunch with a colleague working in projects orbiting an ancient mainframe-system originally written in 1980's (we were having Nepalese food btw, my favorite). The problems they were having in his company are very common. The legacy-system was originally written by a small team - probably less than 10 people - and through the years the size and role of the system grew to something like a CRM. Yes, this must sound very familiar.

Now the people who wrote the system are closing 60 years of age and are planning retirement. The system will probably die as the last original developer leaves the company. How do you cope with a situation like this?

The most common solution to this problem is to buy a very expensive CRM-software and start a small integration-project that aims to replace the old system as-is. Usually about two years later you have an enormous, failing integration-project in your hands and instead of one, you now have two legacy-systems and even the original one is now burning more man-hours than before because "slight" modifications were necessary.

So how would I do it? Here's how:

I'd assemble a small team with the best developers I could get my hands on. I'd provide them with the best tools available and seat them in a comfortable office with free drinks and, most importantly, the original developers and experts with the knowledge of the legacy system. I'd start small, something like reproducing one core function of the old system. The developers of the new system could ask 100 times a day how some minor detail works. The new team could write acceptance tests from what they've learned of the old system.

Then maybe a year later the old and the new system could start to run side-by-side, some transactions of the old one routed to the new one. This is what is done with plants (hardening) so why not with software. One function at a time the new system would then replace the old one and grow to a complete system with all the functionality of the legacy software working as it did before, the difference between them being that the new one is done with (hopefully) a modern, flexible and light architecture that enables it to grow along with new business requirements. And what's even more important is that you now have a new team of people who know the new legacy system inside-out.

I wonder if it was Brooks again who has said that there are only two kinds of software-projects: ones that fail and legacy nightmares. As bleak as that sounds, in my opinion, a legacy-system is a nightmare usually only after the original developers of the system are no longer available. So after the system is done, make sure the developers have no reason to change jobs! Do not disassemble the team!

The lesson here, in my opinion, is that large systems cannot be built. They are grown instead by a long-living team. Anything else and you're probably going to fail.

My next blog will not have anything to do with legacy-architecture, I promise :)

Back To The Future

No. This blog entry isn't about the new game about the old movie with car-enabled time travel. Instead try to think back to late 1970's and early 1980's. Ready? Good.

Banks and other businesses with intrinsic need for computers were, at latest, building their information systems. CICS-systems with whopping 64 kilobytes of memory was pretty much the standard fare. Debugging meant printing out the core dump (after all it was ~64 000 characters, not that much). Used language was Cobol with static memory allocation. It's amazing to think it was possible to create large banking systems with that technology, isn't it?

What's more amazing is that many of these systems are still in use. If you use a cash dispenser (at least here in Finland), it's very likely that the transaction is eventually run in a CICS-system. Only companies that have been established after 1990's or so have more modern systems.

The question to ask, of course, is why haven't these antique systems been rewritten with Java, .Net or Haskell even? It's not from lack of trying, I assure you. I've personally witnessed a few very large projects and heard of many others. They were all failures by most standards. There never is one simple answer as to why a software project fails, but one question especially has haunted me for a while now: Why can't we seem to succeed in rewriting software that was originally done with such limited tools?

My theory about this is that back in the past, developers were able to concentrate better on the essential complexity of the software they were creating.

Commercial vendors have clouded us from the essential complexity of any given business-domain with process engines, portals, executable models, frameworks, predefined domains, predefined development processes...on and on. And the problem isn't just commercial vendors. Let's say you're going to use a very popular open source tool Grails to do your project (note, I have nothing againts Grails, I actually like it very much). Here's an incomplete list of techonlogies and frameworks you'll need to master in order to get a large project done:

Groovy, Java, JavaScript, XML, Ant, Gant, GORM, GSP, Spring core, Spring MVC, Hibernate, Log4j, HTML, jQuery, YUI, HQL, SQL, Ivy, Maven (repositories).

I used Grails as an example here because it's easy and productive compared to many other options. Still, the sad fact is that we're drowning in tools and frameworks. And as developers and architects, we've been conditioned into thinking we're helpless without them.

What many don't seem to grasp is that every framework, every library you add to a software stack adds accidental complexity which makes it more difficult to concentrate on the essential complexity. And no, the correct solution is NOT to start rolling your own framework. The solution I'm proposing is to minimize the use of frameworks and especially commercial products in an architecture.

I'll give an example: About a year ago I created a small ERP-solution from scratch. Sounds crazy, right? ERP's are monstrously complex and even starting to use one in a company takes huge amounts of effort. Well, you know what, it's not that complex when you can concentrate on the essential parts of it - the parts that matter to the problem at hand.

I ended up creating a very light (micro?) architecture where the domain was running on JPA-annotated classes in an embedded Jetty-server exposing REST/XML services using a small tool I created for creating XML-messages imperatively. The point is that the following code snippet (a whole service for returning all persons in the database) only requires Java 6 runtime, nothing else. No frameworks to learn or debug, no added complexity.

public void all() {
        ensureLogin();

        Message m = okResult();

        Query q = jpa().createQuery("select p from Person p");
        List persons = q.getResultList();

        for (Person person : persons) {
            m.set("person/@name", person.getWholeName());
            m.set("@id", person.getId());
            m.parent();
        }

        write(m);
    }

I'm not suggesting to stop using frameworks. I'm suggesting to evaluate very carefully whether they're worth the accidental complexity they bring. Do everything possible to minimize accidental complexity while providing developers tools to express the essential complexity!

And learn to make the difference between a tool and a framework. I will blog about tools vs. frameworks at some point. However I think my next blog will be about how (in my opinion) legacy systems could be rewritten successfully.