Tampere Goes Agile 2013

I have a two year old tradition of going to TGA with a friend of mine. Its usually the only trip to Tampere each year for both of us. So I thought since going there is a tradition, why not writing a blog summary about the event as well.

I was really tired this year after only a few hours of sleep the night before so it might have brought some bias, but I didn't feel quite the same level of good vibes and community as last year. I'm thinking the venue (Tampere-talo) might have something to do with this. The place is ultra clean and areas between presentations weren't really good for sparking ad hoc discussion.

So here are the presentations that I saw and what inspired me.

Note I didn't see all of the presentations and who knows how amazing things I missed. Not being mentioned here is not a sign your favourite presentation wasn't great in my opinion.

Science!

Laurent Bossavit: The Art of Being Wrong. How good are your guesses and estimates? How well calibrated are you and do you ever check your assumptions after the fact? These were the questions Laurent brought up. I've written about this same topic earlier in my own blog - only more focused on testing.

Laurent is the author of a book I intend to read, called The Leprechauns of Software Engineering. Superficially it looks very similar to the book  What We Actually Know About Software Development, and Why We Believe It's True which I did read earlier this year. I believe the topic is important, but Wilson's and Oram's book was more like a collection of academic papers demanding more academic research about software engineering. Which is good and great, but honestly not a very useful read for a practicing engineer like myself. So I'm really looking forward to reading Laurent's book and will get back to the subject once I've read it.

Lean!

Next interesting presentation was by Marko Taipale. He was a presenter last year as well and did a very similar presentation about Lean Startup methodology. This time Marko drilled deeper and specifically into how his own company uses lean startup tools in developing their own business models. Lean thinking and Lean Startup -movement is a huge topic and I will not go much deeper here. Instead, I'll just provide a link to Marko's presentation.

Ok I will say one thing about Lean Startup -movement after all. Its starting to get real criticism as well which is always a good sign. It means its being used and people have opinions and hopefully even hypothesis about if and how well it works. To balance things out a bit, read for example some good critique by Dan Norris. Would love to have Marko Taipale and Dan Norris in a panel discussion about Lean Startup!

Contracts!

I'm a freelance contractor myself, been for about 3 years now. And even before that I was a consultant for 10 years doing hour-based contracting for various companies. Antti Kirjavainen from Houston Inc did a very thought provoking and important presentation about the current state of manhour-based contracting work and why it sucks so much for everybody. 

I'm hereby officially giving the Best of TGA 2013 prize to Antti! I might return and write a whole blog entry about the topic, but for now, here is Antti's presentation.

Contracting and how the dynamics in it are built right now will end up hurting everyone in our industry at some point. Delivering warmed seats instead of identifiable value and treating each developer as seat warmer for which the hourly price needs to be minimised, is a doomed model and needs to change at some point.

I was honestly too tired to have a chat about this with Antti at TGA, but I will definitely do so at some point if I get the chance.

#noestimates

Nice to have a word for what I've done intuitively as a team lead if there has been no pressure for strict Scrum or other Methodology! Henri Karhatsu did a very inspiring presentation from a real life project he had been leading, transforming the process from poorly working traditional agile planning to a much more streamlined custom process without estimates.

I got a few new ideas to try out from Henri's presentation. I really can't say if my next project will use estimates or not. Sometimes they are good. Sometimes they aren't that useful. There is no predefined rule on how to determine which is the case beforehand.

What I really liked about Henri's presentation was advocation of thinking for yourself. This worked for him, but might not work for you - at least not copied blindly. 

What often most ticks me off about Methodologies in software engineering, is that they are essentially attempts in externalising thinking. "Read this book and follow the steps and you won't have to think for yourself. We have solved this for you". 

Do what makes your developers happy and result in correct business need being fulfilled

This needs to be fine tuned and figured out in every company, in every project of any significance. That's just how it is. Learn to live with it. Hire a Henri to figure these things for you :)

And finally...

Very much thanks and hugs for the wonderful people who organised this free (!) event and the sponsors who paid for it all! 

My sincere suggestion for next year is that the event wouldn't be entirely free anymore. Now apparently over 60 people who had registered never showed up. Make it 50e a pop and people might not reserve a seat just in case they feel like going. 

Ajatuksia Apotista

This blog post is in Finnish because of the topic. 



Apotti-hanke on herättänyt ennennäkemättömän määrän keskustelua sekä Suomen IT-piireissä, että julkisuudessa. Mahdollisesti ensimmäistä kertaa yleiseen keskusteluun on noussut se, miten ongelmallisia ja kalliita julkisin varoin toteutettavat IT-järjestelmät ovat.

Olen mielenkiinnolla seurannut aiheen ympärille syntynyttä Facebook-ryhmää. Keskustelun laatu on korkeatasoista ja siihen osallistuvat sekä IT-alan että terveydenhuollon ammattilaiset.

Tämän kirjoituksen tarkoituksena on kerätä teemoja, jotka vallitsevat keskustelun taustalla - valitettavasti usein hukkuen yksityiskohtien ja tarkan argumentoinnin sekaan. Pyrin nostamaan tarkasteltavan ongelman sellaiselle tasolle, että voidaan päästää irti kehää kiertävistä argumenteista (esimerkki: "ketterät menetelmät auttaisivat - ei voi käyttää koska X, Y, Z"). Yritän myös tuoda keskusteluun muutaman uuden käsitteen ja idean, jotka ehkä jäsentävät aihetta niiden mielissä, jotka tämän kirjoituksen lukevat.


Millaista on hyvä ohjelmistokehitys?

Jotta voitaisiin verrata vaikkapa Apotin toimintamalleja johonkin, esittelen ensin oman parhaan näkemykseni siitä, miten järjestäytyy toimivia, kestäviä ja kauniita ohjelmistoja tuottava kehitysmalli. Listaan ominaisuuksia, joita tällaisella mallilla voisi olla. Mitä enemmän näitä saadaan käyttöön samassa hankkeessa, sitä varmemmin se tuottaa arvokkaita tuotoksia.

1. Ekosysteemeitä hierarkkisen hallinnon sijaan.
2. Varhainen ja usein toistuva altistuminen todellisuudelle.
3. Osapuolilla on tasapainoinen riskijakauma. 
4. Ohjelmiston rakenne tukee kohtaa 1.

Katso mitä Apple, Google, Facebook ja muut uuden sukupolven teknologiayritykset tekevät, niin ymmärrät. Onko erityisiä syitä, miksi näin ei voitaisi toimia julkishallinnon IT-hankkeissa? Varmasti tuhat syytä. En vain usko, että mikään niistä kestää tarkempaa tarkastelua.

Ekosysteemeitä hierarkkisen hallinnon sijaan

Julkishallinnon IT-hankkeissa istuu norsu keskellä huonetta. Kaikesta muusta keskustellaan, mutta silmiini ei ole osunut kritiikkiä sitä kohtaan, että hankkeita vedetään täysin ylhäältä alaspäin johdettuna. Korkeintaan vaaditaan parempaa osaamista sinne norsunluutornin huipulle.

Parempi suunta olisi pyrkiä minimoimaan hallinnon määrä ja keskittyä siihen, että pystytään reagoimaan nopeasti ärsykkeisiin.  

Tähän päästään mielestäni niin, että luodaan pieni ja erittäin asiantunteva organisaatio, jolla on niin korkean tason mandaatti "tehdä kaikki tarvittava onnistumisen varmistamiseksi", että asioiden hyvin tekeminen ei muutu mahdottomaksi.  

Tällainen organisaatio omistaa kaiken koodin, infran yms. mitä hankkeessa syntyy. Erityisesti tämä organisaatio vastaa ns. ytimestä, joka löytyy jokaisen hyvin tehdyn tietojärjestelmän keskeltä. Käyttöjärjestelmissä sen nimi on "kernel", potilastietojärjestelmässä se on potilastietojen optimoitu tietomalli, sen toteuttava tietokanta ja nämä kaksi yhdistävä logiikka. Tämä ydin on suojattu rajapinnoilla, jotka tämä organisaatio omistaa ja joiden kehityksestä ja muutoksesta se vastaa.

Jokainen ekosysteemi siis tarvitsee ytimen. Applella on AppStore. Linuxissa on Linus Torvaldsin hallinnoima Linux Kernel. Tällaisen ytimen päälle voidaan tuottaa erilaisia laajennuksia, käyttöliittymiä, sovelluksia. Mutta niin, ettei mikään yksittäinen osa pysty rikkomaan kokonaisuutta.

Tällaisessa mallissa pärjätään pienellä ydinorganisaatiolla, muu työ voidaan kilpailuttaa. Mikä parasta se voidaan kilpailuttaa pienilläkin yrityksillä, jolloin ei olla jumissa kahdessa tai kolmessa toimittajassa, joiden välillä on ehkä kilpailua, mutta ei juurikaan eroa lopputuloksen laadussa tai hinnassa (lue: huono ja kallis).

Kun tähän lisätään hyvin tehty käyttöliittymästandardi, ei pitäisi käytettävyydessä olla suuria eroja eri osajärjestelmien kesken.

Varhainen ja usein toistuva altistuminen todellisuudelle

Sähköisen reseptin viimeaikainen uutiskynnyksen ylittänyt kompurointi on tästä mainio esimerkki. Hanketta on jossain muodossa kehitetty tai valmisteltu 1980-luvulta asti. Vuonna 2013, kun se on saatu pikkuhiljaa käyttöön, käy ilmi että järjestelmä ei ihan vastaakaan todellisuuden tarpeita.

Ei. Oikea vastaus tähän haasteeseen ei ole entistä tarkempi ja perinpohjaisempi määrittelyprojekti. Ei, vastaus ei myöskään ole juristiarmeijan laatimat sopimukset, jotka kieltävät virheiden tekemisen toimittajilta.

Ohjelmisto, joka altistuu todellisuudelle vasta kun se on valmis, on kuin ihminen joka altistuisi elämälle vasta 18-vuotiaana. Kompleksit järjestelmät (ihminen, ohjelmistot) vaativat evoluutiota tullakseen toimiviksi. 

Tämä on tärkein yksittäinen asia, minkä ketterä kehitys on tuonut (takaisin) ohjelmistokehitykseen. 

Todellisuus on niin monimutkainen, että sen kanssa toimivan järjestelmän on pakko kompuroida ensin, että siitä saadaan toimiva.

Ja juuri niinhän nyt on sähköisen reseptinkin kanssa käymässä. Ongelma vain on, että altistuminen tapahtuu liian myöhään. Lisäksi järjestelmä ja varsinkin sen hallinto ovat liian kankeita pystyäkseen mukautumaan tällaisiin "yllätyksiin". Sen sijaan nyt joudutaan odottamaan puolitoista vuotta(!), jotta loppukäyttäjän näkökulmasta täysin triviaali muutos saadaan tehtyä.

Osapuolilla on oltava tasapainoinen riskijakauma

Englannin kielessä on osuva sanonta "to have skin in the game".  Siitä tässä on kysymys.

Julkishallinnon IT-hankkeissa erityisesti ja julkishallinnossa ylipäätään on valtava ongelma tässä suhteessa. Virheistä, vakavistakaan, ei yleensä tule seuraamuksia sen tekijöille. Toisaalta myöskään onnistumisesta ei palkita. Tämä on virkamiesjärjestelmämme pimeä puoli.

Vapailla markkinoilla sekä virheistä että onnistumisista rangaistaan/palkitaan. Käynnissä on jatkuva evoluutio, joka tuottaa koko ajan parempia ratkaisuja todellisiin ongelmiin.

Aiemmin mainitsemassani ytimen omistavassa organisaatiossa työskentelevillä julkishallinnon työntekijöillä olisi aivan ehdottomasti oltava "nahkaa pelissä". Tuntuvat rahalliset korvaukset, jos hanke onnistuu. Potkut koko porukalle, jos epäonnistuu. Näin karrikoidusti. Palkitsemisjärjestelmien asiantuntijoilla on varmasti hienosyisempiä malleja.


Ohjelmiston rakenteen on tuettava ekosysteemin rakentamista

Apotin kaltainen monoliittinen ostos (jos ollaan edelleen ostamassa Epic-järjestelmää) ei toteuta tätä vaatimusta. Kovan ytimen tunnistaminen, suunnitteleminen ja rakentaminen on erittäin vaativaa. Vielä vaativampaa on sellaisen ostaminen. En oikeastaan osaa edes kuvitella, miten sellainen ostettaisiin "valmiina".

Ihan lonkalta heitettynä miettisin itse vaikkapa Suomen terveydenhuollon IT:n omaa pilviarkkitehtuuria. Tässä pilvessä sitten ajettaisiin ytimiä, joiden ympärille voidaan vapailla markkinoilla tuottaa toimivia osasia. Tätä on ansiokkaasti edistetty Taltioni-palvelussa.

Totta kai tässäkin tarvitaan hallintoa ja valvontaa. Mutta kyseessä ei ole edelleenkään rakettitiede. Apple tutkii ja hyväksyy jokaisen sovelluksen ennen sen päästämistä AppStoreen. Ongelma on aivan varmasti ratkaistavissa, jos keskitytään ongelman ratkaisuun, eikä byrokratian laajentamiseen.

Kenties olisi kannattavampaa viedä julkisen IT:n hallinta pois puhtaasti julkishallinnollisen byrokratian piiristä, jos näyttää siltä, että asioita ei yksinkertaisesti saada toimimaan? Toimiva ratkaisu olisi valtion omistama voittoa tavoittelematon yritys. Sen henkilökunta ei olisi virkamiehiä, vaan kunnollisilla kannustimilla motivoituja ammattilaisia.

Big Bang

Lopuksi vielä muutama sananen aiheesta, josta on ainakin Facebook-ryhmässä keskusteltu, ja joka kaipaisi laajempaakin huomiota.

Suomen julkishallinnon ja etenkin terveydenhuollon rakenne on todella hajanainen. Toimijoita ja erilaisia prosesseja, johtamistapoja ja eritysvaatimuksia on käsittämätön määrä näin vähäväkiselle valtiolle. Tämä ongelma on pyörinyt paljon niissä argumenteissa, joilla on haluttu puolustaa nykyistä tehotonta ja kallista toimintakulttuuria.

Kyllä, prosesseja on muutettava, eikä tietojärjestelmä voi korjata huonoja toimintatapoja. Tietojärjestelmä voi vain tehostaa (parhaimmillaan) sitä toimintatapaa ja prosessia, jonka mukaan toimitaan muutenkin.

Mielestäni Apotissa yritetään saada aikaan ns. Big Bang, eli paljon muutosta aikaan kerralla ja keskusjohtoisesti. Ehkä ajatellaan, että vain tarpeeksi suuri ja mahtipontinen hankejärkäle saa byrokratian juoksuhaudat revittyä auki. Voi olla, mutta tulokset eivät näytä kovin vaikuttavilta tähän mennessä.

Paljon helpommin hallittavissa oleva lähestymistapa olisi rakentaa Apotin ydinjärjestelmä ja joku rajattu määrä sovelluksia ytimen päälle yhteistyössä yhden pienen ja muutokselle valmiin sairaanhoitopiirin (tai jopa yksittäisen laitoksen) kanssa. Mieluusti vielä niin, että tulevat loppukäyttäjät ovat mukana varmistamassa toteutuksen altistumista todellisuudelle alusta asti.

Kun on saatu yksi onnistuminen, voidaan alkaa neuvottelemaan muiden yksiköiden kanssa. Tässä kohtaa mielestäni vaaditaan raakaa liikemiestaitoa ja mahdollisesti jopa lainsäätäjän painostusta siihen, että jokaiselle sairaanhoitopiirille ja peräkylän terveyskeskukselle ei tehdä poikkeuksia järjestelmiin, varsinkaan sen ytimeen. Joko käytät yhteistä järjestelmää ja muutat prosessit sen kanssa yhteensopiviksi tai sitten olet oman onnesi nojassa.

Of TDD and Unit Testing

Inspired by an amazing talk by Greg Wilson with the topic, What We Actually Know About Software Development, and Why We Believe It's True, I will not accept truths about software development without citation anymore!

Its amazing to realize after doing software professionally soon for 13 years that the things we take as truths in this profession are often not much more than opinions.

So the first thing I decided to ask Google was: Is there proof that unit testing and TDD really results in better software? The most promising result was a research paper published by Boby George and Laurie Williams: An Initial Investigation of Test Driven Development in Industry.

The results seem to back up the claims of TDD (and by reference also unit testing, I'd assume). 18% increase in software quality with only 16% increase in development time compared to the control group (and the control group didn't mostly do any unit tests, even afterward so the increase in development time is probably skewed).

So based on this research I'd have to admit to being wrong about TDD. But...yes, of course I have a but. The researchers admit themselves that the software that was developed was trivial (200 LOC, a bowling game) with static completely known requirements. To me this is a very serious problem with the study. All that can be concluded is that TDD works in optimal situations.

Things that this study ignores are:

1. Granularity of unit tests. In large software there are multiple layers where tests could be written. Are we to assume always the lowest possible level (ie. class)? If not, what is the optimum granularity? 
2. Complexity and lability of requirements. How well would TDD approach work in high complexity work with shifting requirements where fast feedback cycles and ability to change are important (ie. software will never make it into production if correct requirements cannot be harvested via fast feedback loops)?
3. Software size. Unit test code is often overlooked in the size of the overall codebase. In large refactorings test refactoring often (in my personal experience) takes most of the time. Is there a limit after which test code amount starts to hamper development speed enough to matter?

Since I believe in functional, automated black box testing via interfaces or user interface, to me it would be much more interesting to compare software developed with rigorous TDD and unit tests to software developed with tests similar to what the researches in the TDD study themselves used to determine the quality of the solutions. More specifically it would be interesting to find answers to these questions:

1. Which approach yields faster development cycles with a mature code base and shifting requirements?
2. Which approach survives large and small refactorings better and how big difference there is between development effort when refactoring?
3. Is there a difference in overall software quality? 

Also what was a bit striking and surprisingly dogmatic for me was this quote from the paper: "The industry standard for coverage is in the range 80% to 90%,  although ideally the coverage should be 100%" (The paper quoted this from Steve Cornett).

If you've ever worked in a large real-life project of any significance, you'll realize how absurd this is. To get test coverage to 100% with white box unit tests requires enormous amounts of very fragile unit test suites basically casting your software in cement. Refactoring is of course always possible, but I would imagine the psychological barrier for doing a large refactoring in a system like this is very high indeed. 

I know what you're saying now. But isn't that just the point? That the unit tests give enough security to do refactorings? Sure, but don't black box functional tests do just the same? In fact they do much more. They make sure your software works identically from the point of the outside world, which is what matters.

I have zero research to back my hypothesis that a much better overall result could be achieved these ingredients:

• Automated black box functional tests for interfaces and user interface where test code to production code ratio is optimised (Another thing I'm surprised to never see mentioned).
• Unit tests for obvious hot spots in the codebase with difficult algorithms that lend themselves to be easily tested with unit tests.
• Minimising accidental complexity and CDD (Curriculum Driven Development) - this is slightly off topic, but I like to keep reminding.

In fact I find it very surprising that functional black box tests aren't among the first things to be done in a new project. Still I keep finding projects where people write 1000 line unit tests for single Wicket-components that themselves have half the amount of code. Yes, you'll find out if a change in the component breaks something, but you're still utterly clueless as to if your actual user interface works as expected as a whole. 

I'm starting to think there is a background in psychology here as well. Its about responsibility and limting it. "My job is to make this single component. I did it and wrote a unit test to prove it. Now I'm done, I don't need to think of the software as a whole."

That may sound cruel, but I have observed it to be true much more often than I'd like to admit.

I keep coming back to psychology so often these days that I'm starting to think of starting the study of Software Development Psychology :)

Tampere Goes Agile 2012

Honestly I haven't been taking part much in the finnish agile movement in the past years. Yesterday was an exception though as I went with a good fellow to the TGA yearly conference.

The place is at the middle of the old Finlayson factory complex and brought back memories form the time I first started studying in Tampere 13 years ago. The building where the conference was located was exactly the same where I attended business english classes :)

The theme of the conference this year was "building the right thing" which is a great topic that has really been missing from the agile mindset for too long. I've often found myself wondring amidst projects: "Why should we strive for 90% test coverage or perfect Scrum-process if its not granted that the original requirements are sane?".

I'm not going to go through all the talks of the conference, instead I'm just going to pick a few topics and talks that inspired me the most.

Ideas

Lots of ideas were floating around. Lots of talking about how to turn ideas into business. By far the best talk was from Marko Taipale. Marko is a very inspiring talker with loads of real life hands-on experience. Also it was Marko's talk that again verified my own thoughts of how common and bountiful ideas are. 

Everybody has ideas. Another talk given by Ralf Kruse really demonstrated this well. He managed to make the audience sputter out really cool ideas by the dozen during his speech that was more like a hands-on session on how to create the initial product backlog out of literally anything. He just picked up a random guy and demonstrated the daily routines in his life (an university student) and how really cool sounding business ideas can be spawned from just about any aspect of a persons life.

Being part of a small startup myself, I've had a lot of discussions about ideas. The traditional approach, and also some kind of natural instinct is to keep your ideas hidden. The reason for this is of course the fear that someone will steal your great idea. This, as I've come to realize, is nonsense. There are so many ideas everywhere that people and companies are drowning in them. 

Back to Marko's presentation that was basically about explaining how do you iterate from a set of business ideas expressed in a standard format to a viable, scalable business. He laid out a very clear and simple method to follow. I loved the clarity in his thinking. Of course its not new stuff. Most of this has been introduced in the Lean Startup movement, but as with agile all in all, its one thing to know stuff and completely another thing to apply that knowledge successfully in the real world.

To me the lesson I took home from this was that ideas are good, they are the fuel that a startup needs. But you should not get stuck with your ideas. And most importantly, the ideas need to be put out there and tested for viability. My favourite quote from Marko was (sorry, might not be quoted correctly): "The facts are not inside your building." 

As long as you just keep fiddling with stuff in your office, all you have are guesses. Personally I'm not down to coding stuff for months or a year just based on a guess. Not anymore at least.

Software artistry

This is a topic probably as old as software industry itself. I found Sam Aaron to be also an inspiring talker. He has basically (and very literally too) combined art and programming. He was a presentation with the title of "Hyper Agile" which made me chuckle. But basically his talk was about approaching programming from a old fashioned craftsmanship direction and building first your own tools that enable you to create beautiful and artful software for your personal pleasure.

I have mixed feelings about this. I have background in arts (music) and Sam's message appeals to that side of me. My pragmatic self wasn't that thrilled. The question that kept popping in my mind was: "How do you create larger software that requires coordinated efforts by a team or even several teams if every developer has their own "light saber" that is ever so slightly different from everybody else's?"

Still, I'm definitely going to try Sam's pet projects Emacs Live and Overtone he used to give a really cool live music performance with. Basically Emacs Live + Overtone seems to be a programmable sound interface that uses Clojure. 

And that brings us to...

Clojure

Now I know I've already written about Clojure shortly and basically dismissed it. However I'm always willing to change my opinions in the light of new evidence, which was kindly provided by the very talented guys at Metosin. These guys were also sponsoring the event and providing some really fancy coffee (being allergic to caffeine, I can't give a better description, sorry).

The guys from Metosin showed me some production quality Clojure code from a Real Project, which was a first for me. Looking at their stuff briefly I was pleasantly surprised. It looked actually readable and not just clever guys being too clever for their own good (as seems to be the standard with Scala). Very much respect for this.

There's also a video in YouTube from these guys giving a demonstration on how to do stuff with Clojure and Eclipse, its here. 

Soo, as for me, the jury is now back out to judge Clojure again. I installed a Clojure plugin to my IDEA and at least so far its looking rather nice. Still, I have to say that part of the reasoning to jump from Java to Clojure was based on arguments that are not that objective. 

Let me explain. If you state that Clojure is better because it doesn't have a 7-level architecture with relational database, OR-mapping, Spring, Wicket and tons of other frameworks, I would argue that you've just decided to use all this crap with Java, but that's not the fault of Java itself. I have written quite a bit about frameworks vs. tools before and all that applies here.

The problem with Java isn't the language in my opinion. It has always been the culture and the fact that huge corporations like IBM and Oracle and SAP have ruined Java for everybody without a very clear vision on what NOT to use when doing stuff with Java.

I'm in the process of writing a multi-part article on how Java can be used to create just as light and flexible architectures. Stay tuned for those! 

Still, I'm giving Clojure a go as I love to learn new stuff. You can expect a report on my adventures on this blog in the future!

Summary

I had tons of fun at TGA. The organisers were super cool and the speakers were all very good. Thanks for this amazing event!

Also shout outs to guys from Nitor Creations and all other colleagues I stumbled on. Looking forward to chances to work with all of you!

JVM languages and why I'd like Kotlin to succeed

Feeling a bit more technical today I decided to write a bit about JVM-languages, those that are not Java, of course.

Does anyone remember when .Net came out and Java-platform was said to be inferior as it doesn't support multiple VM languages like .Net does? In an odd twist of fate, it seems that JVM actually now is the only real polyglot virtual machine where the non-platform-native languages are actually in widespread use.

For quite some time now I've been in search of a Java better than Java. Without luck so far I have to admit. I doodle for a while with a new language and then eventually return to Java for reasons that are actually quite important for real-life development:

• Java has superior IDE-support, especially with Intellij IDEA
• In Java the language constructs are simple enough to be able to have really intelligent IDE support that makes the tools feel almost like magic sometimes
• The core libraries are, despite some ugly ones, very concise and high quality
• JPA
• Readability of the code
• Simple type system with static typing

There are many more things, but these are the ones that generally drive me back to Java.

Jython

In some ways Python is my favourite language syntax. Its very short, elegant yet still readable (if you write it properly) and very beginner friendly.

I have done some programming with Netbeans back when it had built in Jython/Python support and it was a very pleasant experience. 

Only once you start to get into things like XML-processing, concurrency, persistence and pretty much anything that requires you to do coding on non-core libraries or, god forbid frameworks, things start to get pretty bad pretty fast.  

This is of course mostly related to standalone Python, not Jython. 

At the end of the day what stops me from doing any large development with Jython/Python is lack of static typing. Its fine for scripts, but I'm just not willing to write the unit tests for all the things a compiler (IDE actually does most of the compile time checks these days so you don't even have to compile to get the warnings) will do for free.

Scala

After I gave up on Python/Jython, I found Scala. A statically typed language with actor model concurrency, implicit typing and functional goodies sounded amazing! I bought the green/white book and read it for weeks and I was very impressed.

Scala is a language, I feel, that caters for the computer science laboratory -types among us. It is a very academic language and I might imagine many of its proponents visit regularly on Lambda The Ultimate. 

I still think Scala is a great programming language. I just don't see much value of its added complexity in the development tasks most programmers face daily. The syntax is partly almost mathematical in notation and that way very short and concise, but also almost impossible to understand at a glance, which is a very important quality for any source code that will be read aside from writing.

To give an example, read this and explain within 15 seconds what this piece of code means:

type Out[C[_], B] = C[_ <:B] 

Also it has to be added that tools and frameworks for Scala don't really seem to be made with real life development in mind. My experience was that API's are broken without backwards compatibility without thinking twice and most libraries and frameworks have a two-level versioning meaning they have a separate version per each Scala version.

If I worked in CS lab, I might think Scala is great, but as I'm not, my advice is to steer clear from it.

Clojure

Lisp on JVM! I very much understand the appeal of Lisp and its a very good language to teach how programming languages work over all since its a programmable programming language itself.

But to do something serious with Clojure? No thanks.

Groovy

Groovy gets much less credit than its due in my opinion. Sure, its a dynamically (or should I say optionally) typed language with not that great runtime performance, but I've rarely been as productive as I was when doing a project with Grails.

I really like the simplicity of Groovy's syntax. The closures (lambda-expressions) are also simple to read and understand, although possibly not as complete as in Scala or Clojure.

If I had to pick non-Java JVM language to do my next web-project with, it would be Groovy and Grails. Still, Groovy doesn't satisfy my need of static typing and flawless IDE-support.

Jruby

I remember it was 2005, JAOO-conference and the creator of Ruby On Rails was giving a presentation of his brand spanking new framework himself. Everybody was touting Ruby as the next big language and I suppose it became pretty big.

I never understood why. I understand the idea of RoR, but I think Grails implemented it much better. Ruby as a language isn't much in my humble opinion.

I haven't followed the Ruby-camp for years now to be honest. But when I was looking into it a bit, I saw massive problems with runtime lacking basic scalability and concurrency features we take for granted on JVM. Also the attitude of Ruby-programmers left me perplexed. It was like looking to the backstage of amateur theater rehearsal.  

JRuby might be an ok idea, but not very interesting. I reserve to change my opinion of this given new evidence though.

Kotlin

I have high hopes for this language. For one its being created by JetBrains, the creator of IDEA, the best IDE available for Java (in my opinion). I think they have the correct approach for developing a new general purpose language. 

Freely quoted list of reasons why the language was initiated (from their FAQ):

• Java compatibility
• Complies as fast as Java (Scala for example compiles very slowly and has been reported as a problem in larger projects)
• Safer than Java (NPE's are gone forever),
• More concise than Java with type inference, closures, operator overloading and mixins, extension functions etc (but without going absolutely mad like Scala)
• Simpler than Scala (a huge advantage in my opinion)

With these foundations, I'm expecting a lot of this language and I hope it gets popular and also that they manage to reach the 1.0 milestone.

If there is one JVM language I see that could overtake Java in my daily work, its Kotlin.

Of introversion, Steve Jobs, offices and agile

What have Pixar, Tom DeMarco, Steve Jobs and 37Signals in common?

For one, they are all mentioned in a book I've been listening while driving lately: Quiet: The Power of Introverts in a World That Can't Stop Talking by Susan Cain.

Ok so what does her book have to do with software development, you may be asking. On the surface, not much. The book is about the differences between extroverted and introverted personality types. Introverted persons prefer to observe, think, analyse and generally enjoy solitude more than crowds.

As a sweeping generalisation it could be said that software engineers tend to be introverted, like myself for example. In fact as an even more sweeping generalisation it could be said that many people in occupations that benefit from creativity are populated by introverts.

Let's take me for example. I don't like socialising with people much. I hate large crowds and especially loud places where I have to raise my voice. I never talk just to hear my voice and I think meetings are mostly unnecessary venues for extroverted people to have audience for their monologues. Or arguments if there is more than one dominant person in the meeting. I'm mortally afraid of public speaking and doing things in front of an audience. To get work done optimally, I need peace and quiet and freedom.

Now imagine that someone asked me: "What would be the worst possible place to work you can imagine as an introverted person?".

Off the top of my head I came up with a list of these attributes:

• No privacy
• Having to share my working space with many other people
• No freedom to arrange my working environment
• Lots of obligatory meetings where there's no real discussion of things that matter to my personal work
• Demanded presence, but no freedom to engage in ad hoc conversations
• Noise that I can't avoid
• Interruptions I can't control

Sadly and not surprisingly I just ended up describing the average open office working environment most of us have to work these days. This is bad. Really bad. 

For a while in the beginning of 00s, it seemed like things were getting better in at least one regard: You could escape the horrendous office spaces and work remotely. Then the agile boom swept the industry and oh boy, look at what's being implemented by literally everybody, even the government IT-projects: mandatory 100% on-site presence. All the other fancy stuff of agile is just too complicated or too cumbersome to do, but hey, let's make the bastards all sit in 20 square meter space with absolutely no privacy and we'll reap the benefits of agile!

In all honesty, I'm avoiding high profile agile projects these days for this sole purpose. Or if I'm forced to, I'm quite inclined to simply break the rules and hope I will be forgiven because of how much more productive I am than other people in the team who obediently sit in less space than zoo animals are given and try to desperately get some privacy for actual work by listening to music with insulated headphones.

I would love to see a reform in the agile project culture that would relax the fundamental attitude regarding this. Linux kernel wasn't and isn't done by people sitting in the same room. World wide web wasn't created like that, in fact it was created just for the opposite purpose.

A very interesting subtopic of Cain's book is that creative teamwork, contrary to common belief, isn't more productive than working alone. The times when great results have been made, are most often when the team members have been able to work by themselves and then share and collaborate online through distance!  

And here, of course, come Pixar Headquarters and Steve Jobs into this story. If I had to work 100% of my working time at an office, that is where I would like to work. Pixar headquarters are usually mentioned because of the huge main atrium that provokes unplanned, ad hoc sharing of ideas. But just as important, probably more important, is the fact that besides this atrium where you can go and meet people when you feel like it, the creative employees have private space where they do the actual work. I think for many reasons, this is optimal and the closer you can get to this, the better.

But office space is really expensive! It is. So why not have just the atrium and some space for people who really want to be at the office most of the time? And for the saved renting costs, give a small budget to your employees to decorate their home office, buy them a good chair at home and a good desk. Then have one day a week when most people meet at the office atrium and have a good lunch. Let them share war stories from the projects they are working on, talk about new ideas. Anything, but making them listen and watch through a slideshow by a manager about manager stuff. 

But wasn't the point of being in the same room that you could just lift your head and ask a teammate if you wanted to know something? The funny thing is, my personal observation is that the physical distance of the two people have much less to do with this happening than the willingness of these two people to communicate in the first place. 

Here's where I wholeheartedly agree with 37signals founder Jason Fried. Use good online communication tools that enable passive communication. And then try to create a working atmosphere where this passive communication is encouraged. A good way to do this is to have a team so well spread to different locations that not one person involved feels like she's in a position to expect the others to come to her physically to share information. 

Oh I almost forgot about Tom DeMarco. I mentioned him because I've read his Peopleware -book many times. He's been saying this same thing since 1980s and we've only gotten worse. Working environment is crucial for people who do creative work, especially when they are introverts - like many people in creative arts and crafts are.

Positive effects of project crisis

A while ago I had the pleasure to be the lead developer/lead desginer of a mid-sized software project. When I entered the project, it had been running for about six months already with very high goals and the general attitude of "we're making the perfect thing here".

Time for a confession: I find projects like the above described distressing. Weird huh? Most programmers see these projects as the "dream jobs". Seemingly unlimited budget, finally a place to use all of the cool new frameworks and tricks you've seen all the rockstar developers use. Maybe even do some coding in Clojure, right? And when wearing my programmers hat tight in my head, I agree.

I'm sure there are happy endings for such fairy-tale projects, but my own experience says otherwise. The crash with business reality seems almost unavoidable. At some point the project's budget shows up on the radar of the evil executive who has no understanding of the beauty of Clojure nor how important test coverage of over 90% is. What he is thinking of is: "Ok, we're pouring a ton of money in that project, but the ROI is either unknown or fairly small, either the ROI needs to go up or the expenses go down, preferably both."

And so the budget cutting begins. I've experienced it many times now, but it always feels just as bad. Quite often the whole project is eliminated and possibly code worth months of hard work is thrown out basically meaning you could have spent the last months at home playing Skyrim.

So having been through this myself numerous times during my career (at some point I had a period of 3 years of development without a single line of code gone into production), I've become almost obsessive about avoiding this for the very simple reason that I really hate creating software that is going to end up in the garbage bin.

But back to the project I was leading a while ago. As you can probably guess by now, the budget was cut. Not completely though and with a chance that if the project could produce something of business value with the remaining budget and a really tight time-schedule, it might go into production. What us software engineers want is to get our babies in production so everybody in the team went to a kind of "lifeboat" mode where everything unnecessary was thrown away and all our efforts were focused on producing business value as soon as possible.

This is of course a very crude simplification of the situation to suit my needs for this blog entry. Against odds we succeeded and the project survived and went into production and the story continues, but the interesting bits are on the table now. So let's investigate a bit.

Essentially what happened was that the project's life was threatened, but with a chance to survive if real value could be produced quickly. As a result the project went through a metamorphosis and changed into a very focused ultra slim (yes, yes "lean" is the word to use, I know) creature that could produce business value fast and with a very small budget.

As a joke to a colleague working in the project I once said: "Maybe we should establish a software development method that would deliberately drive projects to the brick wall right in the beginning to get rid of all the fluff from the get-go". The thought was really hilarious at first, but its been bugging me ever since.

So here's what I'm going to do the next time I'm leading a project that's starting either from scratch or starting a new release: I'm going to conduct a workshop with the stakeholders after an initial backlog and roadmap have been produced and have them go through the exercise of "what if the budget is cut in half" and "what if the budget is reduced by 75%". The remaining items on the backlog are the ones that will be done. And with the correct focus from the beginning, its possible to have the desired code quality because there will be less code to begin with.

I suppose you could call the exercise an "emergency evacuation rehearsal" or something like that. In essence its trying to defend from Parkinson's law, which in software development could be expressed as:

Software project always expands so as to fill the budget available for its production

Because let's face it, "Curriculum Driven Development" is a large problem in our industry. And even when that isn't the case, without proper focus even developers with good intentions will produce loads of unnecessary features and code.

The best thing a software project can produce is valuable features, but the second best thing is to avoid producing features that aren't valuable. Let's rephrase this into a "law" of mine that I try to live by:

Best code is unwritten code

Positive Software Development

A few days ago I watched a video by Shawn Achor. Its about positivity and how it can affect our productivity. So that got me thinking: How could I apply this in my work as a software engineer? Wait a minute! I am already!

It's really nice to get scientific support for things you've been doing intuitively until now, isn't it? What I've been doing intuitively while leading software development teams is to try and make the team relaxed and positive. It's not always easy and it doesn't always succeed. But when it does succeed, the result is loads of fun, productivity and friendship.

So how does it work actually? It might help to think of software development as a learning process. Here's a quote from Wikipedia article on Lean software development:

Software development is a continuous learning process with the additional challenge of development teams and end product sizes. The best approach for improving a software development environment is to amplify learning.

And how do we amplify learning? In his speech, Shawn Achor lists the following things:

• 3 gratitudes (3 positive perceptions every day)
• Journaling (going through one positive event of the past day)
• Exercise 
• Meditation
• Random acts of kindness

Ok so many of those are something you as a team leader can't do much about. The last one is really important tool for a team leader though. Giving random, peer-to-peer positive feedback makes people generally feel good about themselves. Your feedback does have to be down to earth and honest though. People can tell - at least highly intelligent people you work with - when you're like a kindergarten teacher. For this to work you need to understand what your team is doing. This is also why praise from a manager usually just makes the team uneasy.

What else can you do? Apparently there's a consensus in the academia that fear of failure is a big obstacle when it comes to learning. See Andrew Martin's work for example. I have an in-built high fear of failure and I'm sensitive to what other people think of me so naturally I always try to do everything I can to alleviate this. 

Software developers are for some reason often unforgiving when it comes to making mistakes. We're doing possibly one of the most complex things us humans are able to do with a staggering amount of variables and forces that can affect the outcome. Yet I often meet attitudes of "I don't make mistakes, ever, and I don't expect anyone else to do them either". What's terrible about this is that its impossible not to make mistakes when developing complex software systems. And in an environment that sneers on people who make mistakes, the mistakes are of course hidden resulting in crippling technical debt and loss of productivity.

In my opinion its almost the duty of a good team lead to set an example of failing fast and failing positively. With this I mean that mistakes and failures should be treated positively as chances to learn together and even more importantly, get rid of the fear of failure. When I work as a team lead, I always bring out my mistakes in the open, make sure everyone understands that I'm not perfect and that I don't expect anyone else to be either. That its ok to make mistakes and show them to others so they can learn from them as well.  My habit is to make jokes of my mistakes because, let's face it, if you're not able to laugh off some of the absurdity of our work, you're going to end up insane.

The funny thing is, I've sometimes been accused of being "unprofessional" when I'm trying to create a relaxed working atmosphere with laughter and permission to make mistakes (as long as you fix them, of course). You can safely ignore this. It's the 21st century and its ok to have fun at work, especially as its now scientifically proven that its more productive that way!

About online collaboration

A long hiatus again, almost forgot I even had this blog. The more experienced I become in the world of IT, the less interested I seem to be of the latest and greatest stuff. Of course I keep my eye on the new tools and technologies, but increasingly I find myself looking in the past of IT, to the age when thinking wasn't still polluted with all the things we take as given now.

I've watched Doug Engelbart's Mother Of All Demos recently and if you disregard the bad picture quality, it's simply mind-blowing what he and his team were able to come up with in 1968! That's 43 years ago and we're still not quite there yet.

In my understanding one of the main ideas of his research back then was to augment human thinking with the help of computers. The route to this, in his view, was mainly achieved by good real-time collaboration tools like audio/video-conferencing and real time collaborative document editing among other things.

We've had wikis for some years now, but let's be honest: most of them aren't that great at real time collaboration. Then we have a plethora of different kinds of messengers and Skype of course. And then different sketching tools for managing complex abstractions (like software design). But none of these provide a unified experience.

The closest we've gotten so far was with the failed Google Wave. We still weren't ready for it I guess, but I'm willing to do a small bet that the idea will resurface somewhere in some form. Back when I first saw Wave, my first thought was "wouldn't it be awesome to have a programming environment based on that idea". Maybe something like the apparently dead Code Bubbles project combined with Wave's real time collaboration.

Sadly we're not only restricted by missing technology. A big obstacle in the ways of efficient computer-aided collaboration is company policies and general reluctance to collaborate online. There are exceptions, like the previous company I worked for, Ambientia. While working for them I had possibly the most advanced online collaboration experience so far.

First a bit about their company culture. They're a mid-sized IT-shop, but scattered into multiple small offices. Also their headquarters are "off-site" to most customers which seems to promote more liberal thinking regarding physical presence in the lines of: "Oh, our customers are 100km away anyway so how can we demand that all employers sit in the same office all day". Instead they've become completely transparent to physical location. Every employee is in Skype and roughly 90% of their internal communication happens in project- or topic-specific persistent chats or ad-hoc audio meetings in Skype. The amount of email was minimal.

Anyway, I had the task of designing a UI for a small software with the help of a user interface specialist. Here's how it happened: We went to the project's Skype chat and agreed that we'll need to do something and we can do it now. That turned into a Skype audio chat (video is totally unnecessary for most things) and after a brief discussion we decided to do the design in real time collaboration. We had Confluence wiki with Balsamiq-plugin installed and we agreed that he would be the one doing the drawing. He shared his screen in Skype so I could see in real time what he was drawing and when we had an intermediate version, I could go check it in Confluence right away and add my modifications.

The point to take from this story is that in my experience our one hour online collaboration session was more productive than if we had been sitting face to face. Why? No need to reserve meeting rooms or time, because of location transparency all of it can be done ad-hoc. Another reason is that this toolset made us focus more on the actual task. And most importantly, all this was much less intrusive then a full blown meeting. He could attend another meeting in another city that day and I could stay home with my kid who was sick.

As much good as agile movement has done for the software development world, in this regard its taken steps back. Nowadays it seems to be the norm that we set up a scrum team and force them to sit in the same room 8 hours a day and forget all about any kind of online collaboration.

Here's an idea for the employers out there: use agile, but sacrifice a bit of the efficiency and give your employees freedom regarding location. This way some very talented people, who otherwise might not be interested in working for your company, might take the offer. And you get to extend your hiring anywhere in the world, time zones considered.

Tools versus frameworks

Sorry about the hiatus. I've been busy setting up my own company and it's running now. Today's topic is a difficult one: I'll try to describe the problems of frameworks in software production and how I think a "tool" is often a better way to raise productivity and quality.

What is a framework in general? In construction I'd imagine it to be something akin to a mold, something that sets many of the properties and dimensions before we even start the work. As long as you know these properties and dimensions, you're safe and you can apply the framework to save time and increase quality. I'd say the same semantics apply for software frameworks. They set some properties and dimensions of the software artifact when you decide to use the framework in question.

As long as a framework is relatively small with few properties and dimensions, I see no problem. You'll soon memorize the properties and can easily judge if using the framework is a good idea or not. But even initially small and simple frameworks tend to grow more and more complex either from the pressure of the userbase (OSS) or from the pressure to resell the same thing for the tenth time to the same customers (commercial products). Spring is an excellent OSS-world example of this. At it's very first versions, I thought it was brilliant and simple - more like a tool than a framework really. When I look at Spring now, it feels like holding James Joyce's Ulysses in my hands. And Spring isn't even worst of the frameworks, it's actually pretty good.

There are frameworks for everything these days. The useful ones are usually ones that simplify some technical aspect of software engineering, like UI-frameworks or frameworks for transactional components or frameworks for persisting data. The ones that cause usually more problems than they solve are the ones that try to venture in the non-technical problem space of software engineering. So you're planning to create a system for handling resources in your company? Not to worry! There's a plethora of amazing ERP-frameworks that only need a tad of configuration and you're done! Not.

The illusion here is that while many problems in software engineering resemble each other a lot, like ERP or CRM or information publishing (CMS), I've rarely seen a company actually adapt to the properties and dimensions the framework they're building things on. The thing most CTO's are missing is that if you choose to use a generic framework for CRM, you'll have to be prepared to modify the organization and processes of the company to adhere to the rules of the framework.

From the programmers perspective, frameworks generally suck. In the time it takes to understand all the rules and dimensions of the framework, a good programmer with good tools is usually done with the original requirement. And what happens when the requirement doesn't neatly fit in any of the available properties of the framework? Cludges, hacks, hours and days spent trying to circumvent the bloody thing.

Ok, enough raving about how frameworks suck. Tools - those I like. Why?

Let's consider a hammer, one of my favorite concrete tools. It's small, very simple and doesn't take long to understand how it works. You grab the other end and hit something with the round end or use the other end to pull off nails. But there's more. Once you learn the simple properties of the hammer well (after all, there aren't that many of them), you'll start to use it in more novel ways. For hitting other tools, for breaking concrete, adjusting a heavy shaft, heck you can even use it to forge steel.

A software tool is something similar to a hammer for me. It has one clearly defined purpose of use, for example to create XML-based REST-services with plain old java without external configurations. How does a tool differ from a framework here, you must be asking. A tool doesn't force you to any particular style of REST-services; a tool doesn't abstract away the Servlet API nor HTTP-protocol; a tool doesn't force you to change your architecture if you decide to use it; a tool lets you adapt it easily to better fit your need. A framework doesn't have these qualities.

I seem to be developing an undercurrent running through all my blog posts. Here it is again: It takes an experienced and courageous architect to decide against using massive frameworks. Sure, there are architectures that are a perfect fit for framework X, but only after coming up with the architecture without using a framework as a crutch. A good architect always first starts with a clean table, trying to imagine the simplest possible solution that would fulfill only the known requirements and nothing more. It's easy to add more complexity to that, but extremely hard to remove unneeded complexity afterwards.

The lesson here, if there is such, is to be prepared to sometimes take a few steps back in your work and try to think as the great minds in our field did 30 or 40 years ago. Sometimes it's the correct solution to create a tool for the task you're doing. Think of the Unix-philosophy where there are dozens of small clearly defined command line tools. They're still used and useful and, although I might not agree, I know people who think there isn't a problem in software engineering that couldn't be best solved by combining a bunch of Unix-tools in a creative way.

How to rebuild a large legacy system

Nice and catchy title, isn't it? First a disclaimer: I've never got the chance to try this in real life.

I was having lunch with a colleague working in projects orbiting an ancient mainframe-system originally written in 1980's (we were having Nepalese food btw, my favorite). The problems they were having in his company are very common. The legacy-system was originally written by a small team - probably less than 10 people - and through the years the size and role of the system grew to something like a CRM. Yes, this must sound very familiar.

Now the people who wrote the system are closing 60 years of age and are planning retirement. The system will probably die as the last original developer leaves the company. How do you cope with a situation like this?

The most common solution to this problem is to buy a very expensive CRM-software and start a small integration-project that aims to replace the old system as-is. Usually about two years later you have an enormous, failing integration-project in your hands and instead of one, you now have two legacy-systems and even the original one is now burning more man-hours than before because "slight" modifications were necessary.

So how would I do it? Here's how:

I'd assemble a small team with the best developers I could get my hands on. I'd provide them with the best tools available and seat them in a comfortable office with free drinks and, most importantly, the original developers and experts with the knowledge of the legacy system. I'd start small, something like reproducing one core function of the old system. The developers of the new system could ask 100 times a day how some minor detail works. The new team could write acceptance tests from what they've learned of the old system.

Then maybe a year later the old and the new system could start to run side-by-side, some transactions of the old one routed to the new one. This is what is done with plants (hardening) so why not with software. One function at a time the new system would then replace the old one and grow to a complete system with all the functionality of the legacy software working as it did before, the difference between them being that the new one is done with (hopefully) a modern, flexible and light architecture that enables it to grow along with new business requirements. And what's even more important is that you now have a new team of people who know the new legacy system inside-out.

I wonder if it was Brooks again who has said that there are only two kinds of software-projects: ones that fail and legacy nightmares. As bleak as that sounds, in my opinion, a legacy-system is a nightmare usually only after the original developers of the system are no longer available. So after the system is done, make sure the developers have no reason to change jobs! Do not disassemble the team!

The lesson here, in my opinion, is that large systems cannot be built. They are grown instead by a long-living team. Anything else and you're probably going to fail.

My next blog will not have anything to do with legacy-architecture, I promise :)

Back To The Future

No. This blog entry isn't about the new game about the old movie with car-enabled time travel. Instead try to think back to late 1970's and early 1980's. Ready? Good.

Banks and other businesses with intrinsic need for computers were, at latest, building their information systems. CICS-systems with whopping 64 kilobytes of memory was pretty much the standard fare. Debugging meant printing out the core dump (after all it was ~64 000 characters, not that much). Used language was Cobol with static memory allocation. It's amazing to think it was possible to create large banking systems with that technology, isn't it?

What's more amazing is that many of these systems are still in use. If you use a cash dispenser (at least here in Finland), it's very likely that the transaction is eventually run in a CICS-system. Only companies that have been established after 1990's or so have more modern systems.

The question to ask, of course, is why haven't these antique systems been rewritten with Java, .Net or Haskell even? It's not from lack of trying, I assure you. I've personally witnessed a few very large projects and heard of many others. They were all failures by most standards. There never is one simple answer as to why a software project fails, but one question especially has haunted me for a while now: Why can't we seem to succeed in rewriting software that was originally done with such limited tools?

My theory about this is that back in the past, developers were able to concentrate better on the essential complexity of the software they were creating.

Commercial vendors have clouded us from the essential complexity of any given business-domain with process engines, portals, executable models, frameworks, predefined domains, predefined development processes...on and on. And the problem isn't just commercial vendors. Let's say you're going to use a very popular open source tool Grails to do your project (note, I have nothing againts Grails, I actually like it very much). Here's an incomplete list of techonlogies and frameworks you'll need to master in order to get a large project done:

Groovy, Java, JavaScript, XML, Ant, Gant, GORM, GSP, Spring core, Spring MVC, Hibernate, Log4j, HTML, jQuery, YUI, HQL, SQL, Ivy, Maven (repositories).

I used Grails as an example here because it's easy and productive compared to many other options. Still, the sad fact is that we're drowning in tools and frameworks. And as developers and architects, we've been conditioned into thinking we're helpless without them.

What many don't seem to grasp is that every framework, every library you add to a software stack adds accidental complexity which makes it more difficult to concentrate on the essential complexity. And no, the correct solution is NOT to start rolling your own framework. The solution I'm proposing is to minimize the use of frameworks and especially commercial products in an architecture.

I'll give an example: About a year ago I created a small ERP-solution from scratch. Sounds crazy, right? ERP's are monstrously complex and even starting to use one in a company takes huge amounts of effort. Well, you know what, it's not that complex when you can concentrate on the essential parts of it - the parts that matter to the problem at hand.

I ended up creating a very light (micro?) architecture where the domain was running on JPA-annotated classes in an embedded Jetty-server exposing REST/XML services using a small tool I created for creating XML-messages imperatively. The point is that the following code snippet (a whole service for returning all persons in the database) only requires Java 6 runtime, nothing else. No frameworks to learn or debug, no added complexity.

public void all() {
        ensureLogin();

        Message m = okResult();

        Query q = jpa().createQuery("select p from Person p");
        List persons = q.getResultList();

        for (Person person : persons) {
            m.set("person/@name", person.getWholeName());
            m.set("@id", person.getId());
            m.parent();
        }

        write(m);
    }

I'm not suggesting to stop using frameworks. I'm suggesting to evaluate very carefully whether they're worth the accidental complexity they bring. Do everything possible to minimize accidental complexity while providing developers tools to express the essential complexity!

And learn to make the difference between a tool and a framework. I will blog about tools vs. frameworks at some point. However I think my next blog will be about how (in my opinion) legacy systems could be rewritten successfully.

Why should I care about complexity?

While driving to work one morning, I remembered a discussion I had with a colleague maybe a year ago about complexity in software. Our discussion was about Fred Brooks' terms accidental complexity and essential complexity.

Accidental complexity is something we don't want. It's all the "fluff" that creeps into software projects: overly complex architectures, unneeded frameworks, "cool" code written by programmers, the list goes on and on. You'll notice that my definition is different from the original by Brooks'. That's because the original accidental complexity has been mostly solved (we don't write assembly anymore nor do we read core dumps printed on paper). Instead the modern accidental complexity is coming from the bazillion tools, frameworks and platforms all claiming to solve every software problem ever known.

Essential complexity is, well, essential. It is something that has to be solved in order to solve the problem at hand. It cannot be reduced, but it can be tackled. And more, it needs to be clearly visible so people are able to concentrate on solving it with minimum wasted effort.

I have a whole blog entry waiting to be written about how recreating old mainframe-systems these days seems impossible and how it relates to grown accidental complexity in software production, but today I'll address the connection of these two types of complexity and the role of a software architect.

Back to my old French car and my thoughts about our discussion. In a short moment of clarity, I realized what is the most important thing a (good) architect tries to do in every project:


Minimize accidental complexity and maximize the visibility of the essential complexity.


This might sound given or then consult-type jargon so let me explain. One very common reason why software projects fail (aside from plain bad salesmanship and contract negotiation) is that the essential complexity of the problem domain is either hidden completely or overshadowed by accidental complexity. Here's an example: A customer has bought a very expensive <insert your favorite commercial bloatware here> platform that will solve all their business needs - all that needs to be done is some customization, configuration and such. What follows is an often failed project where all of the time that should be spent tackling the essential complexity is spent instead in swimming in the accidental complexity of the given platform.

The tools to make essential complexity visible are well known. The most important is good domain-modeling of the problem domain. Another is projecting use cases on the domain-model to make it response to the concrete usage scenarios better. Nothing new or exciting here.

What's much more challenging is fighting the accidental complexity. This is where the value of an architect is weighed. Do you have the balls to say: "This framework/product is unnecessary and we won't include it in the architecture." Then you need the skills to keep programmers from producing code that is not essential to the problem at hand. This is where it's easiest to see the difference between a junior programmer and a seasoned one too. Seasoned programmers (if they've learned anything) solve the given problem with smaller amount of code - code that is clean, simple and without using unnecessary tools or frameworks.

Keeping accidental complexity at bay is often a matter of life and death for the software being created!

Liferay functional testing

Yesterday and today we've been having our first Fedex Day at Ambientia. The way we did this was to list ideas for it beforehand and then when the day started, we could group around the ideas freely.

I had submitted an idea regarding functional testing on Liferay and also ended up with it. So what to do about such a large topic in 24 hours? Well, first I googled the topic. Everyone including Lifreay itself is using Selenium, but I wanted something lighter, smaller and not dependent on external browser.

Initially I installed Canoo Webtest and started playing with it, but it felt a bit clunky and old fashioned with all the xml-files and rather weird runtime build on top of Ant. I had already decided to use Groovy for the tests since Canoo also supports it, but the documentation just wasn't there and the syntax was too ant-line also.

I've been toying with Grails Functional Testing Plugin recently and I've really enjoyed the simplicity and elegance of it. So I decided, what the hell, I have still almost half of our Fedex Day left so I rolled up my sleeves and started to port the grails plugin over to my Liferay-project.

After a while I had a stripped-down version of the grails plugin with all references to grails removed. Then I proceeded to make a very simple ant-script that compiles the groovy-files (we already have groovy-all.jar in our liferay project template so this was easy to do). Then I set up a directory called "funtional-tests" in the Liferay Plugins-SDK project and created the simplest possible test I could think of:

class SimpleTest extends functionaltestplugin.FunctionalTestCase {
  void testHelloWorld() {
    get('http://localhost:8080/')

    assertContentContains 'Hello World'
  }
}

Naturally it took some figuring out to get rid of weird errors, but soon it was working! Call me a nerd, but I think that's pretty cool.

Future plans

This is of course just a beginning. One big question is how to sandbox the functional test or the environment itself. Luckily I have a colleague here at Ambientia who is working on a small tool that can create pages, portlets and other data based on a simple xml-format. Using that we can setup and sandboxed environment where portlets can be tested functionally without having to ruin the actual portal you're building.

Also I'm thinking of trying to run Lifreay in Jetty embedded. This would make it possible (in theory at least) to control the whole environment while running the functional tests. I will try this anyway and report success or failure.

Another issue is controlling the state of the portlets themselves. What if you have a complex portlet done using Liferay's Service Builder that has it's own database and you want to run functional tests on it? I'm thinking maybe the edit-mode of portlets might help here. You might set up things like "create test data" and "destroy database" as simple buttons in the edit-mode and these wouldn't be too hard to call from the functional test. It could look something like this:

class ComplexTest extends functionaltestplugin.FunctionalTestCase {
  void testEditMode() {
    get('http://localhost:8080/home/portletXTestPage')

    click 'edit portlet x'

    click 'create test data'

    //run the actual test
  }
}

There is still lots to be done, but I'm quite optimistic about this.