The sword in the scabbard is a price beyond compare.
Jerome, the first Wayfarer.

Agile Architecture is an interesting subject from a people point of view, especially when funny movies are circulated about a house-building team embracing “Agile”. The movie shows how the team starts constructing happily, but stops somewhere halfway after finishing the ground floor, because other priorities pop up. The point raised is ostensibly that some things simply don’t fit with a agile way of working, comparable to a manager stating “I don’t need to be Agile; I know what I want! Just build it already!” Naturally, when the poster of the movie is cornered about his or her opinions on Agile Software Development, the reply invariably goes towards the comedy value of the content rather than the point it appears to make. So far however, I have yet to find someone making the correct observation that this team lacked a sense of their customer, and probably a Product Owner. As Dave Thomas argued in his presentation “Agile is dead” at GOTO 2015, agility is about taking small steps towards your goal, and after every step evaluating what it did for you. Being able to change direction after every step is what makes you agile, not whether you actually do change direction.

When I read Robert C. Martin’s book “Clean Architecture”, I was intrigued by a chapter called “Details”, which started with “The Database is a Detail”. It discusses the application of agile architecture, and not over-architecting when a simple solution will do. The point is that even the choice of your database technology is a detail which might not be important enough to solve up-front. If, like me, you work in a large organization which values the selection of a standardized Relational Database, that approach can be quite a challenge.

The Professional Hobbyist’s Dilemma

When I was a student, I happily hacked away on several personal projects, the biggest an extension of TRS-80 BASIC with commands for 3D graphics. I also wrote a minimalist word processor, because who didn’t, and an application my high-school used for keeping lists of addresses and printing them on stickers. As I learned more about the “proper” way to write applications, I noticed the speed of development going down. Whereas I used to just go ahead and start crafting code, I ran into enough rewrites to start appreciating the value of planning ahead.

So when I started doing really big projects, and appreciated the joys of having tools like really powerful database engines, I naturally wanted to use them for my hobby as well. The problem was that there were only limited options for combining “powerful” with “fit for hobby usage”. In the height of the “cathedral and the bazaar” days, when “an itch” was supposed to spur you into building your own open source equivalent of big business originals, my problem was that I wanted to solve what I was working on, and not postpone that indefinitely to make time for writing my own database engine. At the same time I couldn’t really feel happy about writing an application that condemned users to downloading and running Oracle Express, and consuming CPU cycles, memory, and disk space, in amounts that fitted a company rather than a hobbyist.

Why couldn’t I just forget about using an enterprise scale database, and just go with flat files? Well, mainly because I ran into so many enterprise scale applications using flat files… kind of. I had learned to distrust cheap and simple solutions, because company managers were so good at downplaying the future impact of quick and dirty. They wanted a “good enough” solution, now! Then when the impact finally did become unbearable, the remediation costs had skyrocketed. IT managers, developers, and database administrators, we all learned to protect ourselves by insisting on thorough designs and standard solutions. Sure the business promises the application will only live for a few months, or that they won’t expect high performance or high availability, but we all know that they will soon forget those promises, and blame us for not warning them properly of the impact of those quick and dirty solutions.

What I really hadn’t expected, was how this affected my hobby: I nearly lost the ability to hack something together, because I kept thinking ahead, and tried to solve problems that might crop up a few years later. Sure there is a lot more to choose from, but since most large open source products nowadays try to be enterprise grade, there is still a large gap between what e.g. MariaDB or MongoDB has solved for me, and what I actually face now. Worse; even though I am a fierce supporter of Lean and Agile Software Development, I was having trouble with this Agile Architecture point of deciding at the last responsible moment.

The Database as a Detail

So back to Uncle Bob and his chapter on details. His example of the database said that, if you are not working with a highly structured data model and large amounts of data, then why bother with an RDBMS? Sure, you can see a future where you may have a need for such a thing, but if the customers won’t come in those hoped-for numbers, it’s wasted effort. Time better spent on some other thing. Right… But my company has Brand X databases as a simple, self-service product. I can just go to some intranet portal and order me one. What’s wrong with that? The effort isn’t that high, I can keep the data model simple, and just go from there. Better still, if the customers do come, I’m ready for them, right?

Then I receive an email about an upgrade to the newest before-last version of Brand X (we’re a cautious company) and I’m stuck with unplanned downtime. Well, not planned by me anyway. Then we add some features that would really benefit from queries over the (JSON formatted) data and we’re stuck with with remodeling our data to allow for those Brand X JSON features to be used. Or we could just load all data in memory and do it that way, but now we’re really wasting the cost of a highly-available database instance on essentially flat-file storage. Then something else comes up: the application does take off, but in an unexpected way, with the need to distribute the data across the globe. Easy with our cloud provider, but expensive in combination with an RDBMS. Something like MongoDB would have fitted that a lot better, just when we started started using a properly modeled relational data model, if only for a reason that MongoDB could have solved just as well.

Now this is what is meant with treating the database as a detail: sure there are a lot of options, but if all we need is a flat file, then by all means, let’s use a flat file. We may replace it with something else next year, but a flat file with all data in memory is not that big an investment, so hardly any wasted effort. Had we given in to the enterprise approach, and hosted a properly normalized relational data model on an enterprise grade HA database cluster, then we had definitely wasted a lot of money, with no conceivable profit.

Building a house

Now the question 100% certain to pop up is: what if we do know what we want? Take that house from that video; what is the sense of delaying decisions if we know we need a house? Well, are we talking about a single house or a whole street of glorious mono-build? The contractor will take that job happily either way. Are we going for lowest price, standard units? Than they’ll architect the hell out of it so they can squeeze the last penny out of doing things in bulk. For sale or rent? If for sale, how much customization do you want to allow the buyers? If for rent, let’s take some catalogs and choose interiors. See the pattern? The contractor will allow you time to choose and adjust, up to the last responsible moment, and then work will stop until you finally make the decision. Forgive me for saying it, but that contractor is definitely employing agile architecture, right?

Ok, so now we want to replace an ageing application.
Don’t tell me you mean…
Yep. Business hasn’t changed in those years? You sure you want to wait a year or so while I rebuild your old application in the cloud with something hot and sexy like Go or Kotlin? Same functionality, right?
Well yes, as long as you take along everything we added in and around it. Oh yes, and next year we start with that new product that requires email addresses for the whole family, so make sure you can facilitate that, and oh, don’t forget to do something about integration, because SOAP is out and REST is in. And our helpdesk employees need to be identified by email addresses as well, and with Office-365 we’ll be switching from phones to Skype…
Yeah… right…