Open letter/discussion: Hotfixes & Patches

I'd like to start some constructive community discussion around patching and hotfixes in response to the latest hotfix announcement earlier this week (12.5.1 hotfix 12) as well as Wednesday's development update. I haven't seen too much of these topics in the general discussions and want to hear how other IT folks' thoughts differ (or not) from mine.

HF12 appears to be MUCH larger than what I've become accustomed to for typical hotfixes. Not just a SQL script, it requires client files and even updating services with TIM. Why, I wonder, was this not presented instead as a new patch release at version 12.5.2? I am most curious what the network's typical habit (of doing releases only once or twice per year and then continually issuing hotfixes) is modeled after. Comparatively overwhelming numbers of projects tend to follow, roughly, "Semantic Versioning" (see http://semver.org), where the version number alone can tell you if it has bug fixes, new features, or large-scale non-backwards-compatible changes. Tessitura's pattern of development seems to follow these types of changes, but cumulative bug fixes are never (or very rarely) released as a new version, and I wonder how people feel about this.

One of the issues I see are that hotfixes can be selectively applied by the licensees being supported by the network. Were I in the network's position, I would see this as a huge problem—as hotfixes increase, you have a geometrically expanding number of possible configurations to support, but with cumulative patches, this growth is only linear; and the only option for licensees experiencing an issue with any version is to apply the latest cumulative patch.

As a licensee, I would MUCH rather have a single cumulative patch version to update to than to have dozens of different hotfixes to evaluate separately and decide if they are worth my time to install each one. But, I would want to have a good level of confidence in two things: 1. That the changes made in the patch releases have been developed and QA'd to the same standards as any other release, and 2. That patch versions (12.5.x) ONLY contain bug fixes, and no feature changes that would require my organization to do a complete upgrade testing sequence. I think it goes without saying that in this release model, upgrade deadlines would only apply to major versions.

Finally, I recall some discussion from the network around making hotfixes mandatory, or at least easier to install, but also a notion that a hotfix was different from a release because it wasn't held to the same QA standards. I don't know how I would feel about mandatory hotfixes if I knew they weren't being held to the same standards as code for general release. Even in a perfect world, I think hotfixes will continue to exist, but I would like to see a shift from them being an awkward institutional vehicle for delivering patches, to being a temporary-by-design means of getting an organization back on their feet when they can't wait a couple of DAYS for the next patch release.

I have a suspicion that the network's release workflow is a bit more process-intensive than clicking a button. And I imagine that might pose some difficulties for the more frequent releases I am advocating. If that is indeed the case, then I would certainly be speaking for my own organization when I say "Please, spend our money on QA resources, streamlining, automating; whatever it takes to make releases easy and fun and not dreadful and terrifying!" (See also: Why every development team needs continuous delivery) I think it's better for everyone when fixes as well as new features get into our hands when they are ready, and not just when there's a critical mass of stuff to justify an arduous release process.