STE VE CASSIDY
Steve spends time with Jenkins, a modern-day butler for DevOps teams, and wonders if Apple and Microsoft’s lawyers need to have a tête-à-tête
Steve spends time with Jenkins, a modernday butler for DevOps teams, and wonders if Apple and Microsoft’s lawyers need to have a tête-à-tête.
How on earth can you get from a coffee-can Mac Pro strapped into a rack-mount gurney to arcane web infrastructure concepts such as continuous delivery, DevOps and cloud computing? Easy: step inside the world of Jenkins.
This quaint-sounding English word certainly isn’t what you might expect from an organisation founded by a very Japanese-sounding CTO (Kohsuke Kawaguchi). The naming is partly historic, following an opensource spat with Sun Microsystems, but it’s relevant for two reasons. First, because Sun’s project was called Hudson. Second, they needed to reflect the job this thing does, which is fundamentally to be a code butler.
This sounds a long way from networks, let alone mysterious Mac Pro references: code is code, right? And it all runs up in one of about five basic types of cloud server these days, so the tools you need are the ones relevant to the management platform of your cloud of choice, right?
I’m sure there are thousands of web developers and site constructors who will make these assertions, and have every chance of not being contradicted for the rest of their working lives – and thousands of businesses that take those words as the justification for a lot of spending on development and developers. Pulling an all-nighter over the big site update weekend is a billable activity, as well as offering a chance to be hugely engaged with the client’s interests – gold dust, in other words, in a business not normally associated with high levels of customer engagement.
There’s an overlap, too, with the business of tech journalism. We live in a labyrinth of embargoes, release dates, bug fixes and vulnerabilities ourselves. The role of an editor in looking at what someone like me produces absolutely includes a need to intervene in the process to make sure I haven’t said something stupid – or, worse still, libellous. Human stopping points in the road from creation to production are our immune system.
So there’s a double layer of incredulity when it comes to the idea of a tool that automates those careful rounds of testing, signing off, taking responsibility and then releasing – rituals common to both software development and technical writing, trying to make intangible achievements more remarkable to our paymasters and audiences.
Jenkins sits in Java, almost certainly on a VM within Amazon’s cloud services layer. It automates the process of system compile/build cycles, testing the resulting site code against all possible combinations of user and device, so that errors or odd behaviours that might otherwise only emerge when the code gets to the consumer in a traditional development space can be monitored and caught early. You’re very likely to end up writing your own Jenkins script to define what your test actually entails. For example, what would count as a pass or fail, how many
stages there are to it and who should be told when there’s a problem.
Now, I used the word “consumer” specifically, because the world of software in business-to-business transactions is totally different from that in the consumer world. Broadly speaking, those consumers now have almost no patience whatsoever. They expect sites to work, to be understandable, and to move quickly when there’s a problem. That produces higher expectations inside companies that largely deal with TGU (an acronym I just thought up that could mean either Those Glorious Users or, rather more honestly, The Great Unwashed). The demand for systems testing before TGU get hold of your code is exceedingly high, even though the rationale for being sloppier when dealing with other, equally qualified professionals isn’t defensible.
Consumers like to use all kinds of crazy machines for accessing your freshly assembled website, which brings us to the technicalities of being sure you’ve really tested against all the permutations of browser and OS before you release the code to the public. With most PC operating systems, this is achievable via straightforward wrangling of VMs – although, admittedly, it’s made a lot easier if the VM hosts are inside your own data centre. Where all that breaks down is around Apple.
There are, of course, naughty ways of running an Apple OS in a VM. You simply need to follow some web-hosted instructions on obtaining an image of the Apple system ROM and then include it in the VM you’re making. But you’re also then illegal. Apple explicitly prohibits running its desktop OS on anything except Apple hardware. This isn’t an issue for the little guy (although there’s the risk that Apple might one day choose to make an example of someone for a flagrant-enough breach of its terms), but it’s a red flag for larger businesses. They must be able to show they aren’t in breach of any applicable terms – and Apple says that only Apple hardware can run Apple software.
MacStadium are the guys with the craziest answer. And they’re happy to talk about the size of some of their clients’ machine farms. Thousands of Mac Minis? No problem. There’s even an economic justification for Mac Pros – not the big alloy cases of old; the modern, 12-core coffee-can jobs – to be laid on their side, base air slots open to the cool air, in a sort of long metal sled or coffin along with an expander gadget that presents multiple fibre-channel optical fibre interfaces via a Thunderbolt adapter to the host Mac. These machines earn their keep by firing up a multitude of macOS VMs when there’s a requirement to test that day’s code. And user-testing such as this (as anyone who’s seen the PC Pro benchmarks will attest) is no flicker of activity: with run times getting on for hours, the faster the machine, the more tests you can do in a day.
And remember, this is a VM-driven environment. If it can run macOS VMs for test purposes then these devices can also run other, less restrictively licensed resources.
One big problem with claims about the nature of the cloud and extensible, elastic computing is that it’s hard to get mid-flight assessments of just how big your job is. I understand that this is what the various monitoring or dashboard apps are supposed to be for, but seriously – if the whole point of the cloud is that it’s on someone else’s computer then a real sense of system load is hard to come by. The MacStadium guys were talking about Microsoft placing orders on the scale of thousands of machines, just so it could run and re-run its tests, zooming up into the cloud when the environment suited the requirement, and back down to some local racks when Apple’s legal demands ruled out the cloud altogether.
I was having these conversations at Jenkins World 2017, but what stood out at the conference was just how limited progress has been over the past several decades with regards to the coding tools developers use daily.
I was around for the first flowering of Turbo Pascal, QBasic and friends, when having a compiler on a PC was a big deal. Even then, language quirks and common pitfalls could sink projects, either on the day the code shipped or many years down the line – and this spawned an evolutionary explosion of coding tools and concepts in compilers. The various Pascals with their confusing syntax and unseemly obsession with data types were expected to give way to the clearly rapidly changing Modula-2’s and Modula-3’s. The first version of Visual Basic was greeted with sighs of relief and enthusiastic early adoption, at least until folk realised that easy access to the internals of Windows might not actually be that much of a blessing.
The current architecture seems to be based on entire tools and technologies that demand incredibly large and constantly updated delivery platforms. As in every day. Three
“Jenkins tests the site code against all possible combinations of user and device”
“It takes only a tiny number of initial conditions to spawn a monstrous, unstoppable beast”
shifts of programmers are preferable. All the time. Having that sink in was a vital prerequisite to talking to any exhibitor or to people from CloudBees, the event sponsor.
This is, to be sure, a tremendous intellectual achievement – one of the keynote slides identified the challenge of being a Jenkins administrator as “high cognitive load”. Given the already high capability of the presenters and the audience, this is quite some problem. However, the question of whether it’s the right place to put the effort becomes tricky to avoid after a while. Rather like wondering if the job done by huge racks of Mac Pros just slavishly running test suites might be better managed by a sensible, one-to-one chat between lawyers at Microsoft and Apple – surely a rational provision of Apple software courtesy of some simple low-key USB dongles is a better way for modern businesses to spend their money?
One can’t help but think that this entire industry – whose start I saw with Red Hat at another convention a few years ago – could be swept away by a very mild outburst of lateral thinking.
Black Duck: a petabyte of patches
Perhaps the most matter of fact stand exhibitor was the mysterious Black Duck. You can read its blurb at blackducksoftware.com, but I found it more helpful to have absorbed much of the information around Docker before going to see what Black Duck was doing. The exasperated developer blogs about Docker aren’t especially numerous, but they certainly seem heartfelt, and it was obvious from the difference in attitude between Black Duck and all the other stands at the show that there was a story to tell here.
From the rest of the show you might imagine that container-based VMs are a dead certainty for the technology of the future, and that they are as well-engineered and proofed against disastrous malfunction as the best of the hypervisor-style VM environments (or multitenant non-VM based web hosting – that being the other way of achieving the desired result). Black Duck, and the developers blogging merrily away, beg to differ.
The problem is a modern-day version of Goethe’s 1797 poem, Der Zauberlehrling. If 18th-century German poetry isn’t your thing, you might recognise it as The Sorcerer’s Apprentice: a cautionary tale about capacity management and disaster planning, possibly softened too much by Disney casting Mickey Mouse as the apprentice. The moral is that it takes only a tiny number of initial conditions to spawn a monstrous, unstoppable beast. In the case of a system such as a container-based web-hosting environment, the developers play the part of Mickey Mouse (yes, I realise they won’t like that comparison) and the unstoppable monster is the entire open-source code movement.
If you’re going to completely recompile and test all your code for the whole of your hosting farm every day, how can you possibly keep track of the changes to the open-source repositories that contain all the code cut by all the developers – casual, professional, commercial, military, even criminal – who have gained access to make changes? How can you even make the right choice between the numerous selfreplicating forks that have taken place in the genealogical history of all the code that compiles to make the host OS and the containers and the guest partitions that together make up your environment?
It’s this kind of complexity – Mickey Mouse’s water-carrying broomsticks with only two to three degrees of freedom added, but supercharged with a healthy dollop of computing capacity – that makes it almost impossible to say which subset of versions out in the wild can be cleared for use with your product, and which can’t.
Much of this problem is down to the intimate relationship between containers as VMs, and their host operating system: there’s little of the classic hypervisor “black box” approach here. Containers are a way to have lots of guests, provided they can all share the same OS, services, code base, and so on. As such, it fits the way that hosting companies have grown up, charging almost by the byte with “packages” that are laughably small by today’s hardware standards.
Purists will say that you can have a perfectly workable e-commerce website in 100MB of storage and 256MB of system memory, and it’s probably possible to make a container with sizes that small. But the hidden cost comes not from a bit of lab work, content with making the platform stand up for a few weeks before disastrous updates strike; it’s in making it run 24/7/365.
This is where Black Duck has taken what might seem like extreme measures, just to give a few commercial enterprises a degree of comfort that they chose the right hosting. It has an 11-year deep archive of every patch, every note, every added bit of code, to all the major active container OS opensource projects.
It’s a Hadoop database, for those who are turned on by such things, and it’s hovering somewhere in the half-petabyte range at the moment. The scary notion is that as far as Black Duck is concerned, despite the extreme longevity of this resource, it’s only been displaying signs of sharply accelerated growth in the past 18 months or so. This reflects the adoption of container-based VMs for hosting by the bigger players in that market and suggests that they might not get to the end of 2018 without breaking the petabyte barrier.
Both Black Duck and the remainder of the exhibitors see this as an indication that they’re on to a commercial good thing. I don’t agree: I think it’s a sign of a platform and a methodology that’s in need of a radical rethink.