What is software management?

“Managing software” and “managing software development” is a complex subject which is surrounded by a lot of bruhahah and fancy marketing presentations full of neat-looking flow charts and ambiguous terminology. I’m trying to establish some common vocabulary so we sort-of know what we’re talking about.

We can identify several common activities which apply to most software management processes.

  • versioning. Sane developers use a version control system such as subversion to keep track of the changes to source code, especially when there’s multiple developers working on the same software.
  • building. The wikipedia description talks about converting files from one form into another, which is one part of the process for constructing software. Since we’re talking about a “directed transformation” (we’re mostly looking at compilation, not decompilation), we use a term to indicate this. In analogy to what engineers working in the physical world do, we tend to talk about a “build tool” or “build system”.
  • testing. Usually considered part of “the build”, there are often automated tests for software (and sometimes also manual tests) to verify that the software was built/installed/packaged/configured correctly. Sometimes, it is possible to skip the tests. Sometimes it is hard to run them or interpret the results.
  • packaging. Once software has been built (or, also common, before it has been built), we often want to share it with others (or sell it). To this end, we tend to package up the software into some kind of “archive format”.
  • configuring. Source code is often written to be able to function in a variety of environments, such as on different operating systems or even on different kinds of hardware. Some software may also function differently depending on what other software is installed on the system already. Lastly, users may wish to customize how the software behaves or is instaleld. So we often have to configure “the build” to do the proper things for any particular environment.
  • installing. After software has been built and configured, we often copy it into specific locations on the target system, register it with the operating system as a service, or something simliar.
  • distributing. After we package software (or, also common, before we package it), we share it with others by publishing it on the internet, a cd-rom or similar medium, or by submitting our changes to a centralized or
    decentralized version control system (called a “checkin”).

Marketing blah blah

“SCM” is a marketing term often used by pointy-haired bosses in the business of selling “software consultancy” services to other pointy-haired bosses. It stands for software configuration management, and its mostly about “managing” these different activities by following some sort of process, and integrating that “software process” into a “business process”.

There’s a lot of different terms and phrases for describing this kind of thing, such as CMM(I) (levels 1 through 3) a whole bunch of ISO standards, as well as fancy terms such as “eXtreme Programming” or “agile development” for when you
know CMM is a silly phrase yet you still need to sell your company as “conforming to” or “applying” some sort of process.

There’s also something akin to the “open source software management process” which involves ideas such as “automated everything” and “just use apt-get” which is generally a much more sensible (if under-documented) “process”. Karl Fogel
has written a book on producing open source software which goes into this to some detail (http://producingoss.com/html-chunk/index.html).

Real life software management

In reality, no-one in the real world developing real software really follows the fancy flow charts. A rather generic SCM flowchart which incorporates many common process looks roughly like this:

Flow chart depicting typical real-life software management

In this chart, each and every step could probably be skipped or replaced by a “no-op”, and some arrows are followed more commonly than others.

For example, most software developers tend to follow an iterative process when writing code. They’ll “switch into development mode”, change some source, rebuild and test the change, change some more source, rebuild and test, check in
the changes, rebuild and test, create a distribution package, “switch to production mode”, get the distribution, build and test it, do some configuration, rebuild, then install. Then when they get called in the middle of
the night because the software just crashed, they’ll often not follow any kind of process at all and do things like change the running program on the live machine.

If the above paragraph sounds confusing, trust me, all this stuff is. “Managing software” is not a simple task. No wonder the average programmer or system administrator has a strong caffeine addiction.

Building a better (make|ant|maven|…)

Software build systems are broken. Way broken. I want them fixed. I have been thinking about that for many years now. I still haven’t figured out how to go about fixing it all (yes, I really do mean all). Like so many other people, I’d love to.

Background

I have been annoyed and intrigued with the process of compiling, building and installing software ever since first being told to “just type ./configure && make && sudo make install“. I asked “if its always the same, why isn’t it just install?”.

I have the same experience with build systems that average Joe Programmer has, and then a little more.

  • I’ve fixed (and of course, also broken at times) build systems for just about every apache project I’ve ever worked on.
  • I’ve built my own linux from scratch at some point before there was linux from scratch.
  • When I wrote huge and ugly javascript libraries, I wrote an rpm-like system to manage them.
  • It seems I tend to drift toward the position of build/release/integration manager (“slave” is a better word actually) for the projects I get involved in.
  • the most core of core technologies on which maven 2 is built is based on a design and an architecture I helped a little to dream up.
  • I have been one of the most active developers for one of the most misunderstood continuous integration servers for a long time now.

Research

I’ve been watching the way all the java projects at apache manage their builds since before Ant ever became anything close to a standard. I’ve been watching the development of Ant and Maven and then Maven 2 for the longest time. I’ve read most of their codebases.

For the last year or so, I’ve been learning more and more about all the non-java build systems, integration systems, packaging systems, deployment systems. Well, not really all of them, just the open source stuff.

  • I’ve read all the documentation I’ve been able to find on make, autoconf, automake, xmake, qmake, cons, scons, jam, gcc, yacc, lex, rpm, apt, yum, FreeBSD ports, and many similar tools and systems. Xcode, too.
  • I’ve used all of those tools (or tried to use them) on different projects, and encountered and fixed (or usually, worked around) many of the common problems one finds with these kinds of tools.
  • I’ve read large parts of the GNU make and autoconf source code.
  • I’ve read all of the core scons codebase at least twice.

Money, Business, Open source

Since I’m now actually trying (succeeding too, so far, fortunately) to make a living by writing software (and stuff), I’ve also been thinking about how I might make a living working on this itch of mine that I have. I want to fix all those frigging broken build systems out there, and fix them properly. I want to help all those brilliant (and the not so brilliant too) software developers out there be a little less frustrated when their stuff just won’t build. I want to help all those CTOs out there to be able to reduce the “overhead of the build”.

So, I’ve been looking at how others make a living working on build systems. It seems most of these things are actually built mainly on the free time of its developers, out of passion, or more likely, frustration. Companies that try and focus on solving these problems are either giants like IBM, Microsoft or Red Hat, or don’t seem to last very long, or don’t seem to succeed at helping define a new sort of standard. Open source is probably (and hopefully) a key factor to success.

I’m sort-of interested in what’s going on with Mergere, a startup funded by Simula Labs, which employs a few of the Maven 2 people. I have a hunch I wouldn’t enjoy working there (or at a similar company) very much. And so far it seems they’re all about java, which is just too boring.

I’m less interested in how most other “open sourcy” players in this space go about their business. Eg CollabNet, SourceLabs, Canonical. It seems they have rather obvious business models and a rather solid evolutionary plan that helps to gradually improve the software market place. I imagine this is great stuff if you’re building a solid long-term company with employees who have kids with mouths to feed, but I’m not looking to build a company, I don’t have kids, and I’m not too hungry either.

Hack the planet

Heh. I’d like to do something a little more disruptive, something a little revolutionary.

I know there’s a lot of people out there who share largely the same itch. I wonder, I wonder, what would get those people together with their noses in the right direction to produce a single absolutely stunning and brilliant solution which will still be in use many many years from now…

Memories from ApacheCon US 2005

Leo Simons at the ApacheCon US 2005 Lightning Lottery Talks, holding two beers and an iBook

Talk about blogging backlog. Over on my old blog, the last entry is about the day before the hackathon before apachecon. The longtime reader will know by now that after that I got a little too busy doing other things to be blogging about those things. Fortunately, other people took pictures. Shockingly, this picture of my 5 minutes of fame bashing the ASF at the lightning lottery talks is not under a creative comments license, so I don’t think I’m allowed to reproduce it here. On the other hand, I think I’m allowed to show a thumbnail…

Update: Ted promptly put his ApacheCon photos on flickr under the creative commons license. Thank you!