Running as Root

I finally had something useful happen at work. We had a consultant that uses one of our products during the course of his job come in and give a quick presentation on how he uses the tool. Unfortunately, it's not the product I work on, but it was still interesting for a number of other reasons. As quick background, the tool demo'ed is a storage monitoring tool.

Three Users of Enterprise Software

Nothing new here, but Enterprise software users typically fall into three categories: experts in a domain using the software to make their tasks easier, users largely ignorant of the domain that want an expert in a box to tell them what to do, and executives that want some charts to determine if things are running smoothly: the infamous "dashboard."

The person we had in was definitely an expert in the domain. Listening to how he uses the software to determine if a customer's installation is running smoothly was very educational. Often, features seem to get added to a product by people that don't really understand how those features are going to be used. It's put in just in case someone might find it useful or, worse yet, a customer demanded it without explaining why the wanted it.

This part interested me because, through his demo, he identified a ton of information that the "expert in a box" customer would drool over. Apparently the way he used different pieces of information wasn't just news to me, it was news to the current development team as well. They furiously scribbled while he showed how he used combinations of charts to find configuration errors, determine the need for new equipment, and a number of other things. He also suggested some features that would make his life easier and explained why he didn't use certain parts of the system at all.

Hired Guns

I, like most people, really value the input of a customer or, if that's not available, a customer advocate. The problem is that most product owners in a typical company don't seem to really understand what a customer needs. They pay more attention to what the competition is doing or at most what the squeakier wheels in their customer base say they want.

As early as possible you should post a position on one of the online job sites looking for an expert in whatever problem domain you're working. Throw them a few thousand dollars or so to just come in and show you what it is they do. Do all the standard information mining stuff on the poor bastard. Make sure the developers are involved. Try and build a tool that is useful to him and at the same time captures some of his knowledge for the EIAB (expert in a box) user. After you build a few iterations of the product, ask another one in to evaluate the product. Repeat, get the EIAB user to check it out, etc etc. Did I mention that you should make sure the developers are involved?

Ship your product, sell your product, host a message board for customers of your product, make the developers participate on the board. Keep that interaction with the customer going. At some point, throw a few extra grand at an expert that uses your product and record a demo session to include in your product. Something to orient the expert customer and maybe teach the EIAB user to be more effective. It's not like anyone reads the manual. That training video can double as sales literature, too.

No Shit

Yeah, it's pretty obvious. A veritable no-brainer. I'm ashamed for even suggesting such an obvious idea. Now why, if it's so obvious, is it that I've never seen this done at any organization at which I've worked?

We Can Rebuild It

The dream of most of the software developers I know is to get the opportunity to rewrite a system they work on from scratch. I think as things move more toward test driven development (TDD) this desire may fade a bit, but never entirely disappear. TDD should allow developers to aggressively refactor an existing product rather than starting from scratch. There will be less fear of hopelessly breaking the system if you've got solid tests and good test coverage. Of course, even getting the opportunity to refactor may be rare in many cases. The product owners don't always care how dirty the implementation is if there are no visible effects in the final product (at least not right now anyway). The customer won't buy the product because it has pretty source code behind it. A damn shame.

The incessant desire of software developers to rewrite all code they encounter is perfectly natural in my opinion. It's the plight of the knowledge worker. Each day you collect new information and techniques. When you look at someone else's code (maybe even your own from 6 months ago) you can't help but feel that there is a better way of doing it. If you have to write new code within the constraints laid down by this less than ideal solution, you're eventually going to come to the conclusion that you should just rewrite the whole thing. You just don't like having to work with / around whatever it is you've been given. Maintaining someone else's software is like putting someone else's sweaty clothes on. That opportunity to refactor is crucial.

From Hacks to Good Software

I was talking with another of the developers on my team a few days ago about how we should fix a problem we're having. We're working under a deadline and may not have time to implement the more ideal solution. I proposed a couple of different stopgap solutions that would get us to our release date. Most of the ideas were met with the infamous developer credo, "but that's a hack." I definitely understand the sentiment, but unfortunately we have to ship some software. And in my experience, most good solutions started as a hack. If you properly encapsulate everything you can go back and improve that hack into a decent solution with minimal impact. But again, willingness to accept the less than ideal solution is much easier if you know you will be able / allowed to refactor it later.

Now, does the world work this way? Hell no. So far, for me anyway, TDD is hard. I seem to always find myself smack dab in the middle of a container or a framework that makes effective testing extremely difficult to do. That inevitably gets coupled with a product development cycle that allows no time to retro-fit tests and test harnesses into the existing product. On top of that, the product managers often don't see the value of refactoring unless they get a ton of new features with it.

Usually in a simple situation you just start doing TDD for all features moving forward and write a test for every bug you find. The idea is that eventually you'll get there. For cases where you can't do the initial work in small chunks I guess you're just screwed. Eventually you'll have to bite the bullet and do whatever big tasks are necessary to start you down the whole TDD road. If you can't do that, it's probably time to move on and try to make sure that the next place doesn't suffer from the same problems.

Any Questions?

In the back of my mind I keep filing away little tidbits like this for later use so that I can come up with a decent set of questions to ask any potential employers. In a typical interview I forget the importance of grilling the employer. I need a set of questions like the Joel Test. I think the first one I need to add is: Can I see a copy of your code coverage reports, preferably over the last 3 months?

Update

Matt Kinman rocks!

I'm farting around with CruiseControl at work to kick off a Maven build script. CruiseControl is a pretty handy tool, although I still manage to hit my head on its limitations whenever I want to do something odd. Like associate a publisher specifically with one of the builds within the schedule. Or make only one of the builds in the schedule ignore the need for modifications in the source repository before kicking off. Whatever. It's still well worth the money.

A couple of weeks ago I'm doing some stuff and things with the Perl guy I work with. We'll call him John because that's his name. Well old John checks in a change in the maven.xml that breaks the build. We know it broke the build because CruiseControl sent us a hateful email. We just fixed the problem and checked in the file. CruiseControl should pick up the change and send out a "build fixed" email. Well, it never did.

After looking at the build box we found out that the changes weren't being picked up from our Subversion server. We were in a bit of a hurry so we just did a manual svn up on the CruiseControl box (a virtual machine actually ;). Well, today I finally got my dumb ass to figure out what was wrong. The source control update within CruiseControl happens like this: maven scm:update

Meaning, if you broke the build so badly that it won't even run, you'll never get the changes that fix the problem–because they happen through the build. Get it? They happen through the build. (crickets) I guess you had to be there.

At work, I have a separate test machine I use to deploy the software on which I'm working. The demands the container(s) and framework make on the host machine are borderline too high to run on my laptop comfortably so the test machine is a very good solution. As I mentioned before we use a lot of virtual machines here. My test server is running Debian but the environment I'm deploying to is actually a VM running Windows via an instance of VMware Player. I run the player by ssh'ing to the test machine and then do some X11 port forwarding to my local Cygwin X server just so I can futz around with the instance of Windows if need be…

I'm planning on working from home this afternoon and some of this weekend. In order to test my changes I need somewhere to deploy the application. Unfortunately, I haven't set up our VPN client yet (plus I hear our VPN is dog ass slow). Well, thanks to those handy-dandy VMs, I can pick up a copy of my server and take it with me. Then I can either run it on my laptop (and temporarily suffer through the somewhat acceptable performance degradation) or I can drop it on any of the machines I have at home. Oh, daddy likes.

My favorite way to communicate with people, besides delivering an open handed slap in the mouth, is through the use of similes, metaphors, and analogies. One of my co-workers told me that our man Klobe dropped a good one recently. I am paraphrasing, so please forgive me if I get it slightly wrong.

He said that software architecture is like a big sticky ball (I envision a ball of duct tape, sticky side out). You roll it around on whatever concepts or features you want to implement and it "picks up" anything you roll it over–for awhile. Eventually though, all of the sticky spots are covered up and it just won't pick things up any more.

It means that your product can definitely outgrow your architecture. After that happens, it's nearly impossible to keep ramming features in unless you fix the bigger issue. The new stuff just won't "stick." Maybe you started with a small ball of tape that is the equivalent of a set of shell scripts you used to do things. This may have become an actual commercial product later on. However, given these humble beginnings, you can only add so many features before you need to address the fact that you don't have an architecture that can support the set of "enterprise level features" you want to deliver.

Most people that really work on software get this, but it seems like a really good metaphor one could use to quickly sum up the issue for an executive, especially when you're trying to justify all of those overhead tasks you need to do in order to fix things.

It's (Third) Party Time

A lot of computer products, particularly in the monitoring / administration domain, are centered around some third party piece of hardware of software. Usually a company finds that they can greatly enhance and simplify the admin and reporting on another company's flagship product. Presumably the original company is either unable or unwilling to make it easy manage their stuff because they're concentrating on their core problem domain.

As an example, if you build backup servers, you're working on getting that backup stuff working smoothly, not necessarily on making it easy to manage or get information out of it. Sure, you include some out of the box support for doing some of the stuff, but ultimately it's an immature solution that feels like it was tacked on at the last second. It's particularly bad with hardware vendors. They usually can't design decent user software or APIs.

This creates a great opportunity for software companies to swoop in and carve out a spot in (or create a market for) the area of third party management software. If you can write a piece of software that makes the administration of Foo servers more simple, particularly in a large enterprise environment where the out of the box tools break down, then you'll be sitting pretty for a while.

You Mean, We Can Make Money Off This?

One of the problems occurs when the original vendor suddenly realizes that their software sucks. If they ever get off their ass and write some decent management stuff around their product and bundle it for free, then customers have much less incentive to go looking for your offerings in the first place. If other companies begin offering management solutions around the same product, you will see even more of a decline in sales.

The temptation seems to be to get into a feature war with everyone else. Once you've exhausted all of the useful features you start to include numerous niche features and one-off customer requests. Dump these onto a product sheet in no particular order and you look just as good, if not better than all of your competitors. A lot of these features aren't likely to matter very much to the people that use the software, so you need to get some marketing materials, product advocates, consultants, etc to push your product to the people that make the decisions (and don't actually use the product). For more on this you can see the ClearCase situation in another post.

Now you've lost your way. You started out simplifying the administration of another company's product and have created what is, more than likely, an overly complicated and hard to use product of your own.

Wider Is Better

Rather than freak out and get all feature crazy when the competition starts, it's better to diversify what it is your product simplifies. For example, if you've a database tool for a particular backend, why not add support for another database backend before exhausting your laundry list of niche features? Of course this works better if you're simplifying or abstracting multiple offerings from vendors in the same problem domain that are likely to be found uneasily co-existing in large organizations.

By that I mean, you want to do something like take three operating systems and present a common management interface for a problem domain (like security or backup) that sensibly abstracts away the subtle differences. This allows an admin to more easily administer the hodge-podge of systems he's found himself in charge of. And, since the same person is likely to have to deal with all three, your tool is quite handy. Handier in fact than if three people each only deal with one of the situations. Then the fact that your tool acts as an abstraction for all three isn't particularly useful, especially since each of them will have their own, very deep on functionality, tool that they prefer. No, I think the better target is to go wide and relatively shallow. It's also better insulates you from the competition, either from the original vendors (who usually don't care about working with products from other vendors) or other third party companies (who may not have their act fully together).

And Your Point Is?

What's my point? Up to now, what I'm talking about is fairly obvious to most people. The issue comes in when you consider how your original, one vendor product came to be. I've seen a lot of software teams and companies paint themselves into corners by making the assumption that their product only works with one vendor product or by tightly coupling domain / business logic to the first product they support. When they finally get around to adding support to more products they often find out their "abstraction" doesn't play well with others. By then it's too late. It's all well and good to only build exactly what you need at the time, but you need to think about where you're going. This is something I haven't fully figured out with agile development. This "architectural escape velocity" doesn't seem to come easily in the bite size chunks that agile advocates. I'm not saying it can't happen, it's just very easy to get it very wrong.

As early as you can, hopefully when you initially start your project, you might want to start asking these questions:

• What other vendors' products could we manage?
• How have the vendors represented this domain differently?
• How can we create a unified view (abstraction) of this domain?
• If something doesn't fit the abstraction, how can it still fit into our product?
• How do we keep one-off customer requests from making our core product less maintainable?
• How can customers get information into and out of our system?

That's just a start. I'm sure there are more (and probably better) questions to keep asking yourself. Answer these early and often (like you release) and let it affect your planning. Don't be tempted to just say, "we'll worry about that later" because by then it'll cost too much to change.

VMs in the New Job

I started a new job in early November of 2005. The first thing I noticed when I started the new job was that all of the programmers and QA testers were constantly talking about their VMs (virtual machines). Each developer has a physical test machine (in addition to their desktop / laptop) located in a lab. This machine typically runs something like Debian and then uses either VMware or Xen to run a virtual machine, usually with a different operating system. The developers then use the VM to install different versions of the product.

This completely isolates the install / test environment from the dev environment. While this setup is not particularly useful for testing performance, it is amazingly useful for managing different configurations and versions of the product on which you're working. For example, let's say you're working on version 1.1 of the product and testing the upgrade scripts from version 1.0. You set up a VM with version 1.0 of the product. Since the VM itself is merely a set of files you can then keep a baseline copy of this configuration in another location. You run your install scripts on the VM and if there are any problems you can simply stop the VM, copy some files, and be back to your uncontaminated 1.0 install–ready to test the scripts again after you fix the problem. The same applies for testing the installer for the product (and the uninstaller I suppose).

Dog Food Buffet

You can also keep copies of Windows 2000, Windows XP Pro, Windows 2003 Server, and different distros of Linux around for testing without dedicating an entire machine to each OS / product configuration combination. Using a VM to dedicate a machine to a product install also means it's easier for people to eat their own dog food as Cote' is fond of saying. The IT department of my new company also uses VMs to run the Subversion server, the CruiseControl server, a maven proxy, and any number of other services. The VM has its own IP address so you can pick up and move the entire "machine" any time you want to re-arrange things.

Making it Happen

This VM-based approach has a lot of potential. It's what motivated one of my new co-workers to only half jokingly name 2006 the "year of the VM." Are there problems with this approach? Of course–there are always trade offs. Performance under a VM can suffer. There can be subtle differences between a VM install and a physical machine install that are extremely difficult to track down. I think the increase in flexibility makes it very much worth it to work a VM-based solution into your IT environment.

Assuming your organization isn't on board with this setup, you may still be able to sneak in some limited VM action into your workflow. To start with, VMware's VMware Player is currently free. You usually need a copy of VMware Workstation to create new VMs (although a number of pre-built VMs are available). However, you can find quite a few hacks on the internet that allow you to create VMware VMs for free. The virtualization.info blog provides a good starting point for locating these tools. Then all you need to do is get a copy of the target OS (usually not a problem in a large organization) and create a new VM. Ideally, you can run this VM on some dedicated physical machine (again, most developers in a normal IT organization have a test machine of some sort), but in a pinch you can run it on your main machine–you'll just feel the pain of poor performance.

Now that you're set up you can do things like create a copy of this VM for each major version of your product. Whenever you need to see how something worked in another version, you just fire up a VM and check it out. Or, if you're working on an upgrade path between versions, you can do the baseline copy of the pre-upgrade install like I mentioned earlier. If it works out well for you, start spreading the word to other developers and trading VMs (or setting up a central repository for more useful configurations). Soon enough it gets enough momentum to become an official part of the process.

The project I'm working on is looking into adding some code coverage reports to the nightly build. The benefits of this should be fairly obvious–it allows you to see how good of a job your unit tests are doing. Up until recently, I had stared longingly at Clover. Unforutnately, it is a commercial product and, as usual, it is much easier to convince others something is useful if the product itself is free (ignoring any TCO issues).

That's why I was initially so excited to stumble upon Emma recently. It seems to do everything I need, is easy to work with, produces attractive HTML reports, and is free. Just when I thought everything was settled, I also stumbled upon Cobertura. It is a fork of the GPL version of jcoverage. I avoided jcoverage initially because I don't particularly like using the free version of a commercial tool. They're usually intentionally crippled. Cobertura however seems to once again do everything I need. It also has some pretty swank looking reports.

I plan on looking at Cobertura over the next couple of days and, barring any painful integration / use issues, I should be able to make a final suggestion based solely on how good the repots look.

Here is a quick look at some samples of Emma's reports along with Cobertura's samples.