Running as Root

We Can Rebuild It

The dream of most of the software developers I know is to get the opportunity to rewrite a system they work on from scratch. I think as things move more toward test driven development (TDD) this desire may fade a bit, but never entirely disappear. TDD should allow developers to aggressively refactor an existing product rather than starting from scratch. There will be less fear of hopelessly breaking the system if you've got solid tests and good test coverage. Of course, even getting the opportunity to refactor may be rare in many cases. The product owners don't always care how dirty the implementation is if there are no visible effects in the final product (at least not right now anyway). The customer won't buy the product because it has pretty source code behind it. A damn shame.

The incessant desire of software developers to rewrite all code they encounter is perfectly natural in my opinion. It's the plight of the knowledge worker. Each day you collect new information and techniques. When you look at someone else's code (maybe even your own from 6 months ago) you can't help but feel that there is a better way of doing it. If you have to write new code within the constraints laid down by this less than ideal solution, you're eventually going to come to the conclusion that you should just rewrite the whole thing. You just don't like having to work with / around whatever it is you've been given. Maintaining someone else's software is like putting someone else's sweaty clothes on. That opportunity to refactor is crucial.

From Hacks to Good Software

I was talking with another of the developers on my team a few days ago about how we should fix a problem we're having. We're working under a deadline and may not have time to implement the more ideal solution. I proposed a couple of different stopgap solutions that would get us to our release date. Most of the ideas were met with the infamous developer credo, "but that's a hack." I definitely understand the sentiment, but unfortunately we have to ship some software. And in my experience, most good solutions started as a hack. If you properly encapsulate everything you can go back and improve that hack into a decent solution with minimal impact. But again, willingness to accept the less than ideal solution is much easier if you know you will be able / allowed to refactor it later.

Now, does the world work this way? Hell no. So far, for me anyway, TDD is hard. I seem to always find myself smack dab in the middle of a container or a framework that makes effective testing extremely difficult to do. That inevitably gets coupled with a product development cycle that allows no time to retro-fit tests and test harnesses into the existing product. On top of that, the product managers often don't see the value of refactoring unless they get a ton of new features with it.

Usually in a simple situation you just start doing TDD for all features moving forward and write a test for every bug you find. The idea is that eventually you'll get there. For cases where you can't do the initial work in small chunks I guess you're just screwed. Eventually you'll have to bite the bullet and do whatever big tasks are necessary to start you down the whole TDD road. If you can't do that, it's probably time to move on and try to make sure that the next place doesn't suffer from the same problems.

Any Questions?

In the back of my mind I keep filing away little tidbits like this for later use so that I can come up with a decent set of questions to ask any potential employers. In a typical interview I forget the importance of grilling the employer. I need a set of questions like the Joel Test. I think the first one I need to add is: Can I see a copy of your code coverage reports, preferably over the last 3 months?

Update

Matt Kinman rocks!

I'm working on a very small team at the moment and everyone I need to talk to is, at most, several cubicles away (don't get me started on how much cubicles suck, by the way). Today I started working on something completely new to me. Luckily, one of my nearby co-workers is intimately familiar with it and he sits 8 steps away. When I got started I asked him everything I thought I would need and went back to my desk. Several minutes later expectations weren't meeting reality and I had more questions. He answered those and asked some of his own about something on which he's working. About twenty minutes later he had another question about something in Windows and I had another question about my stuff.

I've worked on teams with members in different cities that were 1) in the same time zone, 2) in time zones a couple of hours earlier / later, and 3) with people half a world away. While number 3 sucked the most (over a 12 hour turnaround time on email, no hope of phone conversations) the inability to fire off a quick question or two without picking up the phone, the complete lack of feedback from facial expressions and body language, and the lack of being able to overhear a conversation related to what you're doing have all been irritating and wasteful.

I think a lot of higher ups are too concerned at using the resources they already have in different cities or of saving tons of money by offshoring a part of the project. From down here at ground level I would say that it is a mistake. The best and most efficient working situations I've been in involved everyone being less than 30 seconds of walking distance away from each other. Less time is wasted on trying to exhaustively and preemptively document / explain everything. You don't have to wait for return phone calls. It's harder to blow people off or hand off crappy work when you have to see them in the halls (not that I would ever do such things).

Although there are solutions to alleviate problems when it is just impossible to have a co-located team (video conferencing, teleconferencing, instant messaging, VNC / remote desktop, etc) I think most of the people that have to do the actual work would agree there is no real substitute. As such, I think I need to work this into my employer interview questions.

It's (Third) Party Time

A lot of computer products, particularly in the monitoring / administration domain, are centered around some third party piece of hardware of software. Usually a company finds that they can greatly enhance and simplify the admin and reporting on another company's flagship product. Presumably the original company is either unable or unwilling to make it easy manage their stuff because they're concentrating on their core problem domain.

As an example, if you build backup servers, you're working on getting that backup stuff working smoothly, not necessarily on making it easy to manage or get information out of it. Sure, you include some out of the box support for doing some of the stuff, but ultimately it's an immature solution that feels like it was tacked on at the last second. It's particularly bad with hardware vendors. They usually can't design decent user software or APIs.

This creates a great opportunity for software companies to swoop in and carve out a spot in (or create a market for) the area of third party management software. If you can write a piece of software that makes the administration of Foo servers more simple, particularly in a large enterprise environment where the out of the box tools break down, then you'll be sitting pretty for a while.

You Mean, We Can Make Money Off This?

One of the problems occurs when the original vendor suddenly realizes that their software sucks. If they ever get off their ass and write some decent management stuff around their product and bundle it for free, then customers have much less incentive to go looking for your offerings in the first place. If other companies begin offering management solutions around the same product, you will see even more of a decline in sales.

The temptation seems to be to get into a feature war with everyone else. Once you've exhausted all of the useful features you start to include numerous niche features and one-off customer requests. Dump these onto a product sheet in no particular order and you look just as good, if not better than all of your competitors. A lot of these features aren't likely to matter very much to the people that use the software, so you need to get some marketing materials, product advocates, consultants, etc to push your product to the people that make the decisions (and don't actually use the product). For more on this you can see the ClearCase situation in another post.

Now you've lost your way. You started out simplifying the administration of another company's product and have created what is, more than likely, an overly complicated and hard to use product of your own.

Wider Is Better

Rather than freak out and get all feature crazy when the competition starts, it's better to diversify what it is your product simplifies. For example, if you've a database tool for a particular backend, why not add support for another database backend before exhausting your laundry list of niche features? Of course this works better if you're simplifying or abstracting multiple offerings from vendors in the same problem domain that are likely to be found uneasily co-existing in large organizations.

By that I mean, you want to do something like take three operating systems and present a common management interface for a problem domain (like security or backup) that sensibly abstracts away the subtle differences. This allows an admin to more easily administer the hodge-podge of systems he's found himself in charge of. And, since the same person is likely to have to deal with all three, your tool is quite handy. Handier in fact than if three people each only deal with one of the situations. Then the fact that your tool acts as an abstraction for all three isn't particularly useful, especially since each of them will have their own, very deep on functionality, tool that they prefer. No, I think the better target is to go wide and relatively shallow. It's also better insulates you from the competition, either from the original vendors (who usually don't care about working with products from other vendors) or other third party companies (who may not have their act fully together).

And Your Point Is?

What's my point? Up to now, what I'm talking about is fairly obvious to most people. The issue comes in when you consider how your original, one vendor product came to be. I've seen a lot of software teams and companies paint themselves into corners by making the assumption that their product only works with one vendor product or by tightly coupling domain / business logic to the first product they support. When they finally get around to adding support to more products they often find out their "abstraction" doesn't play well with others. By then it's too late. It's all well and good to only build exactly what you need at the time, but you need to think about where you're going. This is something I haven't fully figured out with agile development. This "architectural escape velocity" doesn't seem to come easily in the bite size chunks that agile advocates. I'm not saying it can't happen, it's just very easy to get it very wrong.

As early as you can, hopefully when you initially start your project, you might want to start asking these questions:

• What other vendors' products could we manage?
• How have the vendors represented this domain differently?
• How can we create a unified view (abstraction) of this domain?
• If something doesn't fit the abstraction, how can it still fit into our product?
• How do we keep one-off customer requests from making our core product less maintainable?
• How can customers get information into and out of our system?

That's just a start. I'm sure there are more (and probably better) questions to keep asking yourself. Answer these early and often (like you release) and let it affect your planning. Don't be tempted to just say, "we'll worry about that later" because by then it'll cost too much to change.

Executing

Guy Kawasaki has a great post on the art of execution. There are many good points in it and, besides being a guide to organizations just starting out, it would serve as a great refresher to organizations that are losing their way. The problem can often be convincing companies that they're doing something wrong in the first place. Often, a company experiencing some level of success is very reluctant to admit that they're screwing something up ("We're making money, so we must be doing things right"). I'm very fond of the idea that one of the greatest impediments to progress is your current level of limited success. Executives also always seem to get unnecessarily defensive about too many suggestions of how to improve the status quo. I digress.

Although every point is relevant and valuable, from my experience the most powerful points (and the ones most organizations fail at) in the article are:

• set achievable goals
• communicate the goals
• measure progress on a weekly basis
• establish a single point of responsibility
• reward the achievers
• heed your “Morpheus”

Achievable Goals

Nothing is more demoralizing than having a set of impossible goals. Managers sometimes mistakenly believe that having these difficult goals will motivate people. It doesn't. Usually, after the initial planning / kickoff meeting the people responsible for attaining those goals go back to their offices and immediately start telling each other how ridiculous the goals are. Then, throughout the rest of the release, they make only passing sarcastic remarks to the preposterous goals and accept the reality that they won't even come close to the goals set for them. They then work the normal amount (maybe even slightly less) and complain to each other in passing about their current work situation. Management is labelled as a bunch of out of touch idiots and this serves as another reason for people to get another job.

Communicate the Goals

Management usually does a mostly adequate job of communicating goals. I mention this item here mainly as an opportunity to increase the trasparency within an organization. Management should start a blog laying out the goals and post regular updates as to the progress toward those goals.

Measure Progress on a Weekly Basis

Despite being told that it's the ultra cool generation Y that likes getting regular reviews of their performance, I think everyone in the organization (regardless of their generation moniker) needs a regular update on progress. It's the only way to make the necessary course corrections along the way. It's also very agile, which is greatly in vogue at the moment (and rightfully so).

Establish a Single Point of Responsibility

This is probably one of the greatest failings of organizations of which I have been a part. I think it's essential to have a point person for any goal. Unfortunately, no one wants their ass in the fire when things go wrong. If that's the case, you probably have a company that engages in way too much CYA and blamestorming during a project. Accountability can be a good thing, but when people are held accountable for not meeting impossible goals with inadequate support / tools, then no one will want to be saddled with any kind of responsibility. And, if no one is willing to step up, nothing is going to get done.

Reward the Achievers

This mostly speaks for itself. Sometimes, though, rewards go to the wrong people or they don't trickle down from project managers to the people that really made things happen. I've also seen organizations that are scared to reward people because other employees will get jealous of those rewarded. If your reward policy is clearly defined and fairly applied, those people just need to get over it. If it's not, then of course you need to fix the system rather than "not reward" everyone equally.

Heed Your “Morpheus”

This is a reference to the Matrix and the fact that the character Morpheus gives Neo the pill that exposes the world for what it really is. These people are the ones in an organization that aren't "drinking the Kool-Aid." The risk is that they're simply labelled as bitchers and complainers then ignored. As these people often have valid points see also Cassandra. The role of the "monkey wrencher" is severely undervalued in many companies. Managers and fellow employees don't want to hear how things aren't going to work, especially if they're all powerless to fix things. You need to find a way to make this a constructive role and protect the person that's pointing out that the emperor has no clothes. Ignoring risks is not the same as managing them–when they're brought to the surface, make sure you heed the warning and don't shoot the messenger.

Do These Things

Do these things well and you increase your chances of success. You're also likely to be light years ahead of your competition, since most places don't apply these points very well, if at all.

Humble Beginnings

Around ten years ago I was working for a pretty backwards IT department. We wrote and maintained an internally deployed call center application in Foxpro. We didn't use any source control. The shared libraries for the multiple, customer specific versions of the application were on a shared network drive. When you ran the build script, a new version of the application was instantly put on another shared drive and almost as quickly wound up on a phone agent's desktop.

Over a weekend, one of the lead developers decided that we should have some sort of source control. Rather than use any of the many mature (and free) versions of source control, he decided to write his own–in Foxpro of course. His version of source control simply made everything on the shared library drive read only. You then ran his program (named "vc") on some component and it would log that you had it checked out. When you checked it in, it made the shared version writable, copied your version over it, and logged that you checked it in. That's it. There was no tracking of revisions, no versioning, not much of anything really.

When the department finally decided that this solution wasn't good enough (a couple of years later) several people recommended using CVS. However, now that they were intent on improving their process, the management felt that CVS just wasn't good enough for our unique needs. Someone from outside the company sold them on using ClearCase at several thousand dollars per developer. Of course, we didn't use or need any of the supposed features of the product. In my eyes, ClearCase was/is no better than CVS. From the management point of view it was better because we theoretically got support, it better fit our "complicated" development process, and someone outside the company told them it was better.

New Company, Same Issues

A couple of jobs and many years later I found myself in a much bigger, more mature IT department. The new problem wasn't source control, it was the process. We were using a painful waterfall development process. The management over the IT organization decided we should switch to a more agile process.

Most of the developers I worked with viewed this as a very positive change. Up to this point, all project management occurred in Microsoft Project and Word files. Now that we were changing things up, we could probably use a different set of planning tools. One of my co-workers (Matt Ray) had used XPlanner at a previous job and felt it was "good enough." Somewhere in the decision making process it was decided we'd have Rally come in and teach us how to be more agile. We were also going to use their project management tool to organize our tasks. Being well outside the decision making process on this, I'm not exactly sure how much money all of this cost us, but I suspect it was quite a bit. Was Rally's mentoring valuable? I think that it was, but I think that its primary value was that it is always easier to listen to someone from outside of your own company rather than your own people. Is Rally's tool better than XPlanner? I think it is a better overall tool, but it doesn't really do anything significantly better. XPlanner really is good enough for what we needed in my opinion.

The Tools Aren't the Thing

I'm now at another job in a relatively immature IT department. We've started using an agile development process and are using XPlanner to plan our stories and tasks for each iteration. A couple of days ago, during iteration planning, one of the developers was wondering if Rally's product had a feature they felt was missing from XPlanner. In this particular case, it didn't, but it made me think about peoples' love of tools.

It never seems to fail–an IT organization decides that they're doing something wrong and the solution is to pay someone from outside the company to tell them what expensive product will fix it. Usually these outsiders tell you what the more competent people in your organization already know. However, it gives the upper management great peace of mind to have someone that makes a living telling people how to do things give them the answer to their problems. I can almost understand this but it still baffles me that, if you believe you've hired good people, you aren't listening to them. The much stranger thing is that so many people are willing to go from using a bad solution to shelling out big money for a tool with a set of features and support options they'll never use. They're convinced that one of the major problems with their process isn't the process itself–it's the tools. The same thing happens with people on a personal level. I could get buff if I had a gym membership and I could get organized if I had a PDA. This is the easy "fix." Just buy a silver bullet to make the problem go away. People and organizations need to learn that they need to address the root of the problem first, usually with free tools that are "good enough" (if not outright better than their commercial counterparts). Once you do that, you're free to buy some tools to eek out a few extra points of efficiency if you still feel the need. But, most of what you need to do is purely behavioral and largely free to start solving–at least in terms of money.

Update

I just saw another tool named ScrumWorks that apparently recently became free to registered users. If XPlanner doesn't do it for you, you might give this tool a whirl. I personally haven't looked at it very much.

When I switched jobs recently, one of the reasons I cited was our frustratingly slow and/or incorrect adoption of an Agile methodology. I always have problems organizing my thoughts as to what we were doing wrong, but here's a good post the summarizes some common Agile stumbling blocks. I think we were having problems with variations on 2 through 5 primarily:

No Executive Sponsorship

Well, we sort of did. It was more of a lip service type of thing. Other developer experience certainly varied from mine, but my personal experience was, "if you see anything wrong please point it out so we can promptly ignore it." I think this is a consequence of the fact that it's a company that grew through acquisition (so there are a lot of turf related politics keeping things from getting done) and that it's such a large company. I was simultaneously amazed and disappointed at the rate of progress, if that makes any sense.

Offshore/Outsourced Developers

Well, we had some offshore QA team members in my case, not developers. But we did have developers in two cities. They were in the same timezone though, so I guess that's something. I feel that having everyone in the same office would have been a huge help. That notion of eliminating that wasteful communication and coordination layer by being able to meet face to face seems very appealing. Of course, when you're a big company, you use all these resources you already have regardless of whether or not they're the best location/configuration for the job. Kind of like using WWII surplus supplies during the Korean "conflict." You see, watching all of those MASH episodes finally paid off.

Lingering Waterfall elements

I think this is one of the biggest things we suffered from. Speaking only for the team I was on, we went into our Agile adoption with a strict set of deliverables and a hard deadline and very little flexibility with regards to either. We also did not have shippable software at the end of each iteration and we knew it. There also was no time in the schedule for going back to revisit/improve "completed" features after feedback from the product owner (who was never there).

No Customer

I would say this was a pretty big one in a lot of peoples' eyes. Our product manager / customer advocate was omni-absent and never really understood the product very well.

All's Well That Ends Well?

Now, the funny thing is, at my new company, I work as a third party developer using my previous company's software. I've told everyone I used to work with the same thing: when you work with this stuff away from the daily development issues, it's really pretty impressive. Especially so when you know the timeframe and numerous problems that were faced during its development.

That's where too much success can be a dangerous thing. I've been at several companies that had the attitude, "We're making money, so we must doing something right." It's hard to quantify lost opportunity in the presence of existing success. However, I believe if we had addressed the issues listed above, everything could have been better: software quality, employee happiness, turnover rate…We'll never know.

While at lunch the other day, the subject of new year's resolutions came up briefly. I always have some grand idea of what I want to accomplish in the coming year, which ultimately falls well short of the reality. I think the problem I have is that I use a waterfall methodology "planning" my coming year. I need to agile it up a bit. So, this year I think I'll list some big items and then try and set up some weekly or monthly milestones along the way, reviewing and adjusting as I go. One of the items I know will make the list is to increase the frequency with which I update my blogs, hopefully without lowering my already treacherously low quality of content. We'll see how it goes.

I was reading a short post on Herding Cats about project management being about finding the truth. There are quite a few other good posts on there that follow the one I mentioned. It sounds like the author has a pretty sensible approach to most project management problems.

The reason the "search for truth" post struck me is that on nearly every project I've been on someone (most often either the product manager or upper management) has at some point been in heavy denial about the various risks to the delivery of the software. Anywhere from thinking that although we've been behind on every milestone, we'll somehow make it all up at the end (and keep 100% of the functionality). The "maybe the bad things won't happen" approach to risk management, which is just risk ignoring, has allowed too many projects to spiral out of control and ultimately caused way too much unnecessary "crunch mode" and developer unhappiness in my life.

I hate to always point the finger outside of development, but it's the way I think. My anecdotal experience is that the ball droppers are almost always in management. That's why competent managers that address problems in a realistic way evoke such a sense of loyalty from me.

I just read a nice little post called Making the Date. It's about correctly managing your software development situation to make a date. It's full of nice little analogies that non-IT people should be able to get, but probably won't. I find it funny that so many places have an "IT problem" in terms of getting good software on time and on budget, especially when so many developers are telling them the exact problem and solution. It also has a couple of items on what the manager should be doing in the process. It's a good little read.

I just finished reading a short article named Are Iterations Hazardous to Your Project? by Alistair Cockburn. In the article he points out a few issues about how Agile methodology vocabulary can be used to disguise Waterfall practices as well as some pretty good points arguing for longer iterations within Agile (a subject near and dear to my heart). I definitely "smell what he's cooking" and I like his suggestions. I particularly like his example of test driven development.

Part of the problem I've been experiencing first hand is being part of an organization that is trying to move from a classic Waterfall methodology to Agile. I don't think this is the kind of change that happens overnight. In fact, I think it is necessary to first transition to an iterative, mini-waterfall approach. This is an immediate improvement in my eyes. The danger seems to be that, because there are instant benefits, people get overly comfortable and begin to believe that they're truly doing Agile or XP just because they're using the "right" vocabulary to describe their practices. They lose site of the fact that they need to iteratively improve their process just as they do their software. Sure, they go through the motions, but ultimately they come to rest at a point well short of a goal the would bring much greater benefits. It'll be interesting to see if my place of work can break the inertia of complacency.