Running as Root

I've been listening to back episodes of the Agile Toolkit Podcast. One of the things that really caught my attention was the frequent mentioning of something called CodeGreen Labs. From elsewhere on the internet:

We are dedicated to creating a unique training experience that uses real projects with real deliverables. Labs work on projects that make a difference in the world, benefiting organizations that are working to improve the environment, human rights, social justice, health and economic development. By marrying the efforts of dedicated professionals learning Agile Development techniques with not for profit and research communities that need high quality software, we hope to be a small part of a positive change. Each training class will work on a specific software project for a specific cause.

It's volunteering to work on a real software project that is, hopefully, well run so that you can get real world experience. Sure, they get something out of it, but who cares about the other guy? I imagine they get mainly students. A few of the people in my work circles and I have been talking about something exactly like this. The head of my current company has mentioned to me numerous times that he's seeing disturbingly bad candidates coming to him fresh out of college CS programs. I can't comment as to the overall quality of CS programs around the country, but something like CodeGreen Labs seems like a brilliant idea on many levels. Unfortunately, I was a little disappointed at how unfriendly their home page was:

I switched jobs recently and although the current company used to do daily stand up status meetings ala Scrum/XP/Agile/Wagilefall, they had stopped at some point. As near as I can tell, they stopped because they couldn't keep the meetings short and on track. I expressed to my manager(s) that I thought it'd be great to start doing them again and lo and behold we had one on Monday.

Of course, the first issue that came up was the notion of the correct time. Inevitably when you try to start a meeting "on time" you get bogged down with the fact that no one thinks they're late because their clock says they're not. In the past I've used web based atomic clocks, but I'm now convinced that anyone planning to start meetings on time has to invest in a digital clock that synchronizes with the official time. I further suspect that anyone thinking that the people they work with are too mature to have this "argument" is probably wrong. We also haven't settled on the "punishment" for being late. I'm a big fan of cash fines though.

In the Future, Everything Will Be Glue

The entry title is from Weird Science, by the way:

Gary: I was crazy for this little eighth grade bitch.
Wyatt: Crazy, Insane!
Gary: I was dedicated to this girl, I called her every damn night!
Old Pimp Dude in the Bar: You called her every night? On the telephone?
Gary: On the telephone? What's he mean on the telephone, course it was on the telephone!

In my work life, the team I'm on has been working for quite some time fixing the architecture for an existing product. One of the things we've been doing while we've been fixing things is to add a REST API. You can save the REST vs WS-* debate for some other time and place.

Meet the New Product, Same as the Old Product

Unfortunately, there was a distinct lack of excitement about the new stuff we'd been doing from other areas of the company. Programmers seem to immediately "get" why exposing all of the CRUD functionality of your system through a web API is the sheer sweetness in terms of application integration. Sales and executive staff don't always see the possibilities. They saw the new version as architecture changes that improved performance and scalability while maintaining functional parity with the shipping version. That didn't scream sexy.

The programmers, ever fearing being a cost center with a very killable project, decided to start creating proof of concept integration examples that would create internal demand for the nearly finished version.

Pretty State Machine

As background, our system has an operational state that changes the way almost everything behaves. The change in this state is typically user driven. Someone using the software would decide to change the operational state due to an external event such as a Martian invasion. The software has a web interface but we had the idea that it'd be much cooler to change the operational state via a telephone.

We can change our operational state via the REST API by putting (as in HTTP PUT) an XML document representing the system status to the server. The system status has an element that represents the operational state. Now, our IT guy is always telling me the crazy stuff he can do with his home VOIP system. He runs a virtual machine with Asterisk and other assorted software an combination with an IP phone. I ran the idea of executing a shell script from a phone menu by him and he assured me it should be easy to do.

I was working under a deadline (of course) so I created three versions of the XML (one for each state I would be changing to) and got a copy of cURL installed in the Asterisk VM to PUT each document to our server, hopefully in response to someone pressing a button on a phone. After hitting a very serious dead end and wasting several hours trying to get an IVR working (that's what phone people call those voice menus) I finally found an example of an Asterisk iTunes controller. To use it you dial an extension, hear a beep, then press a button to do something. The script in Asterisk then calls a shell script in response to your button press. Five minutes after finding it I had a working example of changing operational state in our product via a telephone (actually a softphone but who's really keeping score at this point). Booyah! Sure it only took me 10 hours discover how to do an hour or so worth of work but then that's the plight of the knowledge worker.

Demo Day

The next day we invited additional people to our iteration demo meeting (really it's just the people that were supposed to be going all along but weren't) and showed the phone demo as our finale. Suddenly, people seemed as interested in our new version of the product as the developers are. There was a palpable sense of excitement. There were tears of joy. The developers were hoisted up onto the shoulders of sales and management and carried out into the streets (think about the scene in Dragon: The Bruce Lee Story).

I'm exaggerating a little bit perhaps. Of course there are all sorts of negatives preceding and following this event, but can't we just bask in the glory of this one little success for just a few moments? On the telephone.

I've already gone over my desire to add more tools and procedures to our distributed team development process. My next issue is with one of the tools that I like quite a bit. I think that the way we're using Wiki is leaving something to be desired.

Get Organized

The first problem I have is that it's horribly disorganized. It's extremely hard to find things. Design docs, developer environment setup, refactoring plans, third party tool evaluations, etc are all jumbled together. You can't navigate from the top level page to anything useful. That's not a huge deal for me since I'm more about searching than the dreaded hierarchy, but the search also leaves something to be desired.

Someone at my company suggested that the problem stems from the fact that it really takes someone with an overarching idea of (and a strong passion for) how to organize things. I don't think it needs to be that strong, but some simple high level guidelines of what goes where would help. It's great that anyone can add or edit a page, but in the absence of any guidelines our Wiki is looking more like a wall with 300 Post-It notes on it than anything really useful.

When Was This Updated?

The next problem is that Wiki pages usually don't get updated when things change. This leaves the Wiki full of a bad combination of mostly accurate and completely incorrect information. The Wiki page on the design of a module or its example code is too far away from the code.

That's why I like Javadocs so much. The documentation is right next to what you're documenting. If we think that not everything is best to document in the code, I'm thinking we may want to at least put a link to the Wiki documents in the Javadocs and make the evaluation of a Wiki update a part of the peer code review check-in process.

You're Not Revision Control

All of the design docs, tips, example code, support notes, and pretty much everything else related to the code or the product are accurate for a particular version of the system. If you're being good about updating things then the pages in the Wiki reflect the way things are more than the way things were.

For multiple versions of software, it's easy to forget how things used to work. Yes, there's some crappy versioning built into most Wikis but it's not really what they do best and it shows (just like their integrated search). Tying a Wiki page to a release of your software requires putting an artificial mechanism in place (like a naming convention, a structure to the hierarchy, or tagging if the Wiki supports it) or finding a Wiki that provides more powerful tools for finding topics by version (repository style tagging and viewing of a Wiki).

Integrate This *Flash*

Probably the easiest solution would be to choose a Wiki tool that supports top level categorization of your Wiki into sub-Wikis (like what Twiki does with webs). One downside would probably be that you would start off each new release cycle of the product by copying the previous category over to the new one, but maybe that's not that bad. You'd probably also want to at least do some of the high level guidelines and have the topic / category structure closely mimic your code where appropriate.

Another idea I was thinking of involves looking into either a file-based Wiki that I can directly put into the revision control system of my choice (Subversion, baby) or that directly integrates with RCS of my choice (did I mention Subversion?). At least for code related documentation (design docs, examples, dev environment setup, etc) this seems to be an intriguing solution. If the topic relates directly to a package / module then its path should directly mirror that. Otherwise it goes under one of several high level categories like I mentioned parenthetically. Reviewing the Wiki documentation would be part of a peer code review regardless of how the versioning issue is solved.

These release specific Wikis would still be published by the build to some internal website so that a super-Wiki could be created for support, QA, etc. I'd also think about maintaining the non-RCS Wiki for any information that doesn't relate to either the product or a specific version of the product (whatever they may be).

I'm still debating which approach I like best, but I am definitely certain that I think what we're doing now is inadequate.

In my last two jobs (including my current position) I have been in a cubicle. You can find plenty of arguments both for and against cube life on the web so I won't bother rehashing them. Suffice it to say that I think the benefits of giving programmers private offices far outweigh the perceived benefits of having a group space or the cost savings of sticking me in a 2 1/2 walled box. Case in point, I can barely work at the moment because two people have been having a conversation about fucking Weight Watchers for the last 20 minutes two cubes down from me. As fascinating as I find peoples' struggles with obesity, I'd really rather be working.

The Current State of Things

I'm in my third week of work at the new place. As with every place I've ever worked, they (we from here on out) have a lot of the same old problems. We have a shipping product, so we've got legacy code. Most legacy code I encounter anywhere is brittle, lacks unit tests, and has a severe lack of encapsulation / cohesion. The code hasn't rotted, peoples' view of the right way of doing things have evolved. When you look back at whatever you did 6 months to a year ago you're horrified. In addition the build, project organization, and installer are all screwed up. That too is typical. These things require more planning than they get on a normal project. They're the areas most likely to grow organically and as such they get messy and out of control very easily. Luckily with software everything if fixable.

The other process level problem we have is that we're dealing with a geographically separated development team. Although this goes against a major Agile rule, it seems to happen everywhere in the software world. We use IRC, revision control commit emails, daily status meetings, and announcement emails for major code changes to try and keep everyone in sync. Given that we're trying to fix all of the code level problems you can imagine that a lot of the system is changing very rapidly. The need for communication is even more heightened than normal.

Problems with IRC

At lunch on Friday with my co-workers we got into a discussion about the concerns about the effectiveness of our communication. A couple of people on the team (me among them) have made the statement that things aren't nearly as bad as other members seem to think. That is to say I have seen worse situations. We have a very good team of highly competent people that have an interest in improving things. The communication is certainly not the worst I have seen. That being said, I certainly think we can fix a few things.

I jumped into the lunch conversation saying that I thought both our email and IRC communication needed addressing. First, IRC is being misused. Someone said it is like hallway conversations for people that are off site. I like that analogy, but we're still missing slightly. We have too many inane, off topic conversations. I'm not interested in reading peoples' small talk. Second, we have too many one on one conversations taking place in the main channel. I think you can usually determine when you need to have a one on one conversation and take it to a private chat or phone call. I don't buy the argument that everyone needs to overhear everything in case they might have input. If these are hallway conversations, it's like every jackass has decided to have their meetings right outside my cubicle.

Another problem is that some people don't understand what is appropriate to communicate via IRC. Last week someone posted a link to a document they wanted the developers to review. Only one developer saw it. That shit needs to be sent in an email. And finally, any decision that gets made in IRC needs to be announced to a wider audience in a more visible way (most likely an email).

Problems with Email

One of the developers seemed to be getting a little irritated with me pointing out problems with our usage of IRC. Since I didn't pick up on it at the time I brought up why I thought our email communication sucked, too.

My problems with email are that it's a little to broadcasty, there's no archiving, no searching, it's non-threaded, and not good for soliciting responses. All of these problems have solutions within email, it's just that I feel that these solutions are tacked on after the fact and may still not be as effective as other solutions that were built with the problems in mind in the first place.

By broadcast, I mean both that I'm on email distribution lists that often have emails I don't particularly care about. Yes, we could make finer grained email distribution lists for a larger array of general subjects, but that gets messy especially since I can't currently control my membership on lists. This has to be done by someone with the correct permissions. Email rules to auto-delete are pretty imperfect as well.

As for archiving and searching, most people say I can archive my email locally and search on it whenever I feel like. This doesn't work unless I'm on my computer and doesn't work for any email sent before I started working here. Yes, you could tack on another product to automatically archive and index the email centrally and expose a web interface. This can even add some threading capabilities. This is one of the better solutions to a problem but I still find it lacking. Threading is typically done by parsing the subject line. Screw that up and you may not be in the right thread. Responding to old threads would require you paste the subject line from the web archive site and if you're really nice paste the body of one of the later posts. That would prevent people from having to run to the archive site to see what the hell you're talking about.

The problem of soliciting response from email is that you send out an email and end it with, "Is everyone good with that?" You almost never get a response. Maybe people don't respond if there's no problem. Maybe they didn't read it. Maybe they disagree but don't want to be the only one. Who knows? Sure you can put email receipt requests on your email, but the client on the other end has to support it and generally the receiver has to explicitly allow it. And so what if they read it? Do you want everyone to respond and say agree or disagree? That seems a little intrusive for most topics.

Bitcher!

At that point the slightly annoyed developer asked how I intended to solve these problems. I said I didn't think I had a solution for every one of these but that I thought there was a better way of doing things. For some of the problems I just don't know. She made the statement that she was tired of me bitching about the problem if I didn't have a solution.

I complain often, so I'm used to that response. People tend to feel that where there's smoke there some asshole standing next to a fire saying, "Someone should really do something about all this goddamn smoke." I feel that premature labelling of "bitcher" (since I've been there 3 whole weeks) is like me seeing dials that say something is going to blow up and being told, "Don't go bitching about the dials until you have a solution. Keep it to yourself." Of course, as always, the truth is likely somewhere in the middle.

My problem is that the first step to improving a situation is identifying the current shortcomings in your situation. Yes, something has to happen after that, but that doesn't invalidate the position of the person saying our status quo is inadequate. I'm sure as the other developer and I get more used to each other's quirks we'll be fine. I'm a non-complaining complainer while she appears to be very passionate and frustrated by institutional inertia that has prevented solutions from being implemented in the past.

Solutions

Since there seemed to be a real desire for me to put up or shut up, I detailed what I thought were some of the solutions. They're not perfect, but they were the best I could do at the time.

I like blogs as a developer diary. This is a place developers can go into more detail about their daily activities. The daily stand up should remain short and very high level. If you want more detail on what I am working on, check my internal blog. Blogs are also a good place to share quick tips on doing things (like how to use find, sed, tr, etc to perform a quick search and replace while copying the original file to another location). Blogs are also good for initial conversation starters and what-if scenarios. If an idea is worth discussing it moves off the blog and into threaded discussion forums.

Blogs aren't great for discussing things, but forums are pretty good at it. RSS feeds are a must so you get better notification of activity in the forums. You pay attention / subscribe to whatever you want. You can control the influx of information, unlike with email distribution lists. It's centralized, archived, searchable, and threaded. What's not to love? If you reach an actual decision move it to the wiki.

The wiki is not good for making announcements, having discussions, or keeping a developer diary. It is good for documenting things that have been decided. Yes, you can continue to tweak an idea, but major change discussions should happen in the forums with the results being captured in the wiki. I have my own problems with wikis that I'll detail in another post (they're very fixable). The wiki itself should have RSS feeds so you can easily see changes to the topics you care about.

If you need to solicit opinions, maybe you need to post a poll on the forums. I'm least happy with this solution, but I think it is a step in the right direction. Big decisions still should have a design review meeting I believe. Again, the designs should be captured somewhere, most likely the wiki.

Email is still used for quick announcements like "I modified the POM, so be sure to regenerate your Ecllipse projects" that don't really fit anywhere else. Email and shared calendar solutions are also great for scheduling meetings. Email is also used to communicate one on one changes like "Hey, I just checked in those changes you needed to complete your task for this iteration." I'm probably missing a lot of valid use cases for email so I'll just assume that email is the traditional catch all for things until we find / agree upon a better place.

There are other tools that I haven't mentioned but that are still needed. For example, you need a bug tracking system and an iteration planning tool.

My, Aren't We Tool Heavy?

I don't believe the tools are the complete solution. If your process is completely screwed up, the tools aren't going to fix it. I do think that the tools are a part of the solution though. For example, I don't like leaving the decision to document a change up to the developer making the change. They're too close to things. I like peer code reviews on check in with one of the items on the checklist being that an announcement needs to be made if the change is "big." A second set of eyes that didn't make the change helps greatly in making this decision. But, you still need a tool of some sort to communicate the change. Peoples' fear of having too many tools seems to make them force the one universally accepted tool (email) into roles it wasn't meant to fulfill. You wouldn't try to track bugs in email, would you? Maybe you shouldn't be using it for a lot of the other things for which you're currently using it…

As always, keep in mind that my ideas are a work in progress. Feel free to chime in if you have advice for better ways of doing things.

The Big Sticky Ball

It's job switching time again. I've had a good time watching another software project that I don't work on struggle to grow, fail, then struggle to survive and reinvent itself. The extremely quick summary is that a system admin with a lot of domain knowledge converted his convenience scripts into a product and a company. More developers were brought on to continue to develop on top of the initial attempt. Like many projects they were very heavy on implementation but very light on planning and process.

At some point, someone realized that they had severe problems with their overall architecture. A product that is really just glorified sys admin scripts didn't work well in large scale customer installations, was difficult to install, and a little buggy / unstable. Doesn't scale, hard to install, and buggy. Hey, just add that no one can tell you how much it costs and I think I just defined Enterprise software! In addition, since everything just kind of "grew" there weren't any real unit tests or, from what I hear, any real encapsulation into semi-independent submodules. Thus making fixing things a very dangerous proposition.

As I've mentioned previously, the idea of system to which you cannot easily add new functionality has been called a big sticky ball problem by one of my managers.

A Man, a Plan, But No Canal

The best plan moving forward seemed to be trying to stabilize the existing product and make it easier to build and install (yeah, their build sucked too). While that was being done, another product would be created to address the many issues of the first product. The first product would initially serve as a good starting point for requirements (both what it did and didn't do) as well as feed data to the new product allowing some measure of scalability through additional instances of the collector. Later, maybe the data collection would be improved in the new product and the first product could go away completely.

The problem is this means you won't be pumping out features like your customers, or worse yet your sales people, are used to. If your revenue model is heavily skewed to getting existing customers to upgrade (rather than a healthy effort to acquire new customers) then your sales are going to drop off since there's nothing to which to upgrade. If this happens, upper management may panic and decide that it's not worth taking two steps back so you can take many steps forward. They'd rather just see something, anything, that denotes forward progress and a return to sales. Why sales couldn't adjust their model and temporarily stop sucking on the upgrade revenue teat, I have no idea. I'm not a sales guy.

And, of course, this is what happened. A drop in sales brought executive attention. The new project was killed because in their eyes it too closely resembled another product being developed. The idea of stabilizing things in the original product took a bit of a backseat to pumping out some features. Then at some point, I imagine, the word "legacy" will be associated with that product and it'll slowly go away after all the developers are either transferred elsewhere or laid off.

And What About Me

My project had a ton of other problems. Briefly, it's a pluggable module into another company's framework. We would sell to a subset of the customers of the other company's product. More accurately, the other company's sales people would sell our product for us.

From the beginning the project was plagued with a lack of resources (test equipment primarily). Then, amid some project killing and reorganizing, the person who had the original vision for the partnership voluntarily left the company, the tech lead on the project left, and the development manager left (not quite in that order). The management to which the project was transitioned works primarily on a different project and seems to feel that my project runs itself. Sales and marketing aren't selling the product directly so they don't feel they need to do anything in the way of requirements.

In a final blow, it looks like the new version of the framework does a better job of what our product is currently doing. There is still room for our product to change its tact, but with no equipment, requirements, and now with one less developer (leaving them with one) I don't think it will happen.

And What Have We Learned

I don't know that I actually learned anything I didn't already know, but it's always nice to get confirmation on things.

From an architecture / code standpoint, you don't just tack on scalability. In order to do this you need to first concretely define your goal. "Must scale" is not an measurable quantity. Once you can measure it, early on make sure you can hit it and that everything you do moving forward won't jeopardize hitting it.

Unit tests don't just show you that your current implementation works. They show you that your refactored solution and all your new stuff didn't break anything that used to work. If you can't figure out how you could possibly test it, you better fucking rethink it.

If you ask someone to be the "build person" and they groan, your build sucks. Fix it and make it a one click build. There is a faster turnaround time to getting builds to QA (you do have QA, right?) and there are fewer human errors in the process (oh shit, I forgot to manually add the doc directory to the top level so this build is bad).

Domain experts aren't necessarily good developers, your developers shouldn't have to be domain experts. Domain experts may not be good customer advocates. There is a difference to knowing all the ins and outs of a domain and knowing what someone that works on it day to day (below the theory level) will want to accomplish. Your customers can tell you what they want to do but will have a limited idea on how they want to do it. For example, the customer wants to be able to mass delete, but they may not have a good idea of how to select what it is they want to delete.

Everyone needs to have skin in the game. If your own sales people don't sell your product, why would they ever contribute requirements (probably bad requirements, but requirements nonetheless)? If management feels their other project is more important, how can you expect them to devote the necessary time to get the developers the things they need? If developers never listen to how hard they've made things for the customer why would they ever change how they do things?

And finally, if a CEO ever tells you that if you don't like the way things are there are plenty of other places for you to get a job, you should probably listen. They're absolutely right.

More Tales of the Expert User

Another thing I noticed during the expert user presentation was how he used the information in the application. He leaned heavily toward using the charts and graphs in the application rather than the tables. From his viewpoint, the tables were useless. He didn't care about 99% of the individual incidents in the tables. He just wanted to see the overall trends. Multiple pieces of information were presented as different components of a single graph with a vertical line cursor that allowed him to compare the timing of different events from different mini-graphs. When he found something out of the ordinary he could click and drag to view a table version of a point in time. This is vastly superior to what I normally see–disconnected table and chart views with the tabular format being the kitchen sink view of things.

The point is that I normally witness user interface design happening in a complete vacuum. UI features are thrown in just like product features, just put everything in there because some of this is going to be useful to someone. Hopefully. Maybe. Now I haven't been here long enough to know how that UI was conceived, but this is certainly another example of the importance of mining your users for information. It's not only important to know what they need, but how they intend to use it.

Hey, Look at Me! I Read a Book

A case in point, I'm reading Things That Make Us Smart and the author uses several really good examples of the representation of information affecting the ease with which certain tasks can be performed. One example from the book is the case of Roman numerals, ticks, and Arabic numerals.

Supposedly it is easier to perform addition with Roman numerals than the other two systems. Read the Wikipedia explanation to see it in action. This is apparently easier to learn. Fortunately I'm too heavily indoctrinated into our current system to tell if it really is. A good thing since multiplication and division would obviously make my wiener hurt.

Ticks are superior for counting or tallying things. This is because their representation of larger numbers is additive. You just add another tick. With Arabic numerals you have to completely change the symbol you're using. This means erase and redraw. Not very efficient for the use case of counting.

And Arabic pretty much kicks the ass of the other two in everything else. But the point is that each of these representations are superior to the others depending on what you are using them for (although I will never use Roman numerals even if I'm only doing addition).

Give Them What They Didn't Even Know They Wanted

Apparently the king of information and interface design is Edward Tufte. I've only seen a few examples of his stuff, but it is very sweet. On top of this there are courses available to learn to improve your presentation of information. Everyone I've talked to that has attended raves about their sheer brilliance. I've decided I'm attending the very next chance I get. It's a step toward the final piece of the puzzle which is knowing how to not just get the information the user wants but to know how to present it to them in a manner that greatly facilitates their understanding and usage of it.

Now that's something that should impress customers and make you stand out from the competition…

I finally had something useful happen at work. We had a consultant that uses one of our products during the course of his job come in and give a quick presentation on how he uses the tool. Unfortunately, it's not the product I work on, but it was still interesting for a number of other reasons. As quick background, the tool demo'ed is a storage monitoring tool.

Three Users of Enterprise Software

Nothing new here, but Enterprise software users typically fall into three categories: experts in a domain using the software to make their tasks easier, users largely ignorant of the domain that want an expert in a box to tell them what to do, and executives that want some charts to determine if things are running smoothly: the infamous "dashboard."

The person we had in was definitely an expert in the domain. Listening to how he uses the software to determine if a customer's installation is running smoothly was very educational. Often, features seem to get added to a product by people that don't really understand how those features are going to be used. It's put in just in case someone might find it useful or, worse yet, a customer demanded it without explaining why the wanted it.

This part interested me because, through his demo, he identified a ton of information that the "expert in a box" customer would drool over. Apparently the way he used different pieces of information wasn't just news to me, it was news to the current development team as well. They furiously scribbled while he showed how he used combinations of charts to find configuration errors, determine the need for new equipment, and a number of other things. He also suggested some features that would make his life easier and explained why he didn't use certain parts of the system at all.

Hired Guns

I, like most people, really value the input of a customer or, if that's not available, a customer advocate. The problem is that most product owners in a typical company don't seem to really understand what a customer needs. They pay more attention to what the competition is doing or at most what the squeakier wheels in their customer base say they want.

As early as possible you should post a position on one of the online job sites looking for an expert in whatever problem domain you're working. Throw them a few thousand dollars or so to just come in and show you what it is they do. Do all the standard information mining stuff on the poor bastard. Make sure the developers are involved. Try and build a tool that is useful to him and at the same time captures some of his knowledge for the EIAB (expert in a box) user. After you build a few iterations of the product, ask another one in to evaluate the product. Repeat, get the EIAB user to check it out, etc etc. Did I mention that you should make sure the developers are involved?

Ship your product, sell your product, host a message board for customers of your product, make the developers participate on the board. Keep that interaction with the customer going. At some point, throw a few extra grand at an expert that uses your product and record a demo session to include in your product. Something to orient the expert customer and maybe teach the EIAB user to be more effective. It's not like anyone reads the manual. That training video can double as sales literature, too.

No Shit

Yeah, it's pretty obvious. A veritable no-brainer. I'm ashamed for even suggesting such an obvious idea. Now why, if it's so obvious, is it that I've never seen this done at any organization at which I've worked?

We Can Rebuild It

The dream of most of the software developers I know is to get the opportunity to rewrite a system they work on from scratch. I think as things move more toward test driven development (TDD) this desire may fade a bit, but never entirely disappear. TDD should allow developers to aggressively refactor an existing product rather than starting from scratch. There will be less fear of hopelessly breaking the system if you've got solid tests and good test coverage. Of course, even getting the opportunity to refactor may be rare in many cases. The product owners don't always care how dirty the implementation is if there are no visible effects in the final product (at least not right now anyway). The customer won't buy the product because it has pretty source code behind it. A damn shame.

The incessant desire of software developers to rewrite all code they encounter is perfectly natural in my opinion. It's the plight of the knowledge worker. Each day you collect new information and techniques. When you look at someone else's code (maybe even your own from 6 months ago) you can't help but feel that there is a better way of doing it. If you have to write new code within the constraints laid down by this less than ideal solution, you're eventually going to come to the conclusion that you should just rewrite the whole thing. You just don't like having to work with / around whatever it is you've been given. Maintaining someone else's software is like putting someone else's sweaty clothes on. That opportunity to refactor is crucial.

From Hacks to Good Software

I was talking with another of the developers on my team a few days ago about how we should fix a problem we're having. We're working under a deadline and may not have time to implement the more ideal solution. I proposed a couple of different stopgap solutions that would get us to our release date. Most of the ideas were met with the infamous developer credo, "but that's a hack." I definitely understand the sentiment, but unfortunately we have to ship some software. And in my experience, most good solutions started as a hack. If you properly encapsulate everything you can go back and improve that hack into a decent solution with minimal impact. But again, willingness to accept the less than ideal solution is much easier if you know you will be able / allowed to refactor it later.

Now, does the world work this way? Hell no. So far, for me anyway, TDD is hard. I seem to always find myself smack dab in the middle of a container or a framework that makes effective testing extremely difficult to do. That inevitably gets coupled with a product development cycle that allows no time to retro-fit tests and test harnesses into the existing product. On top of that, the product managers often don't see the value of refactoring unless they get a ton of new features with it.

Usually in a simple situation you just start doing TDD for all features moving forward and write a test for every bug you find. The idea is that eventually you'll get there. For cases where you can't do the initial work in small chunks I guess you're just screwed. Eventually you'll have to bite the bullet and do whatever big tasks are necessary to start you down the whole TDD road. If you can't do that, it's probably time to move on and try to make sure that the next place doesn't suffer from the same problems.

Any Questions?

In the back of my mind I keep filing away little tidbits like this for later use so that I can come up with a decent set of questions to ask any potential employers. In a typical interview I forget the importance of grilling the employer. I need a set of questions like the Joel Test. I think the first one I need to add is: Can I see a copy of your code coverage reports, preferably over the last 3 months?

Update

Matt Kinman rocks!