The Big Sticky Ball
It's job switching time again. I've had a good time watching another software project that I don't work on struggle to grow, fail, then struggle to survive and reinvent itself. The extremely quick summary is that a system admin with a lot of domain knowledge converted his convenience scripts into a product and a company. More developers were brought on to continue to develop on top of the initial attempt. Like many projects they were very heavy on implementation but very light on planning and process.
At some point, someone realized that they had severe problems with their overall architecture. A product that is really just glorified sys admin scripts didn't work well in large scale customer installations, was difficult to install, and a little buggy / unstable. Doesn't scale, hard to install, and buggy. Hey, just add that no one can tell you how much it costs and I think I just defined Enterprise software! In addition, since everything just kind of "grew" there weren't any real unit tests or, from what I hear, any real encapsulation into semi-independent submodules. Thus making fixing things a very dangerous proposition.
As I've mentioned previously, the idea of system to which you cannot easily add new functionality has been called a big sticky ball problem by one of my managers.
A Man, a Plan, But No Canal
The best plan moving forward seemed to be trying to stabilize the existing product and make it easier to build and install (yeah, their build sucked too). While that was being done, another product would be created to address the many issues of the first product. The first product would initially serve as a good starting point for requirements (both what it did and didn't do) as well as feed data to the new product allowing some measure of scalability through additional instances of the collector. Later, maybe the data collection would be improved in the new product and the first product could go away completely.
The problem is this means you won't be pumping out features like your customers, or worse yet your sales people, are used to. If your revenue model is heavily skewed to getting existing customers to upgrade (rather than a healthy effort to acquire new customers) then your sales are going to drop off since there's nothing to which to upgrade. If this happens, upper management may panic and decide that it's not worth taking two steps back so you can take many steps forward. They'd rather just see something, anything, that denotes forward progress and a return to sales. Why sales couldn't adjust their model and temporarily stop sucking on the upgrade revenue teat, I have no idea. I'm not a sales guy.
And, of course, this is what happened. A drop in sales brought executive attention. The new project was killed because in their eyes it too closely resembled another product being developed. The idea of stabilizing things in the original product took a bit of a backseat to pumping out some features. Then at some point, I imagine, the word "legacy" will be associated with that product and it'll slowly go away after all the developers are either transferred elsewhere or laid off.
And What About Me
My project had a ton of other problems. Briefly, it's a pluggable module into another company's framework. We would sell to a subset of the customers of the other company's product. More accurately, the other company's sales people would sell our product for us.
From the beginning the project was plagued with a lack of resources (test equipment primarily). Then, amid some project killing and reorganizing, the person who had the original vision for the partnership voluntarily left the company, the tech lead on the project left, and the development manager left (not quite in that order). The management to which the project was transitioned works primarily on a different project and seems to feel that my project runs itself. Sales and marketing aren't selling the product directly so they don't feel they need to do anything in the way of requirements.
In a final blow, it looks like the new version of the framework does a better job of what our product is currently doing. There is still room for our product to change its tact, but with no equipment, requirements, and now with one less developer (leaving them with one) I don't think it will happen.
And What Have We Learned
I don't know that I actually learned anything I didn't already know, but it's always nice to get confirmation on things.
From an architecture / code standpoint, you don't just tack on scalability. In order to do this you need to first concretely define your goal. "Must scale" is not an measurable quantity. Once you can measure it, early on make sure you can hit it and that everything you do moving forward won't jeopardize hitting it.
Unit tests don't just show you that your current implementation works. They show you that your refactored solution and all your new stuff didn't break anything that used to work. If you can't figure out how you could possibly test it, you better fucking rethink it.
If you ask someone to be the "build person" and they groan, your build sucks. Fix it and make it a one click build. There is a faster turnaround time to getting builds to QA (you do have QA, right?) and there are fewer human errors in the process (oh shit, I forgot to manually add the doc directory to the top level so this build is bad).
Domain experts aren't necessarily good developers, your developers shouldn't have to be domain experts. Domain experts may not be good customer advocates. There is a difference to knowing all the ins and outs of a domain and knowing what someone that works on it day to day (below the theory level) will want to accomplish. Your customers can tell you what they want to do but will have a limited idea on how they want to do it. For example, the customer wants to be able to mass delete, but they may not have a good idea of how to select what it is they want to delete.
Everyone needs to have skin in the game. If your own sales people don't sell your product, why would they ever contribute requirements (probably bad requirements, but requirements nonetheless)? If management feels their other project is more important, how can you expect them to devote the necessary time to get the developers the things they need? If developers never listen to how hard they've made things for the customer why would they ever change how they do things?
And finally, if a CEO ever tells you that if you don't like the way things are there are plenty of other places for you to get a job, you should probably listen. They're absolutely right.
