Running as Root

Hot Flickr Action

Man, I'm liking Flickr more and more lately. I'm trying my best to get everyone I know addicted to it, too (I've started giving Flickr accounts as gifts). What's not to like? A simple method of uploading (try teaching your parents ftp), easy tagging / browsing / searching, quickly make links of resized pics, pretty much unlimited space and bandwidth, a nice API, and lots of mind share. All of this for (currently) $24.95 a year. In addition, they've partnered with other sites to allow you to do cool things like print stamps with your photos on them.

The thing that has struck me most lately is how easy Flickr has made it for people to be more creative. So far this year, I've been contacted by two people writing books and an online city guide all asking to use photos. That's just plain neat. If they didn't have a centralized repository of tons of photos (with clearly marked Creative Commons licenses and good tagging for retrieval) how much more of a pain in the ass would it be for them to get the photos they wanted? Sure, we're not quite at the dream of a full remix culture since there are still legal obstacles to using other peoples' works, but an equally large barrier of simply finding stuff is rapidly disappearing.

The Old Ways

My wife's aunt gets all her digital photos printed at Wal-Mart and then has them burn a CD (despite the fact that she has a CD burner). Then when people are over (if the subject comes up), she'll show off pictures of her last vacation, grandson, etc. I can think of no worse way for me to consume content. It's all info-silo, streaming, push model bullshit. When someone asks about the stuff, I'm along for the ride whether I like it or not. If I happen to like a picture, I'm not likely to ever see it again since it's locked up in her CD. Even for her own use, she can't find that one picture on the ship with the captain that time.

The solution to her problem is simple, of course. Wait for me to give you a Flickr account, then learn to use it / get me to teach you how to use it / tell Wal-Mart they should offer Flickr upload along with photo printing, tag those photos (unlike these people), then hand out cards or post-its or whatever with your Flickr ID on it whenever someone asks about the photos.

Break the Silo

My wife's grandmother has album upon album of photos dating back 1930s. In addition, I'd say she easily has several thousand photos. Her husband was a photography hobbyist. He took photos all over the world when he was in the military during dubba dubba two. They're all sitting in photo albums, slowly fading away. How sad. Again, get a scanner and scan them (I'm positive some of those offspring would help), upload them, don't forget to tag them, and then tell all the children and grandchildren that they won't need to fight over the photo albums when you die because they're all online. That or you could keep watching re-runs of 1970s game shows on cable.

The same goes quadruple for my parents, my wife's parents, every aunt and uncle, etc etc. It's never been easier to get that information (we're currently talking about just photos) out there for everyone to see. These are exciting times. Why aren't you living in them?

The Big Sticky Ball

It's job switching time again. I've had a good time watching another software project that I don't work on struggle to grow, fail, then struggle to survive and reinvent itself. The extremely quick summary is that a system admin with a lot of domain knowledge converted his convenience scripts into a product and a company. More developers were brought on to continue to develop on top of the initial attempt. Like many projects they were very heavy on implementation but very light on planning and process.

At some point, someone realized that they had severe problems with their overall architecture. A product that is really just glorified sys admin scripts didn't work well in large scale customer installations, was difficult to install, and a little buggy / unstable. Doesn't scale, hard to install, and buggy. Hey, just add that no one can tell you how much it costs and I think I just defined Enterprise software! In addition, since everything just kind of "grew" there weren't any real unit tests or, from what I hear, any real encapsulation into semi-independent submodules. Thus making fixing things a very dangerous proposition.

As I've mentioned previously, the idea of system to which you cannot easily add new functionality has been called a big sticky ball problem by one of my managers.

A Man, a Plan, But No Canal

The best plan moving forward seemed to be trying to stabilize the existing product and make it easier to build and install (yeah, their build sucked too). While that was being done, another product would be created to address the many issues of the first product. The first product would initially serve as a good starting point for requirements (both what it did and didn't do) as well as feed data to the new product allowing some measure of scalability through additional instances of the collector. Later, maybe the data collection would be improved in the new product and the first product could go away completely.

The problem is this means you won't be pumping out features like your customers, or worse yet your sales people, are used to. If your revenue model is heavily skewed to getting existing customers to upgrade (rather than a healthy effort to acquire new customers) then your sales are going to drop off since there's nothing to which to upgrade. If this happens, upper management may panic and decide that it's not worth taking two steps back so you can take many steps forward. They'd rather just see something, anything, that denotes forward progress and a return to sales. Why sales couldn't adjust their model and temporarily stop sucking on the upgrade revenue teat, I have no idea. I'm not a sales guy.

And, of course, this is what happened. A drop in sales brought executive attention. The new project was killed because in their eyes it too closely resembled another product being developed. The idea of stabilizing things in the original product took a bit of a backseat to pumping out some features. Then at some point, I imagine, the word "legacy" will be associated with that product and it'll slowly go away after all the developers are either transferred elsewhere or laid off.

And What About Me

My project had a ton of other problems. Briefly, it's a pluggable module into another company's framework. We would sell to a subset of the customers of the other company's product. More accurately, the other company's sales people would sell our product for us.

From the beginning the project was plagued with a lack of resources (test equipment primarily). Then, amid some project killing and reorganizing, the person who had the original vision for the partnership voluntarily left the company, the tech lead on the project left, and the development manager left (not quite in that order). The management to which the project was transitioned works primarily on a different project and seems to feel that my project runs itself. Sales and marketing aren't selling the product directly so they don't feel they need to do anything in the way of requirements.

In a final blow, it looks like the new version of the framework does a better job of what our product is currently doing. There is still room for our product to change its tact, but with no equipment, requirements, and now with one less developer (leaving them with one) I don't think it will happen.

And What Have We Learned

I don't know that I actually learned anything I didn't already know, but it's always nice to get confirmation on things.

From an architecture / code standpoint, you don't just tack on scalability. In order to do this you need to first concretely define your goal. "Must scale" is not an measurable quantity. Once you can measure it, early on make sure you can hit it and that everything you do moving forward won't jeopardize hitting it.

Unit tests don't just show you that your current implementation works. They show you that your refactored solution and all your new stuff didn't break anything that used to work. If you can't figure out how you could possibly test it, you better fucking rethink it.

If you ask someone to be the "build person" and they groan, your build sucks. Fix it and make it a one click build. There is a faster turnaround time to getting builds to QA (you do have QA, right?) and there are fewer human errors in the process (oh shit, I forgot to manually add the doc directory to the top level so this build is bad).

Domain experts aren't necessarily good developers, your developers shouldn't have to be domain experts. Domain experts may not be good customer advocates. There is a difference to knowing all the ins and outs of a domain and knowing what someone that works on it day to day (below the theory level) will want to accomplish. Your customers can tell you what they want to do but will have a limited idea on how they want to do it. For example, the customer wants to be able to mass delete, but they may not have a good idea of how to select what it is they want to delete.

Everyone needs to have skin in the game. If your own sales people don't sell your product, why would they ever contribute requirements (probably bad requirements, but requirements nonetheless)? If management feels their other project is more important, how can you expect them to devote the necessary time to get the developers the things they need? If developers never listen to how hard they've made things for the customer why would they ever change how they do things?

And finally, if a CEO ever tells you that if you don't like the way things are there are plenty of other places for you to get a job, you should probably listen. They're absolutely right.

I finally had something useful happen at work. We had a consultant that uses one of our products during the course of his job come in and give a quick presentation on how he uses the tool. Unfortunately, it's not the product I work on, but it was still interesting for a number of other reasons. As quick background, the tool demo'ed is a storage monitoring tool.

Three Users of Enterprise Software

Nothing new here, but Enterprise software users typically fall into three categories: experts in a domain using the software to make their tasks easier, users largely ignorant of the domain that want an expert in a box to tell them what to do, and executives that want some charts to determine if things are running smoothly: the infamous "dashboard."

The person we had in was definitely an expert in the domain. Listening to how he uses the software to determine if a customer's installation is running smoothly was very educational. Often, features seem to get added to a product by people that don't really understand how those features are going to be used. It's put in just in case someone might find it useful or, worse yet, a customer demanded it without explaining why the wanted it.

This part interested me because, through his demo, he identified a ton of information that the "expert in a box" customer would drool over. Apparently the way he used different pieces of information wasn't just news to me, it was news to the current development team as well. They furiously scribbled while he showed how he used combinations of charts to find configuration errors, determine the need for new equipment, and a number of other things. He also suggested some features that would make his life easier and explained why he didn't use certain parts of the system at all.

Hired Guns

I, like most people, really value the input of a customer or, if that's not available, a customer advocate. The problem is that most product owners in a typical company don't seem to really understand what a customer needs. They pay more attention to what the competition is doing or at most what the squeakier wheels in their customer base say they want.

As early as possible you should post a position on one of the online job sites looking for an expert in whatever problem domain you're working. Throw them a few thousand dollars or so to just come in and show you what it is they do. Do all the standard information mining stuff on the poor bastard. Make sure the developers are involved. Try and build a tool that is useful to him and at the same time captures some of his knowledge for the EIAB (expert in a box) user. After you build a few iterations of the product, ask another one in to evaluate the product. Repeat, get the EIAB user to check it out, etc etc. Did I mention that you should make sure the developers are involved?

Ship your product, sell your product, host a message board for customers of your product, make the developers participate on the board. Keep that interaction with the customer going. At some point, throw a few extra grand at an expert that uses your product and record a demo session to include in your product. Something to orient the expert customer and maybe teach the EIAB user to be more effective. It's not like anyone reads the manual. That training video can double as sales literature, too.

No Shit

Yeah, it's pretty obvious. A veritable no-brainer. I'm ashamed for even suggesting such an obvious idea. Now why, if it's so obvious, is it that I've never seen this done at any organization at which I've worked?

At work, I have a separate test machine I use to deploy the software on which I'm working. The demands the container(s) and framework make on the host machine are borderline too high to run on my laptop comfortably so the test machine is a very good solution. As I mentioned before we use a lot of virtual machines here. My test server is running Debian but the environment I'm deploying to is actually a VM running Windows via an instance of VMware Player. I run the player by ssh'ing to the test machine and then do some X11 port forwarding to my local Cygwin X server just so I can futz around with the instance of Windows if need be…

I'm planning on working from home this afternoon and some of this weekend. In order to test my changes I need somewhere to deploy the application. Unfortunately, I haven't set up our VPN client yet (plus I hear our VPN is dog ass slow). Well, thanks to those handy-dandy VMs, I can pick up a copy of my server and take it with me. Then I can either run it on my laptop (and temporarily suffer through the somewhat acceptable performance degradation) or I can drop it on any of the machines I have at home. Oh, daddy likes.

I was talking with a co-worker today about the type of software we both work on: enterprise software. There are about as many definitions of what "enterprise software" means as there are people working on it. We weren't really concerned with that. We got onto the subject of product management, sales, marketing, etc.

I've said before that, although you could buy software I've worked on, no one could tell you exactly how much it costs. That has always irritated me. I realize it's either the way things have to be or the way things have always been, but there's just something sleazy feeling about no one being able to tell me a price. It feels a bit too much like trying to buy a car with the salesperson trying to "work with" me to "arrive at" some price. That's when the bullshit alarm starts going off and I realize that I'm about to get screwed.

It sure would be nice to not only know how much my software costs but to realize that, much like Saturn's supposed no hassle / no haggle selling, you would know that at least everyone is getting screwed equally.

It's (Third) Party Time

A lot of computer products, particularly in the monitoring / administration domain, are centered around some third party piece of hardware of software. Usually a company finds that they can greatly enhance and simplify the admin and reporting on another company's flagship product. Presumably the original company is either unable or unwilling to make it easy manage their stuff because they're concentrating on their core problem domain.

As an example, if you build backup servers, you're working on getting that backup stuff working smoothly, not necessarily on making it easy to manage or get information out of it. Sure, you include some out of the box support for doing some of the stuff, but ultimately it's an immature solution that feels like it was tacked on at the last second. It's particularly bad with hardware vendors. They usually can't design decent user software or APIs.

This creates a great opportunity for software companies to swoop in and carve out a spot in (or create a market for) the area of third party management software. If you can write a piece of software that makes the administration of Foo servers more simple, particularly in a large enterprise environment where the out of the box tools break down, then you'll be sitting pretty for a while.

You Mean, We Can Make Money Off This?

One of the problems occurs when the original vendor suddenly realizes that their software sucks. If they ever get off their ass and write some decent management stuff around their product and bundle it for free, then customers have much less incentive to go looking for your offerings in the first place. If other companies begin offering management solutions around the same product, you will see even more of a decline in sales.

The temptation seems to be to get into a feature war with everyone else. Once you've exhausted all of the useful features you start to include numerous niche features and one-off customer requests. Dump these onto a product sheet in no particular order and you look just as good, if not better than all of your competitors. A lot of these features aren't likely to matter very much to the people that use the software, so you need to get some marketing materials, product advocates, consultants, etc to push your product to the people that make the decisions (and don't actually use the product). For more on this you can see the ClearCase situation in another post.

Now you've lost your way. You started out simplifying the administration of another company's product and have created what is, more than likely, an overly complicated and hard to use product of your own.

Wider Is Better

Rather than freak out and get all feature crazy when the competition starts, it's better to diversify what it is your product simplifies. For example, if you've a database tool for a particular backend, why not add support for another database backend before exhausting your laundry list of niche features? Of course this works better if you're simplifying or abstracting multiple offerings from vendors in the same problem domain that are likely to be found uneasily co-existing in large organizations.

By that I mean, you want to do something like take three operating systems and present a common management interface for a problem domain (like security or backup) that sensibly abstracts away the subtle differences. This allows an admin to more easily administer the hodge-podge of systems he's found himself in charge of. And, since the same person is likely to have to deal with all three, your tool is quite handy. Handier in fact than if three people each only deal with one of the situations. Then the fact that your tool acts as an abstraction for all three isn't particularly useful, especially since each of them will have their own, very deep on functionality, tool that they prefer. No, I think the better target is to go wide and relatively shallow. It's also better insulates you from the competition, either from the original vendors (who usually don't care about working with products from other vendors) or other third party companies (who may not have their act fully together).

And Your Point Is?

What's my point? Up to now, what I'm talking about is fairly obvious to most people. The issue comes in when you consider how your original, one vendor product came to be. I've seen a lot of software teams and companies paint themselves into corners by making the assumption that their product only works with one vendor product or by tightly coupling domain / business logic to the first product they support. When they finally get around to adding support to more products they often find out their "abstraction" doesn't play well with others. By then it's too late. It's all well and good to only build exactly what you need at the time, but you need to think about where you're going. This is something I haven't fully figured out with agile development. This "architectural escape velocity" doesn't seem to come easily in the bite size chunks that agile advocates. I'm not saying it can't happen, it's just very easy to get it very wrong.

As early as you can, hopefully when you initially start your project, you might want to start asking these questions:

• What other vendors' products could we manage?
• How have the vendors represented this domain differently?
• How can we create a unified view (abstraction) of this domain?
• If something doesn't fit the abstraction, how can it still fit into our product?
• How do we keep one-off customer requests from making our core product less maintainable?
• How can customers get information into and out of our system?

That's just a start. I'm sure there are more (and probably better) questions to keep asking yourself. Answer these early and often (like you release) and let it affect your planning. Don't be tempted to just say, "we'll worry about that later" because by then it'll cost too much to change.

Humble Beginnings

Around ten years ago I was working for a pretty backwards IT department. We wrote and maintained an internally deployed call center application in Foxpro. We didn't use any source control. The shared libraries for the multiple, customer specific versions of the application were on a shared network drive. When you ran the build script, a new version of the application was instantly put on another shared drive and almost as quickly wound up on a phone agent's desktop.

Over a weekend, one of the lead developers decided that we should have some sort of source control. Rather than use any of the many mature (and free) versions of source control, he decided to write his own–in Foxpro of course. His version of source control simply made everything on the shared library drive read only. You then ran his program (named "vc") on some component and it would log that you had it checked out. When you checked it in, it made the shared version writable, copied your version over it, and logged that you checked it in. That's it. There was no tracking of revisions, no versioning, not much of anything really.

When the department finally decided that this solution wasn't good enough (a couple of years later) several people recommended using CVS. However, now that they were intent on improving their process, the management felt that CVS just wasn't good enough for our unique needs. Someone from outside the company sold them on using ClearCase at several thousand dollars per developer. Of course, we didn't use or need any of the supposed features of the product. In my eyes, ClearCase was/is no better than CVS. From the management point of view it was better because we theoretically got support, it better fit our "complicated" development process, and someone outside the company told them it was better.

New Company, Same Issues

A couple of jobs and many years later I found myself in a much bigger, more mature IT department. The new problem wasn't source control, it was the process. We were using a painful waterfall development process. The management over the IT organization decided we should switch to a more agile process.

Most of the developers I worked with viewed this as a very positive change. Up to this point, all project management occurred in Microsoft Project and Word files. Now that we were changing things up, we could probably use a different set of planning tools. One of my co-workers (Matt Ray) had used XPlanner at a previous job and felt it was "good enough." Somewhere in the decision making process it was decided we'd have Rally come in and teach us how to be more agile. We were also going to use their project management tool to organize our tasks. Being well outside the decision making process on this, I'm not exactly sure how much money all of this cost us, but I suspect it was quite a bit. Was Rally's mentoring valuable? I think that it was, but I think that its primary value was that it is always easier to listen to someone from outside of your own company rather than your own people. Is Rally's tool better than XPlanner? I think it is a better overall tool, but it doesn't really do anything significantly better. XPlanner really is good enough for what we needed in my opinion.

The Tools Aren't the Thing

I'm now at another job in a relatively immature IT department. We've started using an agile development process and are using XPlanner to plan our stories and tasks for each iteration. A couple of days ago, during iteration planning, one of the developers was wondering if Rally's product had a feature they felt was missing from XPlanner. In this particular case, it didn't, but it made me think about peoples' love of tools.

It never seems to fail–an IT organization decides that they're doing something wrong and the solution is to pay someone from outside the company to tell them what expensive product will fix it. Usually these outsiders tell you what the more competent people in your organization already know. However, it gives the upper management great peace of mind to have someone that makes a living telling people how to do things give them the answer to their problems. I can almost understand this but it still baffles me that, if you believe you've hired good people, you aren't listening to them. The much stranger thing is that so many people are willing to go from using a bad solution to shelling out big money for a tool with a set of features and support options they'll never use. They're convinced that one of the major problems with their process isn't the process itself–it's the tools. The same thing happens with people on a personal level. I could get buff if I had a gym membership and I could get organized if I had a PDA. This is the easy "fix." Just buy a silver bullet to make the problem go away. People and organizations need to learn that they need to address the root of the problem first, usually with free tools that are "good enough" (if not outright better than their commercial counterparts). Once you do that, you're free to buy some tools to eek out a few extra points of efficiency if you still feel the need. But, most of what you need to do is purely behavioral and largely free to start solving–at least in terms of money.

Update

I just saw another tool named ScrumWorks that apparently recently became free to registered users. If XPlanner doesn't do it for you, you might give this tool a whirl. I personally haven't looked at it very much.

I was just reading a post on gapingvoid about how a corporate blog is just as much (if not more) about changing the inside of your organization as it is about communicating with your customers. His line about making the membrane between the market and your company more porous and thus make it "easier it is for the internal conversation to inform and align with the external conversation" seems particularly insightful to me. As does the observation that "poking holes in membranes subverts hierarchies." A good thing in my eyes.

In software, no one sets out to write bad code that they're embarrassed of, but all too often I think good people make bad products. This happens for a variety of reasons. Developers and requirements "gatherers" may be too insulated from what is really important to the customer. For the developer, not having to figuratively look the customer in the eye makes it easier to throw bad code over the wall. For requirements, they tend to get deluded that the people they're trying to close the deal with right now are representative of what all of your customers want or that their pet feature is more in demand than it really is.

This problem can be alleviated somewhat through site visits or more support roles for developers, among other things. But, it seems to me, the best way to regularly get your staff talking to your current and potential customers (and to get the customers to talk back) is to add some transparency to things. I truly believe that if your employees know that they're going to be out there justifying their current and future thinking to your customers (which they're probably not having to do right now–not directly at least) they'll try harder to produce something they're proud of. But, more importantly, before it's built, people outside of your company will have a more direct way of telling you you're doing something wrong. And I think blogs are the best way to do that.

I think most of the people that would be reading this already use an aggregator of some sort, but I'll put up a quick mention of it anyway (who knows, you may find the tutorial a useful place to which to point people). Also, some of the repeated web hits in my StatCounter reports lead me to believe that some people still do manually check sites. A travesty, especially considering how infrequently many of them update (including this one usually).

If you find yourself manually checking multiple blogs, news sites, etc throughout the day, you're living like a second class citizen. There are fancy tools that will do it for you. Please, for the love of god, read this tutorial if you are one of those people. It will change the way you use the web and cause you to resent and possibly begin ignoring sites that don't have feeds. Bloglines is not the only aggregator out there, by the way, but it's my favorite and probably the easiest to get started with.

I was checking my meager blog stats over on StatCounter (which I recommend) trying to figure out how the 2 people that read my site today arrived there. As a quick aside, if you ever want to get more people coming to your blog you should write a lot about cell phones (like this guy). Just over half of my search engine hits are the result of a quick post on making a Ms. Pacman ringtone. The breakdown by search engine wasn't what I expected either: 66% Yahoo!, 23% MSN, 9% Google.

Anyway, I'm off subject. I eventually wound up at my Google AdSense account wondering about my inability to get good clickthrough. I'm sure pathetic traffic is a part of that. Well, I went to my site and noticed that the ads didn't show up. I went through a few minutes of "debugging" before realizing that my AdBlock extension (with FilterSet.G thanks to Matt Ray) was doing its job very well. Long story short, my ads show up–my clickthrough and traffic just suck.

That brings me to my final stupid point: in the land of integration, the debugging man is king. With SOA (sorry to re-use the over-used, ill-understood acronym, but at least I didn't say Web 2.0), various plug-in architectures, and numerous open source data/communication standards out there it is clear that integration is in your face to the extreme. With that being the case, all of this unforeseen integration is going to lead to bugs (as all things do). The problem will be with 50 unrelated parties molding the final data at 50 different integration points you had better have an architecture made for debugging, a set of tools that give you full control of the chain of events leading to the final data and allow your support staff to do it remotely, and a person that doesn't have their head up their ass. So, if the little children were to ask me today what skill I thought would be most valuable in the economy of the future (as they so often do), I would say, "It's hacking and debugging." They of course would point out that that could be considered two skills and they only asked for one. And that's where the violence comes in…