Archive for the 'Project Management' Category

He's Got This Ultimate Set of Tools

Sunday, May 6th, 2012

"Relax, all right? My old man is a television repairman, he's got this ultimate set of tools. I can fix it." If you don't remember your Fast Times at Ridgemont High quotes you're probably not alone. The scene is worth remembering because the context is ridiculous. So it is sometimes with software development. The cost and effort of fixing the existing implementation is sometimes just too great. The changes cut too deep. You're better off throwing out the current stuff and starting from scratch.

In software development you rarely understand your problem domain perfectly, if ever. You learn what your customers want through trial and error. Sometimes your organization has made such poor attempts at delivering the product people want that you can't help but throw away what you've currently got and try again with what you learned from your previous attempt.

Managers usually hate to hear such talk from developers. Developers always want to rewrite things. But in some rare cases they're absolutely right. Refactoring is great if you're even remotely close to what you want to do. But what if your product is built on bad assumptions of epic proportions?

Could CVS have been refactored incrementally to arrive at git? Could Windows have been refactored to create Linux? Could MacOS have been refactored to create OSX? Could Internet Explorer be refactored to create Chrome? When do you come to the realization that what you want, what you need, is so far away from what you have that you can't get there from here? When is the cost of making changes to your current product artificially inflated by the technical debt and faulty abstractions to the extent that it's better to throw it all away?

That's the advantage your competition has. You've shown them your near miss at a great product. If the people in your organization advocating a rewrite were magically transported into a competing startup that was creating a competing product from scratch would you be at all worried? If the answer is "yes" then you should use the advantages you have (those very same people plus a more intimate knowledge of the problem domain and where you went wrong) and do something about it. Plus if something in your product actually proves useful you can copy and refactor it into the new product.

There are certainly risks but the rewards are incredible.


I Think We're Going to Need a Bigger Box

Tuesday, April 10th, 2012

I was reading this post on the Instagram buyout by Facebook today and it got me to thinking about the benefits of the cloud, DevOps, horizontal scalability (one of my favorites), and well thought out architectures and monitoring.

One of the more interesting things about the $1 billion purchase price is that Instagram has 13 employees and 35 million users. That's just so crazy to me. It also ends up being yet another argument against the "bigger box" method of solving scalability issues. Eventually you cannot simply add more RAM to fix things. Trying to solve your problems that way is like trying to solve world hunger by breeding a single, giant cow.


Let's Just Burn It All Down and Start Again

Saturday, April 7th, 2012

All software sucks to some extent including everything you are working on right now. If you reexamine your code six months from now and don't think it sucks then it probably means you didn't learn anything in those six months. That's the downside of being a software developer. You feel like the code you're working around is some degree of horrible. For the most part you just accept it and try to make incremental improvements to things. If you're lucky you'll work on something that you think is magnificent (and then think it's shit in six months).

But what happens when the code is truly horrific? For example: you wrote your own FTP client, your own templating engine, you have mutating getters, there's database access in your pages and data objects, you cut and paste DDL statements into SQL clients and call it "upgrading the schema", etc. We can argue about whether some of those things are truly bad but from my perspective they're pretty rotten. Throw that into a 100k+ line code base with many active customers and too few developers and then you've got some real fun.

In these situations I can envision a more ideal code base pretty easily. Update the libraries and start using them, fix the schema that no longer matches the problem domain (if it ever did), start pushing things into neat little tiers, get rid of that shitty build, run a continuous integration build server, use Chef or Puppet to manage configuration, scale your shit horizontally and get all elasticy with the cloud, etc. Pretty soon I've built a shining city on the hill in my mind. The only problem is I'm still calf deep in shit and I need to go back to standing on my head just as soon as my lunch break is over.

My solution has always been to burn everything to the ground and start over. It's not a popular position even among software developers. "Let's just slowly fix everything that is wrong," they say. It sounds good but progress on paying down your massive technical debt always seems to take a backseat to a shiny new feature (with its own share of technical debt). Pretty soon you're not even paying the interest on that debt. Nope. Burn it all down. Or at least build a new bridge next to the old bridge and then blow the old bridge up. Maybe you can even be nice enough to divert traffic first.

The "fix in place" crowd always sounds like this to me: "I bought a new motorcycle. It's a Honda. I kind of want a Harley instead. Can you turn it into a Harley while I ride it around? Thanks. xxxooo"

At least I'll always have these rants before the void. Thanks for listening.


Better Programmer Interviews

Thursday, April 14th, 2011

One of my former co-workers wrote some of his thoughts on crappy interview questions as well as some advice on improving the situation. My latest job was the first time I had to write code during the interview. It was interesting although I think the problem was a bit trivial. The thing I liked about it was that it started with an OOAD design question about a specific problem and then segued into you coding your solution. It was kind of a nice "eat your own design dog food while I watch" moment.

However, when I read the post I mentioned above it occurred to me that you might be able to use open source projects to improve on this a bit. The idea I had was to use an open source library on which you depend and have the interviewee either address a bug or add a feature that you've wanted. This can easily be a take home type of question as well. The plus is you get to do a code review on their submission and get a feature you want. The open source software community benefits as well. It's just wins all around, baby!


CodeGreen Labs

Tuesday, June 24th, 2008

I've been listening to back episodes of the Agile Toolkit Podcast. One of the things that really caught my attention was the frequent mentioning of something called CodeGreen Labs. From elsewhere on the internet:

We are dedicated to creating a unique training experience that uses real projects with real deliverables. Labs work on projects that make a difference in the world, benefiting organizations that are working to improve the environment, human rights, social justice, health and economic development. By marrying the efforts of dedicated professionals learning Agile Development techniques with not for profit and research communities that need high quality software, we hope to be a small part of a positive change. Each training class will work on a specific software project for a specific cause.

It's volunteering to work on a real software project that is, hopefully, well run so that you can get real world experience. Sure, they get something out of it, but who cares about the other guy? I imagine they get mainly students. A few of the people in my work circles and I have been talking about something exactly like this. The head of my current company has mentioned to me numerous times that he's seeing disturbingly bad candidates coming to him fresh out of college CS programs. I can't comment as to the overall quality of CS programs around the country, but something like CodeGreen Labs seems like a brilliant idea on many levels. Unfortunately, I was a little disappointed at how unfriendly their home page was:



The Return of the Daily Stand Up

Wednesday, April 30th, 2008

I switched jobs recently and although the current company used to do daily stand up status meetings ala Scrum/XP/Agile/Wagilefall, they had stopped at some point. As near as I can tell, they stopped because they couldn't keep the meetings short and on track. I expressed to my manager(s) that I thought it'd be great to start doing them again and lo and behold we had one on Monday.

Of course, the first issue that came up was the notion of the correct time. Inevitably when you try to start a meeting "on time" you get bogged down with the fact that no one thinks they're late because their clock says they're not. In the past I've used web based atomic clocks, but I'm now convinced that anyone planning to start meetings on time has to invest in a digital clock that synchronizes with the official time. I further suspect that anyone thinking that the people they work with are too mature to have this "argument" is probably wrong. We also haven't settled on the "punishment" for being late. I'm a big fan of cash fines though.


On the Telephone!?

Sunday, September 16th, 2007

In the Future, Everything Will Be Glue

The entry title is from Weird Science, by the way:

Gary: I was crazy for this little eighth grade bitch.
Wyatt: Crazy, Insane!
Gary: I was dedicated to this girl, I called her every damn night!
Old Pimp Dude in the Bar: You called her every night? On the telephone?
Gary: On the telephone? What's he mean on the telephone, course it was on the telephone!

In my work life, the team I'm on has been working for quite some time fixing the architecture for an existing product. One of the things we've been doing while we've been fixing things is to add a REST API. You can save the REST vs WS-* debate for some other time and place.

Meet the New Product, Same as the Old Product

Unfortunately, there was a distinct lack of excitement about the new stuff we'd been doing from other areas of the company. Programmers seem to immediately "get" why exposing all of the CRUD functionality of your system through a web API is the sheer sweetness in terms of application integration. Sales and executive staff don't always see the possibilities. They saw the new version as architecture changes that improved performance and scalability while maintaining functional parity with the shipping version. That didn't scream sexy.

The programmers, ever fearing being a cost center with a very killable project, decided to start creating proof of concept integration examples that would create internal demand for the nearly finished version.

Pretty State Machine

As background, our system has an operational state that changes the way almost everything behaves. The change in this state is typically user driven. Someone using the software would decide to change the operational state due to an external event such as a Martian invasion. The software has a web interface but we had the idea that it'd be much cooler to change the operational state via a telephone.

We can change our operational state via the REST API by putting (as in HTTP PUT) an XML document representing the system status to the server. The system status has an element that represents the operational state. Now, our IT guy is always telling me the crazy stuff he can do with his home VOIP system. He runs a virtual machine with Asterisk and other assorted software an combination with an IP phone. I ran the idea of executing a shell script from a phone menu by him and he assured me it should be easy to do.

I was working under a deadline (of course) so I created three versions of the XML (one for each state I would be changing to) and got a copy of cURL installed in the Asterisk VM to PUT each document to our server, hopefully in response to someone pressing a button on a phone. After hitting a very serious dead end and wasting several hours trying to get an IVR working (that's what phone people call those voice menus) I finally found an example of an Asterisk iTunes controller. To use it you dial an extension, hear a beep, then press a button to do something. The script in Asterisk then calls a shell script in response to your button press. Five minutes after finding it I had a working example of changing operational state in our product via a telephone (actually a softphone but who's really keeping score at this point). Booyah! Sure it only took me 10 hours discover how to do an hour or so worth of work but then that's the plight of the knowledge worker.

Demo Day

The next day we invited additional people to our iteration demo meeting (really it's just the people that were supposed to be going all along but weren't) and showed the phone demo as our finale. Suddenly, people seemed as interested in our new version of the product as the developers are. There was a palpable sense of excitement. There were tears of joy. The developers were hoisted up onto the shoulders of sales and management and carried out into the streets (think about the scene in Dragon: The Bruce Lee Story).

I'm exaggerating a little bit perhaps. Of course there are all sorts of negatives preceding and following this event, but can't we just bask in the glory of this one little success for just a few moments? On the telephone.


Improving Wiki

Monday, August 28th, 2006

I've already gone over my desire to add more tools and procedures to our distributed team development process. My next issue is with one of the tools that I like quite a bit. I think that the way we're using Wiki is leaving something to be desired.

Get Organized

The first problem I have is that it's horribly disorganized. It's extremely hard to find things. Design docs, developer environment setup, refactoring plans, third party tool evaluations, etc are all jumbled together. You can't navigate from the top level page to anything useful. That's not a huge deal for me since I'm more about searching than the dreaded hierarchy, but the search also leaves something to be desired.

Someone at my company suggested that the problem stems from the fact that it really takes someone with an overarching idea of (and a strong passion for) how to organize things. I don't think it needs to be that strong, but some simple high level guidelines of what goes where would help. It's great that anyone can add or edit a page, but in the absence of any guidelines our Wiki is looking more like a wall with 300 Post-It notes on it than anything really useful.

When Was This Updated?

The next problem is that Wiki pages usually don't get updated when things change. This leaves the Wiki full of a bad combination of mostly accurate and completely incorrect information. The Wiki page on the design of a module or its example code is too far away from the code.

That's why I like Javadocs so much. The documentation is right next to what you're documenting. If we think that not everything is best to document in the code, I'm thinking we may want to at least put a link to the Wiki documents in the Javadocs and make the evaluation of a Wiki update a part of the peer code review check-in process.

You're Not Revision Control

All of the design docs, tips, example code, support notes, and pretty much everything else related to the code or the product are accurate for a particular version of the system. If you're being good about updating things then the pages in the Wiki reflect the way things are more than the way things were.

For multiple versions of software, it's easy to forget how things used to work. Yes, there's some crappy versioning built into most Wikis but it's not really what they do best and it shows (just like their integrated search). Tying a Wiki page to a release of your software requires putting an artificial mechanism in place (like a naming convention, a structure to the hierarchy, or tagging if the Wiki supports it) or finding a Wiki that provides more powerful tools for finding topics by version (repository style tagging and viewing of a Wiki).

Integrate This *Flash*

Probably the easiest solution would be to choose a Wiki tool that supports top level categorization of your Wiki into sub-Wikis (like what Twiki does with webs). One downside would probably be that you would start off each new release cycle of the product by copying the previous category over to the new one, but maybe that's not that bad. You'd probably also want to at least do some of the high level guidelines and have the topic / category structure closely mimic your code where appropriate.

Another idea I was thinking of involves looking into either a file-based Wiki that I can directly put into the revision control system of my choice (Subversion, baby) or that directly integrates with RCS of my choice (did I mention Subversion?). At least for code related documentation (design docs, examples, dev environment setup, etc) this seems to be an intriguing solution. If the topic relates directly to a package / module then its path should directly mirror that. Otherwise it goes under one of several high level categories like I mentioned parenthetically. Reviewing the Wiki documentation would be part of a peer code review regardless of how the versioning issue is solved.

These release specific Wikis would still be published by the build to some internal website so that a super-Wiki could be created for support, QA, etc. I'd also think about maintaining the non-RCS Wiki for any information that doesn't relate to either the product or a specific version of the product (whatever they may be).

I'm still debating which approach I like best, but I am definitely certain that I think what we're doing now is inadequate.


I Miss My Office

Tuesday, August 1st, 2006

In my last two jobs (including my current position) I have been in a cubicle. You can find plenty of arguments both for and against cube life on the web so I won't bother rehashing them. Suffice it to say that I think the benefits of giving programmers private offices far outweigh the perceived benefits of having a group space or the cost savings of sticking me in a 2 1/2 walled box. Case in point, I can barely work at the moment because two people have been having a conversation about fucking Weight Watchers for the last 20 minutes two cubes down from me. As fascinating as I find peoples' struggles with obesity, I'd really rather be working.


There Must Be a Better Way

Saturday, July 8th, 2006

The Current State of Things

I'm in my third week of work at the new place. As with every place I've ever worked, they (we from here on out) have a lot of the same old problems. We have a shipping product, so we've got legacy code. Most legacy code I encounter anywhere is brittle, lacks unit tests, and has a severe lack of encapsulation / cohesion. The code hasn't rotted, peoples' view of the right way of doing things have evolved. When you look back at whatever you did 6 months to a year ago you're horrified. In addition the build, project organization, and installer are all screwed up. That too is typical. These things require more planning than they get on a normal project. They're the areas most likely to grow organically and as such they get messy and out of control very easily. Luckily with software everything if fixable.

The other process level problem we have is that we're dealing with a geographically separated development team. Although this goes against a major Agile rule, it seems to happen everywhere in the software world. We use IRC, revision control commit emails, daily status meetings, and announcement emails for major code changes to try and keep everyone in sync. Given that we're trying to fix all of the code level problems you can imagine that a lot of the system is changing very rapidly. The need for communication is even more heightened than normal.

Problems with IRC

At lunch on Friday with my co-workers we got into a discussion about the concerns about the effectiveness of our communication. A couple of people on the team (me among them) have made the statement that things aren't nearly as bad as other members seem to think. That is to say I have seen worse situations. We have a very good team of highly competent people that have an interest in improving things. The communication is certainly not the worst I have seen. That being said, I certainly think we can fix a few things.

I jumped into the lunch conversation saying that I thought both our email and IRC communication needed addressing. First, IRC is being misused. Someone said it is like hallway conversations for people that are off site. I like that analogy, but we're still missing slightly. We have too many inane, off topic conversations. I'm not interested in reading peoples' small talk. Second, we have too many one on one conversations taking place in the main channel. I think you can usually determine when you need to have a one on one conversation and take it to a private chat or phone call. I don't buy the argument that everyone needs to overhear everything in case they might have input. If these are hallway conversations, it's like every jackass has decided to have their meetings right outside my cubicle.

Another problem is that some people don't understand what is appropriate to communicate via IRC. Last week someone posted a link to a document they wanted the developers to review. Only one developer saw it. That shit needs to be sent in an email. And finally, any decision that gets made in IRC needs to be announced to a wider audience in a more visible way (most likely an email).

Problems with Email

One of the developers seemed to be getting a little irritated with me pointing out problems with our usage of IRC. Since I didn't pick up on it at the time I brought up why I thought our email communication sucked, too.

My problems with email are that it's a little to broadcasty, there's no archiving, no searching, it's non-threaded, and not good for soliciting responses. All of these problems have solutions within email, it's just that I feel that these solutions are tacked on after the fact and may still not be as effective as other solutions that were built with the problems in mind in the first place.

By broadcast, I mean both that I'm on email distribution lists that often have emails I don't particularly care about. Yes, we could make finer grained email distribution lists for a larger array of general subjects, but that gets messy especially since I can't currently control my membership on lists. This has to be done by someone with the correct permissions. Email rules to auto-delete are pretty imperfect as well.

As for archiving and searching, most people say I can archive my email locally and search on it whenever I feel like. This doesn't work unless I'm on my computer and doesn't work for any email sent before I started working here. Yes, you could tack on another product to automatically archive and index the email centrally and expose a web interface. This can even add some threading capabilities. This is one of the better solutions to a problem but I still find it lacking. Threading is typically done by parsing the subject line. Screw that up and you may not be in the right thread. Responding to old threads would require you paste the subject line from the web archive site and if you're really nice paste the body of one of the later posts. That would prevent people from having to run to the archive site to see what the hell you're talking about.

The problem of soliciting response from email is that you send out an email and end it with, "Is everyone good with that?" You almost never get a response. Maybe people don't respond if there's no problem. Maybe they didn't read it. Maybe they disagree but don't want to be the only one. Who knows? Sure you can put email receipt requests on your email, but the client on the other end has to support it and generally the receiver has to explicitly allow it. And so what if they read it? Do you want everyone to respond and say agree or disagree? That seems a little intrusive for most topics.


At that point the slightly annoyed developer asked how I intended to solve these problems. I said I didn't think I had a solution for every one of these but that I thought there was a better way of doing things. For some of the problems I just don't know. She made the statement that she was tired of me bitching about the problem if I didn't have a solution.

I complain often, so I'm used to that response. People tend to feel that where there's smoke there some asshole standing next to a fire saying, "Someone should really do something about all this goddamn smoke." I feel that premature labelling of "bitcher" (since I've been there 3 whole weeks) is like me seeing dials that say something is going to blow up and being told, "Don't go bitching about the dials until you have a solution. Keep it to yourself." Of course, as always, the truth is likely somewhere in the middle.

My problem is that the first step to improving a situation is identifying the current shortcomings in your situation. Yes, something has to happen after that, but that doesn't invalidate the position of the person saying our status quo is inadequate. I'm sure as the other developer and I get more used to each other's quirks we'll be fine. I'm a non-complaining complainer while she appears to be very passionate and frustrated by institutional inertia that has prevented solutions from being implemented in the past.


Since there seemed to be a real desire for me to put up or shut up, I detailed what I thought were some of the solutions. They're not perfect, but they were the best I could do at the time.

I like blogs as a developer diary. This is a place developers can go into more detail about their daily activities. The daily stand up should remain short and very high level. If you want more detail on what I am working on, check my internal blog. Blogs are also a good place to share quick tips on doing things (like how to use find, sed, tr, etc to perform a quick search and replace while copying the original file to another location). Blogs are also good for initial conversation starters and what-if scenarios. If an idea is worth discussing it moves off the blog and into threaded discussion forums.

Blogs aren't great for discussing things, but forums are pretty good at it. RSS feeds are a must so you get better notification of activity in the forums. You pay attention / subscribe to whatever you want. You can control the influx of information, unlike with email distribution lists. It's centralized, archived, searchable, and threaded. What's not to love? If you reach an actual decision move it to the wiki.

The wiki is not good for making announcements, having discussions, or keeping a developer diary. It is good for documenting things that have been decided. Yes, you can continue to tweak an idea, but major change discussions should happen in the forums with the results being captured in the wiki. I have my own problems with wikis that I'll detail in another post (they're very fixable). The wiki itself should have RSS feeds so you can easily see changes to the topics you care about.

If you need to solicit opinions, maybe you need to post a poll on the forums. I'm least happy with this solution, but I think it is a step in the right direction. Big decisions still should have a design review meeting I believe. Again, the designs should be captured somewhere, most likely the wiki.

Email is still used for quick announcements like "I modified the POM, so be sure to regenerate your Ecllipse projects" that don't really fit anywhere else. Email and shared calendar solutions are also great for scheduling meetings. Email is also used to communicate one on one changes like "Hey, I just checked in those changes you needed to complete your task for this iteration." I'm probably missing a lot of valid use cases for email so I'll just assume that email is the traditional catch all for things until we find / agree upon a better place.

There are other tools that I haven't mentioned but that are still needed. For example, you need a bug tracking system and an iteration planning tool.

My, Aren't We Tool Heavy?

I don't believe the tools are the complete solution. If your process is completely screwed up, the tools aren't going to fix it. I do think that the tools are a part of the solution though. For example, I don't like leaving the decision to document a change up to the developer making the change. They're too close to things. I like peer code reviews on check in with one of the items on the checklist being that an announcement needs to be made if the change is "big." A second set of eyes that didn't make the change helps greatly in making this decision. But, you still need a tool of some sort to communicate the change. Peoples' fear of having too many tools seems to make them force the one universally accepted tool (email) into roles it wasn't meant to fulfill. You wouldn't try to track bugs in email, would you? Maybe you shouldn't be using it for a lot of the other things for which you're currently using it…

As always, keep in mind that my ideas are a work in progress. Feel free to chime in if you have advice for better ways of doing things.