Running as Root

This comes up every now and then for me and I can never remember how to do it so I'm sticking it here to make it easier for me to find. The problem happens when I'm using Windows Explorer to open or browse a Windows share / Samba share / SMB mount point / etc. Windows Explorer has a tendency to cache the authentication information for the share and doesn't re-present the opportunity to provide authentication information in the event that the cached credentials have become invalid. This happened to me again today when the account I had used in the past had become disabled. You can find and clear the cached authentication(s) by doing the following:

Click Start, Run and type Control keymgr.dll
Remove the entries from the list

OR

Click Start, Run and type Control Userpasswords2
Click Advanced, Manage Passwords

The information is also in the Registry but these worked well enough for me to not go poking around in that rat's nest.

There are quite a few little cameos in The Incredible Hulk but the one I was happiest at having spotted was Rickson Gracie. Remember in Portuguese that 'R' sounds like an 'H', so it's pronounced "Hickson." Unfortunately they've listed him as an Akido instructor despite the fact that he's a bad ass in the family business of Brazilian Jiu Jitsu. I know it doesn't sound like it, but that's a colossal fuck up in martial arts land. Besides that it's a fair movie but after all the rumors about heavy editing changing the nature of the film I'd really like to see what the original "Norton" version was like. Maybe 25 years from now we can have 8 versions of it like Blade Runner.

Yeah, I'm not doing so well at sticking to the new "publish once a week" experiment. I either need a new rule allowing overflow or need to be more willing to stockpile these crappy jokes.

While I'm going to try and publish once a week, I have a hard time sitting on finished comics. I just can't resist the temptation to do two this week.

Hey, we're so agile we're bordering on chaos.

I think I'm going to just start putting my infrequently updated comic directly onto the blog. This will allow my to jam my junk in more peoples' faces and allow people to comment on the comic without having to create a Bitstrips account. I'm also going to try and update the comic at least once a week. We'll see how that goes.

My latest bug adventure has to do with the fact that at work we're transitioning to MySQL from SQL Server, a move I fully support.

First some detail on the way our application works. When our applet client syncs with the server it copies the records locally and stores them in a local database which is not MySQL. When you modify a record in the client it gets persisted first to the local database. Anywhere from immediately to the nebulous "later", the client will sync again with the server. When this happens a summary list of the data you can see is sent to the client. This data includes when the record was last updated on the server. This time is compared with your local records and a sync occurs. Local records with a later modified date get sent to the server and remote records with a later modified date get pulled to the client.

I'm not wild about this setup, mainly because I don't trust the time on the client machine since it's well outside of my control. We're also using the client generated time on the server as the last modified time. I think at the very least we should use the server time (interestingly, this wouldn't solve this problem in this case). Slightly more ideally we should use an incrementing version field that will have the benefit of better detecting update conflicts. That aside, we found that when we moved our test systems to MySQL the client was sending way too many records up to the server. Everything in the client-side database appeared to be newer.

It turns out that MySQL truncates timestamps and dates to second granularity. Anything finer than a second (millisecond, microsecond, whatever) is simply dropped. In the client, we're using a database that supports milliseconds. What this means is that if you modify a record at 11:52:27.421 it gets stored with that timestamp locally. When it gets stored in MySQL it is marked as last modified at 11:52:27. Therefore, your local record is almost always newer by literally a fraction of a second. Cool, huh?

Luckily, there's already a bug report. Given that it was reported over 3 years ago, I'm confident it is very nearly fixed. I am still a bit amazed that a database so popular in the enterprise fails at this very basic level of functionality.

As always, there are workarounds to the problem ranging anywhere from storing sub-second values in a separate field and/or creating a user defined type.

I recently got the task of finding a hosted monitoring solution for our production web site. There are quite a few options out there, so I decided to find a source listing a few of the options out there and sign up for some trial accounts. Most all of these sites are horny for your contact information. I'm fine with being contacted by email, but I don't particularly want to fill in the mandatory phone number field. I don't need to spend my day jawing with some glad-handing sales monkey. I write code for a living. So, I usually just put a 555 number in there (lazy developers never validate that shit). That way they can only get a hold of me if all of this is actually a movie, cleverly disguised to look like reality.

Not just one but two of the sales morons at these companies decided that my 555 number must be a cry for help. I want to talk to them and have them give me the hard sell, I was just confused about my real phone number. Not a problem. They used directory assistance and the rest of my real contact information to look up my company, find out our main number, and call several times. You would think that these idiots would realize that they were only going to piss me off by doing that.

The ultimate irony of it is that the site I was going to recommend was one of the ones that pulled this stunt. Unfortunately for them I'm a petty, angry developer. I'll be doing everything within my power to make sure they are the last solution we seriously consider. Now that's some sales job!

Every so often I have a brief love affair with awk. Today I got curious about the file sizes beneath a directory. In particular I wanted to see the totals by file extension. I did a quick search but came up with nothing. I decided that even if there is something out there to do the job, it'd be a lot more fun to do it myself. Tada:

ls -Rl | \
grep ^- | \
awk \
'{ split($9,e,"."); \
exts[e[length(e)==1?2:length(e)]]+=$5 } \
END \
{ for (ext in exts) printf "%10d %s\n", exts[ext], ext }' | \
sort

Yeah, there's an ugly hack in there to deal with file names that either have no extension or multiple dots in their name.

As an added bonus, looking at the previous post on awk got me all pissed off about "smart quotes" in Wordpress blogs and the problems they cause when copying and pasting code examples. So, out they go.

How's that for a catchy title? To recap recent events: I'm working on speeding up a Java applet, sloppy code in applet libraries try to load resources from the server which then 404, you can avoid this by setting the codebase_lookup property to false in the applet tag, and finally eliminating 23 megs of invisible data can help speed up downloading. Now that we're all caught up, let's turn to today's adventure: "deployment nightmares" OR "why the hell doesn't my test environment match production?"

Applet Won't Load

I finally got the applet to the "good enough for government work" level of load-time performance. Understand that I don't even work on any of the code in the applet, I'm just trying to optimize what's there and how it's delivered from the server. Today was the day we decided to quietly deploy to production.

The first sign of a problem was when the Apple guys came into the office. The applet wouldn't load for them in either Safari, Firefox 2, or Firefox 3. However, it worked fine on every server they tried except the production server. While trying to figure out what was going on it turned out that the issue had to do with all machines using JRE 1.5 regardless of OS or browser. They all worked against every server except production.

Differences Between Production and Test Environments

In production we have some sort of load balancer, Tomcat is behind IIS, and it's an external network. Nothing in our test environment has a load balancer in front of it, only one machine has IIS but works fine, and my external EC2 deployment is obviously off our network. I'm not sure why we don't mirror as much of this as possible in our test environment, but we don't.

Now back to the bug. Turning off the load balancer had no effect. Eventually, someone let their browser sit long enough to see that the applet did in fact load. It just took around 10 minutes. I finally noticed that the Java console would hang on different non-existent resources it tried to load from the server. I used cURL to retrieve the URL and had to wait 2 minutes until it returned an empty reply. Most non-existent resources timed out immediately. Only URLs that contained META-INF or WEB-INF would hang.

Various 3rd partly libraries were trying to load odd things from the server as I mentioned previously. A few of these load attempts point at the META-INF directory. This only happens under 1.5 because I used the codebase_lookup parameter in the tag. Tomcat, Apache in front of Tomcat, and our internal IIS server all return immediately. The first two serve a custom 404 page while the IIS server sends an immediate empty reply.

WEB-INF and META-INF Protection

Both WEB-INF and META-INF are directories that you probably shouldn't be exposing. In fact, in most versions of the Tomcat Connector the connector will automatically 403 or 404 when any resource from those directories is requested. In our case, we were running an older version of the connector that just happened to have a bug that caused requests to either directory to take 2 minutes to timeout. A quick upgrade and an IIS service bounce fixed everything.

So the debugging lessons for the day are: use something like ngrep to watch your traffic, your test environment should mirror your production environment, applets under 1.5 sucks, and check your version numbers on third party libraries (and consider upgrading).