Running as Root

I recently got the task of finding a hosted monitoring solution for our production web site. There are quite a few options out there, so I decided to find a source listing a few of the options out there and sign up for some trial accounts. Most all of these sites are horny for your contact information. I'm fine with being contacted by email, but I don't particularly want to fill in the mandatory phone number field. I don't need to spend my day jawing with some glad-handing sales monkey. I write code for a living. So, I usually just put a 555 number in there (lazy developers never validate that shit). That way they can only get a hold of me if all of this is actually a movie, cleverly disguised to look like reality.

Not just one but two of the sales morons at these companies decided that my 555 number must be a cry for help. I want to talk to them and have them give me the hard sell, I was just confused about my real phone number. Not a problem. They used directory assistance and the rest of my real contact information to look up my company, find out our main number, and call several times. You would think that these idiots would realize that they were only going to piss me off by doing that.

The ultimate irony of it is that the site I was going to recommend was one of the ones that pulled this stunt. Unfortunately for them I'm a petty, angry developer. I'll be doing everything within my power to make sure they are the last solution we seriously consider. Now that's some sales job!

Every so often I have a brief love affair with awk. Today I got curious about the file sizes beneath a directory. In particular I wanted to see the totals by file extension. I did a quick search but came up with nothing. I decided that even if there is something out there to do the job, it'd be a lot more fun to do it myself. Tada:

ls -Rl | \
grep ^- | \
awk \
'{ split($9,e,"."); \
exts[e[length(e)==1?2:length(e)]]+=$5 } \
END \
{ for (ext in exts) printf "%10d %s\n", exts[ext], ext }' | \
sort

Yeah, there's an ugly hack in there to deal with file names that either have no extension or multiple dots in their name.

As an added bonus, looking at the previous post on awk got me all pissed off about "smart quotes" in WordPress blogs and the problems they cause when copying and pasting code examples. So, out they go.

How's that for a catchy title? To recap recent events: I'm working on speeding up a Java applet, sloppy code in applet libraries try to load resources from the server which then 404, you can avoid this by setting the codebase_lookup property to false in the applet tag, and finally eliminating 23 megs of invisible data can help speed up downloading. Now that we're all caught up, let's turn to today's adventure: "deployment nightmares" OR "why the hell doesn't my test environment match production?"

Applet Won't Load

I finally got the applet to the "good enough for government work" level of load-time performance. Understand that I don't even work on any of the code in the applet, I'm just trying to optimize what's there and how it's delivered from the server. Today was the day we decided to quietly deploy to production.

The first sign of a problem was when the Apple guys came into the office. The applet wouldn't load for them in either Safari, Firefox 2, or Firefox 3. However, it worked fine on every server they tried except the production server. While trying to figure out what was going on it turned out that the issue had to do with all machines using JRE 1.5 regardless of OS or browser. They all worked against every server except production.

Differences Between Production and Test Environments

In production we have some sort of load balancer, Tomcat is behind IIS, and it's an external network. Nothing in our test environment has a load balancer in front of it, only one machine has IIS but works fine, and my external EC2 deployment is obviously off our network. I'm not sure why we don't mirror as much of this as possible in our test environment, but we don't.

Now back to the bug. Turning off the load balancer had no effect. Eventually, someone let their browser sit long enough to see that the applet did in fact load. It just took around 10 minutes. I finally noticed that the Java console would hang on different non-existent resources it tried to load from the server. I used cURL to retrieve the URL and had to wait 2 minutes until it returned an empty reply. Most non-existent resources timed out immediately. Only URLs that contained META-INF or WEB-INF would hang.

Various 3rd partly libraries were trying to load odd things from the server as I mentioned previously. A few of these load attempts point at the META-INF directory. This only happens under 1.5 because I used the codebase_lookup parameter in the tag. Tomcat, Apache in front of Tomcat, and our internal IIS server all return immediately. The first two serve a custom 404 page while the IIS server sends an immediate empty reply.

WEB-INF and META-INF Protection

Both WEB-INF and META-INF are directories that you probably shouldn't be exposing. In fact, in most versions of the Tomcat Connector the connector will automatically 403 or 404 when any resource from those directories is requested. In our case, we were running an older version of the connector that just happened to have a bug that caused requests to either directory to take 2 minutes to timeout. A quick upgrade and an IIS service bounce fixed everything.

So the debugging lessons for the day are: use something like ngrep to watch your traffic, your test environment should mirror your production environment, applets under 1.5 sucks, and check your version numbers on third party libraries (and consider upgrading).

Sadly, the Olympics are once again over, leaving me with Olympic cravings until the 2010 Vancouver Winter Olympics roll around. In a couple of years I can look back on this post and remember the high and low points of the greatest Olympics to date. I'm sure I've left out a lot from both lists.

Worst Moments

Here's a brief list of what I found to be the worst moments in the Olympics.

• Taiwan – It's complete bullshit that Taiwan had to participate under a different name (Chinese Taipei), couldn't use their own flag and instead competed under the "Taiwanese Olympic Flag", and would not be allowed to play their national anthem in the event of winning a gold medal (which they didn't).
• Brazilian Soccer – In one of the early games against an over-matched opponent, one of the Brazilian soccer players scored a goal. Right after the ball rolled out of the goal another Brazilian player kicked the living shit out of the ball, sending it into the back of the net again. What a dick move.
• Underage Chinese Gymnasts – Screw innocent until proven guilty. Those female Chinese gymnasts are too young to compete. You can make the argument that the rule itself should be removed (I disagree), but until then it is a rule. I think the Chinese government is involved in falsifying documents in order to allow them to compete. I honestly hope the truth comes out and the Chinese are stripped of the team gold as well as the affected individual medals.
• Gymnastics Judging – An Olympics cannot seem to happen without there being several incidents of questionable judging in a gymnastic event. The worst this time was probably the women's vault finals and the women's uneven parallel bar finals. The vault was a blatant overscore for a vault with many faults, not the least of which was the attempted landing. The uneven bars was a low score that led to a poorly understood tie breaker. I'd prefer to see the days where athletes had to share medal positions in the event of a tie.
• Women's Softball at an End – Unless something happens, this is the last Olympics for Women's Softball. Ironically, the dominance of the American women is cited as the reason for dropping the event (as well as an incorrect association with baseball). It's ironic because the U.S. women failed to win the gold medal this time, losing to Japan (a team they beat in the semi-finals). It's not our fault that other countries oppress their women (more than we do). Are we advocating canceling any event in which a country is dominant? Kiss diving good-bye (China won 7 out of the total 8 diving gold medals available and only one Chinese diver failed to earn a medal of any color). But diving is nothing compared to women's table tennis. 35 or the 86 women competing in table tennis in Beijing were born, raised, and trained in China and for all of table tennis the figure is 55 out of 155. They're so dominant that no one else has any hope of winning unless they import a Chinese player. Cancel it!
• Fernando Gonzalez's Lack of Sportsmanship – Although the outcome of the match may not have been changed, James Blake's ball hit Fernando Gonzalez's racket and Gonzalez knew it in my opinion. He's just too much of a gutless pussy to admit it.
• Women's Fencing is Unwatchable – Despite the fact that the U.S. women swept the medals in individual sabre, I can't watch that crap. It looks like a school yard slap fight with tent poles. But, what makes it completely unwatchable is the banshee-like scream the competitors let out at the end of every point.
• Olympic Boxing Needs an Overhaul – Besides the looming controversy about attempted scoring manipulation by one or more countries, the scoring system and its execution are deeply flawed. The scoring in the Paddy Barnes versus Zou Shiming fight was outrageous. Body shots are never scored. The current score is readily available to each corner. This encourages late fight running by the leader who is assured a win if he can just stay out of scoring range. One of the things I have always liked about boxing is that it's one of the few sports in which you don't even know the score. Oh, and the rounds are too short. The rules of amateur boxing cheapen an already declining sport.
• Lisa Leslie Wearing All Her Medals from Past Olympics – Sure, she's very accomplished being the only athlete to win team gold in 4 Olympics, but wearing all your past medals to the current medal ceremony was tacky.
• Olympic Racewalking – The fact that NBC showed Olympic Racewalking not once but twice in late night coverage instead of showing other events with more mass appeal is moronic. Archery didn't appear once in televised coverage. Even though racewalking has been an official event since the 1908 Olympics, the "sport" is ridiculous. It should be dropped from the roster. At the very least, show something better than an hour and twenty minutes of hip wiggling mall walkers.
• Microsoft Ruined Online Coverage of the Olympics – I couldn't/wouldn't view any of the online coverage (which sucked anyway because there was no audio commentary) because, inexplicably, someone cut a deal with Microsoft so that viewing required Silverlight. As much as I appreciate someone trying to cram their, arguably beta, technology down my throat I would have liked a more widely adopted, relatively platform independent, working browser-based video playback option. Is something suddenly wrong with Flash? Way to drastically cut your online viewership numbers. I hope whatever check Microsoft cut for them was a big one.

Best Moments

There were so many great moments. Here are some that really stuck out for me.

• Opening Ceremony – Despite a terrible accident during rehearsals and other inevitable controversies, it was easily the best opening ceremony in history. The whole thing was amazing. The 897 human powered movable type blocks routine was jaw-dropping.
• Women's Soccer U.S. versus Norway – Two early misplays by the U.S. led to two goals for Norway within the first four minutes of the game. They were the only goals scored in the game, but I enjoyed the fact that nothing is certain. The heavily favored U.S. team screwed up and Norway beautifully capitalized on it. Good stuff.
• Michael Phelps – Some people grew tired of it, but I never did. I watched every heat, every race. I thought the men's 4×100 freestyle relay was the most exciting Olympic moment I had ever seen. That is until I saw the men's 100m butterfly a couple of nights later. For me those are the two most amazing races in Olympic history, swimming or otherwise. I wish they would stretch out the swimming competitions to allow swimmers the possibility of competing in even more events. As it stands, the required turnaround time between heats and events makes it next to impossible even though a swimmer may be a competitive threat in numerous events. This picture of Phelps with a mustache is another favorite of mine.
• Mark Spitz – It's related to Phelps, but I think it deserves a separate mention. The interview Spitz gave with Phelps on NBC was very satisfying. Spitz was incredibly gracious and well spoken. It's very refreshing to see someone whose amazing record has fallen be so incredibly respectful of the next guy, especially after hearing so many cry babies claiming they were better in their time. Let someone else sing your praises. Spitz did it right.
• Dara Torres – She's a great story in general (competing at that level at 41 years old is awe inspiring), but I was even more impressed when she worked to delay the start of a qualifying heat so the Swedish swimmer could finish changing out of her ripped suit. Nice sportsmanship. I wish she had won a gold, but she was still amazing to watch.
• Usain Bolt – Dominance in sports is sometimes extremely entertaining. Watching Usain Bolt prematurely celebrate in the 100m dash and still set a world record was awesome. I don't fault him for showboating. He's young (he turned 22 during the games) and is at the top of his sport. At least he got serious in the 200m when he again broke the world record with a head wind. It was also good to see him and Asafa Powell dancing in the finish area after again breaking a world record in the 4x100m relay. It's too bad the Americans are such butterfingers at baton passing (both the women and men were disqualified in the heats for dropped batons), but they honestly didn't stand a chance anyway.
• Volleyball – All of the volleyball was outstanding. Men's and women's beach and indoor were extremely entertaining to watch. Lloyd Ball and Logan Tom emerged as two of my favorite players for the indoor men's and women's teams respectively. It's a terrible shame that the men's indoor team had to labor under such tragedy.
• India vs Spain Men's Table Tennis Singles – The Sharath Kamal Achanta (India) versus Alfredo Carneros (Spain) match happened in round 1 and wasn't for a medal. What made the match interesting is that the two train together and had the misfortune of meeting in the first round. The two were as friendly as possible given the circumstances, giving each other plenty of time, warning one another when their play surface or paddle became damp with sweat, and apologizing profusely for net or edge balls. At the end of the match it was even hard to tell that Achanta had won because he refused to celebrate in front of his training partner whose Olympics had just ended.

Thoughts on the "China Problem"

China's fucked up, there's no doubt about it. No one likes their human rights abuses, the number of people forced from their homes to make the Olympics happen, their record on the environment, their totalitarian government, and a plethora of other problems and issues the country faces. Should we have boycotted the Olympics? I don't think so. Once they were given the Olympics it wouldn't be fair the the athletes to support a boycott. You could make the argument that they shouldn't have been given the Olympics in the first place, but I would still have to disagree. Even though every success economic or otherwise only makes the current government look better I think that cutting off China from the world isn't the answer. They've proven their ability and willingness to remain isolationists for centuries. I like to think engaging them culturally and economically can help open them up and hopefully turn many of their current problems around. If it happens it'll happen slowly. Only time will tell, but I don't think boycotting the Olympics would make the difference.

Looking Ahead to Vancouver

The next big Olympics will be in Vancouver in 2010. I'm hoping there's no talk of a boycott because of Canada's well documented maple syrup abuses. I'm giddy with excitement that those winter games will kickoff on February 12th, 2010. That means it's only a year and a half away rather than a full two years. Plus, since they're in North America, I'm toying with the idea of attending. Unfortunately, that would mean giving up my 24×7 television coverage. Would being there live make up for the loss? I'll have to think about it.

For the past few weeks I've been working on improving the load time of an applet at work. Someone here noticed a while back that we seemed to have a ridiculous amount of image files in the applet. There are roughly 23 megabytes of images in the final jar which is around 14 megabytes in total size.

I opened a couple of the files in GIMP and resaved them to find that the final file was much smaller than the original. It turns out that the images were created using Fireworks and by default it puts some extra information in the PNG file, things like layers or palette information I believe. A few minutes of searching around on the internet and I found a wonderful tool named PNGOUT that could be used to losslessly optimize the size of PNG files. I used Cygwin (I'm on Windows at work) to run all of the PNG files through the utility via this command: find . -type f -name '*.png' -exec pngout.exe {} \; and waited a few minutes. The end result was as follows:

Before PNGOUT
Total size of images: 24,147,770 bytes
Total size of app jar: 14,086,540 bytes

After PNGOUT
Total size of images: 1,026,186 bytes
Total size of app jar: 4,335,560 bytes

Yikes. Not bad for a few minutes worth of work.

As I mentioned in the last post I'm farting around with applets for work. You may remember that the applet was hammering the server whenever it couldn't find a resource in the jars. Of course, everything the applet needs is already in the jars so if it's not in them then it's not on the server either. It's all because the AppletClassLoader tries to load stuff from the codebase on the server if it fails to load it from the jars.

As part of the fix of setting codebase_lookup to false I did some very quick, unofficial benchmarking. The test environment was an EC2 deployment I've been messing around with–the smallest instance. I timed the startup time of the applet and counted the number of server hits during startup. To further minimize variablility, I did this after the jars had already been cached. This roughly corresponds to the startup time of a visitor to the site that has already successfully launched the applet. The results were as follows:

codebase_lookup – true (default setting)
Startup time: 35 seconds
Server hits: 442

codebase_lookup – false
Startup time: 7 seconds
Server hits: 34

When the the applet does hit the server to look stuff up in the code base, it doesn't just do it at startup. Some libraries that don't cache failed attempts to load a resource keep on hitting the server. In our app it's XFire and a lot of non-existent "aegis.xml" and "doc.xml" files as well as a whole bunch of "BeanInfo.class" attempts thanks to java.beans.Introspector. Each of these attempts takes a tiny bit of time and puts an annoying 404 in the access logs. It's hard to say what the distributed sluggishness of the app will be because of all these little attempts, but it's definitely non zero. It also has an affect on server side scalability when each client is potentially 13 times more chatty with the server than it needs to be.

An additional concern I have is that some of this cavalier attitude toward resource loading in these libraries also happens on the server side. How many failed getResourceAsStream attempts am I not seeing and what impact are they having on overall server performance? At the current traffic levels it's probably insignificant but the idea of that much inefficiency spread out throughout the app kind of bothers me.

XFire Mapping Files

The application I work on uses XFire as a Java SOAP framework (this project has been replaced by CXF which is why this isn't a bug report with a patch) that is launched via Web Start. At some point we decided to allow it to be run as an applet as well. One of the weird things I ran into when we did this was a ton of 404 errors in the access logs on the server. The applet was repeatedly trying to load these "*.aegis.xml" files. These are mapping files used to configure how XFire maps your types to XML. However, you don't necessarily need these files if you elect to configure the mappings another way. XFire apparently still looks for the mapping file to allow you a mechanism for overriding other mapping methods. Everybody still here? Good.

Repeatedly Loading Non-Existent Resources

Since we don't use the mapping files, they're not in any of the jars that would be loaded by the applet. So what happens is XFire calls XMLClassMetaInfoManager.getDocument whenever it needs to figure out how to map one of your objects to XML. The manager then executes a getResourceAsStream to see if you have a mapping file (either as a primary or backup configuration). This winds up in the classloader which in the case of an applet is the AppletClassLoader. The class loader first tries to find the requested resource locally and if it's not there it tries to load it from the codebase. In our case it's not on the server so you get a 404 in the logs. The fact that the resource isn't available anywhere isn't a problem but the fact that this information isn't cached anywhere means that the next time something looks for the mapping file the whole thing is going to happen all over again.

The code in XMLClassMetaInfoManager.getDocument could easily add and check for a key with a null value in the event that it had previously tried to find the mapping and couldn't. This is what it does instead:

	public Document getDocument(Class clazz) {
		if (clazz == null)
			return null;
		Document doc = (Document) documents.get(clazz.getName());
		if (doc != null) {
			return doc;
		}
		String path = '/' + clazz.getName().replace('.', '/') + ".aegis.xml";
		InputStream is = clazz.getResourceAsStream(path);
		if (is == null) {
			log.debug("Mapping file : " + path + " not found.");
			return null;
		}
		log.debug("Found mapping file : " + path);
		try {
			doc = new StaxBuilder().build(is);
			documents.put(clazz.getName(), doc);
			return doc;
		} catch (XMLStreamException e) {
			log.error("Error loading file " + path, e);
		}
		return null;
	}

(Incidentally, I'm really liking the syntaxhighlighter WordPress plugin.) The AppletClassLoader is off the hook because you can easily say that its behavior is a feature, not a bug.

The Quick Fix

Like I said, XFire is in maintenance mode. The prospect of getting the source code, creating a vendor branch in the SCM, fixing / testing, and finally internally hosting the patched artifact didn't appeal to me on the limited time scale on which I was working. The quick fix (that apparently only works in JRE 1.6) is to set the codebase_lookup parameter to false in the applet tag.

Sadly, the 1.6 restriction means I probably won't get to live with this fix for long and will likely wind up either patching XFire or upgrading to CXF. CXF may or may not still have the problem, but at least I'd be writing a patch for an actively developed library.

To further complicate things, there are other incidents of 404s with resource bundles and XML parsers. The nice thing about the codebase_lookup solution is that it "fixes" those issues as well. If I can't rely on JRE 1.6 as a requirement I'll have a lot more work to do than just patching XFire.

Applets. Yay.

Update

It's probably actually XMLTypeCreator that is causing my problem, but the code is pretty much identical. There are also issues with XMLDocumentationBuilder.loadDocument. I don't blame XFire in particular. There are a lot of projects out there with code that uses getResourceAsStream knowing that it could fail and then not caching the fact that it failed. They then repeat the operation multiple times. This assumes that getResourceAsStream and other such operations are very cheap performance-wise or that it's not worth the effort or memory to cache failed attempts. I'm not sure I buy it in the first place but it certainly goes out the window when you're in an applet and every such call that fails locally then gets executed across the network back to your codebase.

I'm pretty disappointed with the online listing of NBC's television coverage of the Olympics. It's not a particularly useful format. I poked around a little and found some crappy reformat attempts and even thought briefly of reformatting it myself. Before committing to actual work I decided to look at Google Calendar to see if anyone had thrown together a public calendar of the televised events. Lo and behold there a metric crapload of calendars over there. I'd link to them here but you need to have a (free) account to see them. Here's a small example (that won't make any sense after the Olympics are over):

Plus, having it as a calendar on Google will allow me to search it and, if I copy an event to one of my own calendars, even send SMS reminders of events. Woot! It would have been a nice feature if I could copy the whole calendar to make it editable by me instead of having to copy appointment by appointment. With a bulk copy I might have gone through and deleted anything I wasn't interested in ahead of time (that would be none of them–it's the Olympics, chump). It would also have simplified the multi-step process of creating an SMS reminder. Oh well. It's still very useful.

Within the product I work on we use OpenAds, which has been renamed to OpenX. Apparently OpenAds may have been somewhat of a pain to get installed since everyone cringed when I suggested we should make a test server for our ads. Since I'm always a fan of the latest and greatest, I decided to install OpenX 2.6.0 in a VM and see what the differences were. The great news is that it was a breeze to install and whatever was changed doesn't affect our product at all.

The really odd thing was that once I got it installed I couldn't see any images from its admin web site in Firefox (but I could in Internet Explorer). I searched around and found out from the internet just how stupid I am. I was trying to see images on an internally deployed online advertising server while I had AdBlock installed. The sad thing is this isn't the first time something like this has happened to me. Perhaps the third time will be the charm.

I found this problem and solution and then backtracked to some Google search results to make sure I wasn't being completely stupid. The issue was that the equals method on one of my objects was returning false when I knew the two objects were equal. A quick trip to the debugger showed me that the method was using some of the private member variables of the provided object to determine equality. Something along the lines of if(this.name.equals(other.name)) (no that's not the whole method, yes I know how to write an equals method, etc). The problem is that if the object provided to the method is proxied by something like a CGLIBLazyInitializer, those private fields may not have values. So, you're best off using the getter method. Also, as another blog and its comments point out, you need to be careful when calling certain methods on the provided object. One example is that you can't rely on using the getClass() method and should use instanceof instead.

As the other post also points out, this is one of the possible odd side effects from using Hibernate, even though it's really an issue with the CGLIB dynamic proxy class(es).