Running as Root

As I mentioned in the last post I'm farting around with applets for work. You may remember that the applet was hammering the server whenever it couldn't find a resource in the jars. Of course, everything the applet needs is already in the jars so if it's not in them then it's not on the server either. It's all because the AppletClassLoader tries to load stuff from the codebase on the server if it fails to load it from the jars.

As part of the fix of setting codebase_lookup to false I did some very quick, unofficial benchmarking. The test environment was an EC2 deployment I've been messing around with–the smallest instance. I timed the startup time of the applet and counted the number of server hits during startup. To further minimize variablility, I did this after the jars had already been cached. This roughly corresponds to the startup time of a visitor to the site that has already successfully launched the applet. The results were as follows:

codebase_lookup – true (default setting)
Startup time: 35 seconds
Server hits: 442

codebase_lookup – false
Startup time: 7 seconds
Server hits: 34

When the the applet does hit the server to look stuff up in the code base, it doesn't just do it at startup. Some libraries that don't cache failed attempts to load a resource keep on hitting the server. In our app it's XFire and a lot of non-existent "aegis.xml" and "doc.xml" files as well as a whole bunch of "BeanInfo.class" attempts thanks to java.beans.Introspector. Each of these attempts takes a tiny bit of time and puts an annoying 404 in the access logs. It's hard to say what the distributed sluggishness of the app will be because of all these little attempts, but it's definitely non zero. It also has an affect on server side scalability when each client is potentially 13 times more chatty with the server than it needs to be.

An additional concern I have is that some of this cavalier attitude toward resource loading in these libraries also happens on the server side. How many failed getResourceAsStream attempts am I not seeing and what impact are they having on overall server performance? At the current traffic levels it's probably insignificant but the idea of that much inefficiency spread out throughout the app kind of bothers me.

XFire Mapping Files

The application I work on uses XFire as a Java SOAP framework (this project has been replaced by CXF which is why this isn't a bug report with a patch) that is launched via Web Start. At some point we decided to allow it to be run as an applet as well. One of the weird things I ran into when we did this was a ton of 404 errors in the access logs on the server. The applet was repeatedly trying to load these "*.aegis.xml" files. These are mapping files used to configure how XFire maps your types to XML. However, you don't necessarily need these files if you elect to configure the mappings another way. XFire apparently still looks for the mapping file to allow you a mechanism for overriding other mapping methods. Everybody still here? Good.

Repeatedly Loading Non-Existent Resources

Since we don't use the mapping files, they're not in any of the jars that would be loaded by the applet. So what happens is XFire calls XMLClassMetaInfoManager.getDocument whenever it needs to figure out how to map one of your objects to XML. The manager then executes a getResourceAsStream to see if you have a mapping file (either as a primary or backup configuration). This winds up in the classloader which in the case of an applet is the AppletClassLoader. The class loader first tries to find the requested resource locally and if it's not there it tries to load it from the codebase. In our case it's not on the server so you get a 404 in the logs. The fact that the resource isn't available anywhere isn't a problem but the fact that this information isn't cached anywhere means that the next time something looks for the mapping file the whole thing is going to happen all over again.

The code in XMLClassMetaInfoManager.getDocument could easily add and check for a key with a null value in the event that it had previously tried to find the mapping and couldn't. This is what it does instead:

	public Document getDocument(Class clazz) {
		if (clazz == null)
			return null;
		Document doc = (Document) documents.get(clazz.getName());
		if (doc != null) {
			return doc;
		}
		String path = '/' + clazz.getName().replace('.', '/') + ".aegis.xml";
		InputStream is = clazz.getResourceAsStream(path);
		if (is == null) {
			log.debug("Mapping file : " + path + " not found.");
			return null;
		}
		log.debug("Found mapping file : " + path);
		try {
			doc = new StaxBuilder().build(is);
			documents.put(clazz.getName(), doc);
			return doc;
		} catch (XMLStreamException e) {
			log.error("Error loading file " + path, e);
		}
		return null;
	}

(Incidentally, I'm really liking the syntaxhighlighter WordPress plugin.) The AppletClassLoader is off the hook because you can easily say that its behavior is a feature, not a bug.

The Quick Fix

Like I said, XFire is in maintenance mode. The prospect of getting the source code, creating a vendor branch in the SCM, fixing / testing, and finally internally hosting the patched artifact didn't appeal to me on the limited time scale on which I was working. The quick fix (that apparently only works in JRE 1.6) is to set the codebase_lookup parameter to false in the applet tag.

Sadly, the 1.6 restriction means I probably won't get to live with this fix for long and will likely wind up either patching XFire or upgrading to CXF. CXF may or may not still have the problem, but at least I'd be writing a patch for an actively developed library.

To further complicate things, there are other incidents of 404s with resource bundles and XML parsers. The nice thing about the codebase_lookup solution is that it "fixes" those issues as well. If I can't rely on JRE 1.6 as a requirement I'll have a lot more work to do than just patching XFire.

Applets. Yay.

Update

It's probably actually XMLTypeCreator that is causing my problem, but the code is pretty much identical. There are also issues with XMLDocumentationBuilder.loadDocument. I don't blame XFire in particular. There are a lot of projects out there with code that uses getResourceAsStream knowing that it could fail and then not caching the fact that it failed. They then repeat the operation multiple times. This assumes that getResourceAsStream and other such operations are very cheap performance-wise or that it's not worth the effort or memory to cache failed attempts. I'm not sure I buy it in the first place but it certainly goes out the window when you're in an applet and every such call that fails locally then gets executed across the network back to your codebase.

I found this problem and solution and then backtracked to some Google search results to make sure I wasn't being completely stupid. The issue was that the equals method on one of my objects was returning false when I knew the two objects were equal. A quick trip to the debugger showed me that the method was using some of the private member variables of the provided object to determine equality. Something along the lines of if(this.name.equals(other.name)) (no that's not the whole method, yes I know how to write an equals method, etc). The problem is that if the object provided to the method is proxied by something like a CGLIBLazyInitializer, those private fields may not have values. So, you're best off using the getter method. Also, as another blog and its comments point out, you need to be careful when calling certain methods on the provided object. One example is that you can't rely on using the getClass() method and should use instanceof instead.

As the other post also points out, this is one of the possible odd side effects from using Hibernate, even though it's really an issue with the CGLIB dynamic proxy class(es).

Today's weird problem had to do with a browser cookie not keeping the value I gave it. The cookie's value is an encrypted string. I noticed in the debugger that what was written out was not what was read in later. The decryption failed because the encrypted value retrieved from the cookie just didn't make sense. Apparently cookies are sent as HTTP headers and can only use US-ASCII characters minus whatever other special characters don't work (you can wade through several RFCs if you like). In my case it was trailing "="s that were being eaten.

The easiest solution (besides just not using those characters) is to encode the values. Something like base64 or using the URL encode/decode utility classes in Java would easily do the trick.

The more interesting thing is that we use other cookies to store all kinds of strings, sometimes internationalized funky character containing strings. Those Unicode characters also don't get handled very well in cookie values so there is a general need to encode/decode them. Maybe it's just safest to always encode on the way out and decode on the way in.

Chalk up another one for things I probably should have known but didn't. Yay!

I heard on today's Buzz Out Loud that Vimeo will be banning gaming videos that aren't considered "creative expression." One of the reasons cited had to do with the fact that these videos are "significantly larger and longer than any other genre on Vimeo."

I don't really care one way or another. However, the idea of encouraging your users to behave in a way that you want them to behave is on my mind more and more these days. It sounds from the post as if the concern has to do with the "expense" of uploading / transcoding these videos with the implication that they aren't watched and therefore aren't valuable content. If you want to steer your users toward providing valuable content without needlessly draining your resources on crap, why not create and throttle an upload limit based on the views / ratings of their videos? It sounds easy to do and on the surface appears to reward quality participation.

Here's a quick note on migrating data from MS SQL Server to MySQL. I found out in a white paper entitled A Practical Guide to Migrating From Microsoft SQL Server to MySQL (free registration required) that the MySQL install (on Windows at least) includes a tool called the MySQL Migration Toolkit.

It has a simple wizard that steps you through all of the options and you're ready to migrate. You can even choose to migrate "live" or save the scripts to files. One minor annoyance is that you can't pick the name of the target database/schema. When migrating from a SQL Server database named "blahblah" with default schema of "dbo", the migration tool will create "blahblah_dbo" in MySQL. I thought I would just rename it only to find that support for RENAME DATABASE has been dropped.

Instead, I decided to just do a backup and restore said backup to a new schema. I then ran into a problem with one of our badly designed tables that has large (not huge) amounts of BLOB data. The restore died with an error saying "MySQL Server has gone away." I finally tracked down the issue which can be fixed with a small configuration change. Since I was testing this in MySQL for Windows, I just added max_allowed_packet=12M to the my.ini under the server directory and bounced the service. Everything worked nicely after that. Your size on that configuration option may vary, of course.

I've been listening to back episodes of the Agile Toolkit Podcast. One of the things that really caught my attention was the frequent mentioning of something called CodeGreen Labs. From elsewhere on the internet:

We are dedicated to creating a unique training experience that uses real projects with real deliverables. Labs work on projects that make a difference in the world, benefiting organizations that are working to improve the environment, human rights, social justice, health and economic development. By marrying the efforts of dedicated professionals learning Agile Development techniques with not for profit and research communities that need high quality software, we hope to be a small part of a positive change. Each training class will work on a specific software project for a specific cause.

It's volunteering to work on a real software project that is, hopefully, well run so that you can get real world experience. Sure, they get something out of it, but who cares about the other guy? I imagine they get mainly students. A few of the people in my work circles and I have been talking about something exactly like this. The head of my current company has mentioned to me numerous times that he's seeing disturbingly bad candidates coming to him fresh out of college CS programs. I can't comment as to the overall quality of CS programs around the country, but something like CodeGreen Labs seems like a brilliant idea on many levels. Unfortunately, I was a little disappointed at how unfriendly their home page was:

Yesterday at work I got a question from a co-worker about doing something under AIX. I don't really work with AIX, but I managed to answer the question. This is for another project that I don't work on. It got me to wondering if I could run AIX in a VM on PC hardware, probably using a PPC emulator. I'll save you the suspense and tell you that I couldn't find a way to do it. There are a couple of projects that are sort of trying to do it but they haven't done it successfully that I could find. Those would be PearPC and QEMU. I'm sure one of these projects will get it working someday, as soon as some really capable programmer wants it badly enough. I briefly thought about suggesting purchasing an RS6000 from eBay but decided it wasn't a project I needed to poke my nose into.

This is not a rant against AIX, although in a world gone mad with a billion distros of Linux, OpenSolaris, and PC hardware that is criminally cheap and available I don't feel the need to brush up on that incredibly rusty skill set. No, this is more about that fact that I don't think I would ever choose to work with any operating system and/or software product that I can't run in a VM on PC hardware–it's just too damn handy these days, especially when it comes to QA. That of course includes any non-hacked up version of MacOS, not that they'll miss my business. Also, sadly, I don't always get to pick what I want to work with. Still, it's definitely something to consider when picking your stack and deployment/hosting environment.

A few years back when I was looking for anti-virus software, I read a review that put Trend Micro's PC-cillin as the best available. I bought it and have used it on all of my ever dwindling number of Windows computers ever since. When I got computers for my parents, I put PC-cillin on them. I've always found it to be a good, non-intrusive, non-resource hungry Windows anti-virus product. My last purchase was a three license package that had a rebate that never came. Whatever.

However, their recent shenanigans have me insanely pissed off. I get a particularly spammy looking piece of e-mail on Monday with the subject of "Your Subscription will be renewed in 2 days." The email explains that my 1 year subscription is going to expire but that it'll auto-renew 7 days before expiration at a 10% discount. Yay! While the email does include my name it lacks any other identifying information. As I mouse over one of the links I see that it's pointed to dr.bluehornet.com. In fact, the email has no links pointing to any Trend Micro site. The email even came from trendmicro.cs@digitalriver.com. Surely this is spam so I ignore it.

On Wednesday I get an email saying I've been billed for the renewal. This time the email actually has more personal (and accurate) information, including a serial number. It's still from digitalriver.com but at least it has one link that points to the Trend Micro site. I begin to get very unhappy. Since when is billing my credit card for software an opt out situation? I follow the link (to digitalriver.com) to cancel the order to find that the order number isn't in that system yet. I cancel it using the email and am told that although they managed to bill me very quickly, it may take up to a week to cancel the order. I also use the original email to go to the same site to opt out of the auto-renewal program. Of course, I make sure that I don't have to provide any additional personal information on any of these sites since I'm still somewhat suspicious of things. We'll see if this manages to actually cancel the order. If not I'll take it up with the credit card company.

A wee bit of research shows that this is not a new situation. I'm still pretty pissed not only about the opt out notion of charging me for shit I didn't request but also by the fact that my information has been shared/sold/whatever to a third party. I'm pissed that an email sent on behalf of an anti-virus, anti-phishing, anti-bullshit software company is made to look so goddamn spammy that I don't want to acknowledge it.

Fuck Trend Micro. I won't buy their products any more. I won't recommend their products. I will, as much as I can, campaign against the use of their products. I'll use an inferior product from another company just out of spite. We are no longer happy customer and vendor. Fuck you, your questionable business practices, your sharing of my information, your opt out auto-renewal program, your inability to instantly cancel my unsolicited order, and your choice of partners. Did I mention fuck Trend Micro and PC-cillin? Eat shit, go fuck yourself, rot in hell. There's no fixing the situation, I will not be "talked down", I cannot be recovered as a customer. Good job, assholes.

I ran across a post in my RSS feeds today that referenced a paper on Bypassing Web Authentication and Authorization with HTTP Verb Tampering. What an awesome title. I'm immediately adding "verb tampering" to the list of things I randomly exclaim in meetings. The short version of the paper (that's represented pretty well by the other blog post) is:

• some developers secure URLs in their web application by URL and method (POST, GET, etc). Everything else is allowed (for some strange reason)
• some servers when they receive an invalid HTTP method or often HEAD will perform a GET and then just discard the body of the response. This is fine and is part of the RFC apparently, since the headers have to match between the two. I just wasn't aware of the fact
• there are still programmers that have non-idempotent GETs in their applications

The scenario is you find these applications / servers and do something like send a HEAD to the URL "deleteUser?userId=27" and then the server does it, despite the fact that you're not logged in.

I'm amazed that this is a problem, for multiple reasons. Who are these people that still don't understand that you don't use GET to do things like delete records from your system? I'd hate to see what one of those crazy internet spiders could do to these guys.

This is also a reason why I'm a big advocate of pushing your security checks as close to the data as you can comfortably stand. Certainly you should have protection at a service or DAO layer to prevent users with inadequate permissions (unauthenticated users fall into this category) from performing most operations in your system. This is also a good practice to ensure that different types of potential front ends don't accidentally grant access to the wrong users. The URL level of security is just icing and fluffery to make the application a little more user friendly.

Of course, that being said I work on an application that relies on mostly on URL level restrictions (I didn't do it and I'm working on changing it) and, if I remember correctly, so does my favorite Java web stack. I should point out that neither of these suffer from the problem described in the paper.