Running as Root

As I mentioned in the last post I'm farting around with applets for work. You may remember that the applet was hammering the server whenever it couldn't find a resource in the jars. Of course, everything the applet needs is already in the jars so if it's not in them then it's not on the server either. It's all because the AppletClassLoader tries to load stuff from the codebase on the server if it fails to load it from the jars.

As part of the fix of setting codebase_lookup to false I did some very quick, unofficial benchmarking. The test environment was an EC2 deployment I've been messing around with–the smallest instance. I timed the startup time of the applet and counted the number of server hits during startup. To further minimize variablility, I did this after the jars had already been cached. This roughly corresponds to the startup time of a visitor to the site that has already successfully launched the applet. The results were as follows:

codebase_lookup – true (default setting)
Startup time: 35 seconds
Server hits: 442

codebase_lookup – false
Startup time: 7 seconds
Server hits: 34

When the the applet does hit the server to look stuff up in the code base, it doesn't just do it at startup. Some libraries that don't cache failed attempts to load a resource keep on hitting the server. In our app it's XFire and a lot of non-existent "aegis.xml" and "doc.xml" files as well as a whole bunch of "BeanInfo.class" attempts thanks to java.beans.Introspector. Each of these attempts takes a tiny bit of time and puts an annoying 404 in the access logs. It's hard to say what the distributed sluggishness of the app will be because of all these little attempts, but it's definitely non zero. It also has an affect on server side scalability when each client is potentially 13 times more chatty with the server than it needs to be.

An additional concern I have is that some of this cavalier attitude toward resource loading in these libraries also happens on the server side. How many failed getResourceAsStream attempts am I not seeing and what impact are they having on overall server performance? At the current traffic levels it's probably insignificant but the idea of that much inefficiency spread out throughout the app kind of bothers me.

XFire Mapping Files

The application I work on uses XFire as a Java SOAP framework (this project has been replaced by CXF which is why this isn't a bug report with a patch) that is launched via Web Start. At some point we decided to allow it to be run as an applet as well. One of the weird things I ran into when we did this was a ton of 404 errors in the access logs on the server. The applet was repeatedly trying to load these "*.aegis.xml" files. These are mapping files used to configure how XFire maps your types to XML. However, you don't necessarily need these files if you elect to configure the mappings another way. XFire apparently still looks for the mapping file to allow you a mechanism for overriding other mapping methods. Everybody still here? Good.

Repeatedly Loading Non-Existent Resources

Since we don't use the mapping files, they're not in any of the jars that would be loaded by the applet. So what happens is XFire calls XMLClassMetaInfoManager.getDocument whenever it needs to figure out how to map one of your objects to XML. The manager then executes a getResourceAsStream to see if you have a mapping file (either as a primary or backup configuration). This winds up in the classloader which in the case of an applet is the AppletClassLoader. The class loader first tries to find the requested resource locally and if it's not there it tries to load it from the codebase. In our case it's not on the server so you get a 404 in the logs. The fact that the resource isn't available anywhere isn't a problem but the fact that this information isn't cached anywhere means that the next time something looks for the mapping file the whole thing is going to happen all over again.

The code in XMLClassMetaInfoManager.getDocument could easily add and check for a key with a null value in the event that it had previously tried to find the mapping and couldn't. This is what it does instead:

	public Document getDocument(Class clazz) {
		if (clazz == null)
			return null;
		Document doc = (Document) documents.get(clazz.getName());
		if (doc != null) {
			return doc;
		}
		String path = '/' + clazz.getName().replace('.', '/') + ".aegis.xml";
		InputStream is = clazz.getResourceAsStream(path);
		if (is == null) {
			log.debug("Mapping file : " + path + " not found.");
			return null;
		}
		log.debug("Found mapping file : " + path);
		try {
			doc = new StaxBuilder().build(is);
			documents.put(clazz.getName(), doc);
			return doc;
		} catch (XMLStreamException e) {
			log.error("Error loading file " + path, e);
		}
		return null;
	}

(Incidentally, I'm really liking the syntaxhighlighter WordPress plugin.) The AppletClassLoader is off the hook because you can easily say that its behavior is a feature, not a bug.

The Quick Fix

Like I said, XFire is in maintenance mode. The prospect of getting the source code, creating a vendor branch in the SCM, fixing / testing, and finally internally hosting the patched artifact didn't appeal to me on the limited time scale on which I was working. The quick fix (that apparently only works in JRE 1.6) is to set the codebase_lookup parameter to false in the applet tag.

Sadly, the 1.6 restriction means I probably won't get to live with this fix for long and will likely wind up either patching XFire or upgrading to CXF. CXF may or may not still have the problem, but at least I'd be writing a patch for an actively developed library.

To further complicate things, there are other incidents of 404s with resource bundles and XML parsers. The nice thing about the codebase_lookup solution is that it "fixes" those issues as well. If I can't rely on JRE 1.6 as a requirement I'll have a lot more work to do than just patching XFire.

Applets. Yay.

Update

It's probably actually XMLTypeCreator that is causing my problem, but the code is pretty much identical. There are also issues with XMLDocumentationBuilder.loadDocument. I don't blame XFire in particular. There are a lot of projects out there with code that uses getResourceAsStream knowing that it could fail and then not caching the fact that it failed. They then repeat the operation multiple times. This assumes that getResourceAsStream and other such operations are very cheap performance-wise or that it's not worth the effort or memory to cache failed attempts. I'm not sure I buy it in the first place but it certainly goes out the window when you're in an applet and every such call that fails locally then gets executed across the network back to your codebase.

I found this problem and solution and then backtracked to some Google search results to make sure I wasn't being completely stupid. The issue was that the equals method on one of my objects was returning false when I knew the two objects were equal. A quick trip to the debugger showed me that the method was using some of the private member variables of the provided object to determine equality. Something along the lines of if(this.name.equals(other.name)) (no that's not the whole method, yes I know how to write an equals method, etc). The problem is that if the object provided to the method is proxied by something like a CGLIBLazyInitializer, those private fields may not have values. So, you're best off using the getter method. Also, as another blog and its comments point out, you need to be careful when calling certain methods on the provided object. One example is that you can't rely on using the getClass() method and should use instanceof instead.

As the other post also points out, this is one of the possible odd side effects from using Hibernate, even though it's really an issue with the CGLIB dynamic proxy class(es).

Today's weird problem had to do with a browser cookie not keeping the value I gave it. The cookie's value is an encrypted string. I noticed in the debugger that what was written out was not what was read in later. The decryption failed because the encrypted value retrieved from the cookie just didn't make sense. Apparently cookies are sent as HTTP headers and can only use US-ASCII characters minus whatever other special characters don't work (you can wade through several RFCs if you like). In my case it was trailing "="s that were being eaten.

The easiest solution (besides just not using those characters) is to encode the values. Something like base64 or using the URL encode/decode utility classes in Java would easily do the trick.

The more interesting thing is that we use other cookies to store all kinds of strings, sometimes internationalized funky character containing strings. Those Unicode characters also don't get handled very well in cookie values so there is a general need to encode/decode them. Maybe it's just safest to always encode on the way out and decode on the way in.

Chalk up another one for things I probably should have known but didn't. Yay!

Here's a quick note on migrating data from MS SQL Server to MySQL. I found out in a white paper entitled A Practical Guide to Migrating From Microsoft SQL Server to MySQL (free registration required) that the MySQL install (on Windows at least) includes a tool called the MySQL Migration Toolkit.

It has a simple wizard that steps you through all of the options and you're ready to migrate. You can even choose to migrate "live" or save the scripts to files. One minor annoyance is that you can't pick the name of the target database/schema. When migrating from a SQL Server database named "blahblah" with default schema of "dbo", the migration tool will create "blahblah_dbo" in MySQL. I thought I would just rename it only to find that support for RENAME DATABASE has been dropped.

Instead, I decided to just do a backup and restore said backup to a new schema. I then ran into a problem with one of our badly designed tables that has large (not huge) amounts of BLOB data. The restore died with an error saying "MySQL Server has gone away." I finally tracked down the issue which can be fixed with a small configuration change. Since I was testing this in MySQL for Windows, I just added max_allowed_packet=12M to the my.ini under the server directory and bounced the service. Everything worked nicely after that. Your size on that configuration option may vary, of course.

Yesterday at work I got a question from a co-worker about doing something under AIX. I don't really work with AIX, but I managed to answer the question. This is for another project that I don't work on. It got me to wondering if I could run AIX in a VM on PC hardware, probably using a PPC emulator. I'll save you the suspense and tell you that I couldn't find a way to do it. There are a couple of projects that are sort of trying to do it but they haven't done it successfully that I could find. Those would be PearPC and QEMU. I'm sure one of these projects will get it working someday, as soon as some really capable programmer wants it badly enough. I briefly thought about suggesting purchasing an RS6000 from eBay but decided it wasn't a project I needed to poke my nose into.

This is not a rant against AIX, although in a world gone mad with a billion distros of Linux, OpenSolaris, and PC hardware that is criminally cheap and available I don't feel the need to brush up on that incredibly rusty skill set. No, this is more about that fact that I don't think I would ever choose to work with any operating system and/or software product that I can't run in a VM on PC hardware–it's just too damn handy these days, especially when it comes to QA. That of course includes any non-hacked up version of MacOS, not that they'll miss my business. Also, sadly, I don't always get to pick what I want to work with. Still, it's definitely something to consider when picking your stack and deployment/hosting environment.

I ran across a post in my RSS feeds today that referenced a paper on Bypassing Web Authentication and Authorization with HTTP Verb Tampering. What an awesome title. I'm immediately adding "verb tampering" to the list of things I randomly exclaim in meetings. The short version of the paper (that's represented pretty well by the other blog post) is:

• some developers secure URLs in their web application by URL and method (POST, GET, etc). Everything else is allowed (for some strange reason)
• some servers when they receive an invalid HTTP method or often HEAD will perform a GET and then just discard the body of the response. This is fine and is part of the RFC apparently, since the headers have to match between the two. I just wasn't aware of the fact
• there are still programmers that have non-idempotent GETs in their applications

The scenario is you find these applications / servers and do something like send a HEAD to the URL "deleteUser?userId=27" and then the server does it, despite the fact that you're not logged in.

I'm amazed that this is a problem, for multiple reasons. Who are these people that still don't understand that you don't use GET to do things like delete records from your system? I'd hate to see what one of those crazy internet spiders could do to these guys.

This is also a reason why I'm a big advocate of pushing your security checks as close to the data as you can comfortably stand. Certainly you should have protection at a service or DAO layer to prevent users with inadequate permissions (unauthenticated users fall into this category) from performing most operations in your system. This is also a good practice to ensure that different types of potential front ends don't accidentally grant access to the wrong users. The URL level of security is just icing and fluffery to make the application a little more user friendly.

Of course, that being said I work on an application that relies on mostly on URL level restrictions (I didn't do it and I'm working on changing it) and, if I remember correctly, so does my favorite Java web stack. I should point out that neither of these suffer from the problem described in the paper.

I'm enamored with the idea of Google Android. I briefly looked at the development kit and was very impressed by it. And then I did nothing with it. Part of the problem is that there is no Android capable phone at the moment. That takes some of the sex appeal out of developing for it. More than that is the fact that I don't have any ideas for applications that I'm all that passionate about.

This weekend I heard that Google acquired Grand Central. They're a company I've never heard of but they've got an impressive list of features. Grand Central is a "web-based voice communications platform". It'll let you do all sorts of cool things with your phones via a single number.

This got me to thinking about cool things you could do with Grand Central and Android. Where I work, we use a product called Contactual to route calls for our support number to any other phone number. You can also use their web interface to place an outgoing call and have the resulting call sent to your phone. This is in place so support technicians don't inadvertently give their personal phone number to a customer via caller ID. Once a customer gets a personal number they think that they have a buddy in technical support that can't help them out whenever they run into a snag, rather than going through the proper channels.

What would be interesting would be to see an Android/Grand Central application that would make a similar scenario as easy as dialing an ordinary number. Perhaps allowing you to select any of your phone numbers from a drop down when making the call.

Many of the other features listed on Grand Central's page also look like they'd be an excellent fit with an Android phone.

• Call Record – Use the phone to signal to the server that you want to record your current call
• Block Callers – Why not use Android to let you mark an incoming call's number (or current call) as a blocked number?
• Call Switch – Use your phone to transfer your current call to any of your other numbers.

All of these things are possible with Grand Central and become that much cooler if you can use your phone to do them more directly. It seems like a great tie in for an upcoming product and a recent acquisition all in the same related problem space. Of course, I'm far too lazy and stupid to code any of it, but it still sounds cool.

I read a post recently about how someone found a password via a search for its MD5 hash on Google. Within the comments someone mentioned a site that had a database and search engine for hashes. The part I found clever was that the database of hashes self populates whenever someone uses the site to calculate a hash off of a string of plain text. I'm sure they populate from another source as well, but the last just struck me as interesting.

In the Future, Everything Will Be Glue

The entry title is from Weird Science, by the way:

Gary: I was crazy for this little eighth grade bitch.
Wyatt: Crazy, Insane!
Gary: I was dedicated to this girl, I called her every damn night!
Old Pimp Dude in the Bar: You called her every night? On the telephone?
Gary: On the telephone? What's he mean on the telephone, course it was on the telephone!

In my work life, the team I'm on has been working for quite some time fixing the architecture for an existing product. One of the things we've been doing while we've been fixing things is to add a REST API. You can save the REST vs WS-* debate for some other time and place.

Meet the New Product, Same as the Old Product

Unfortunately, there was a distinct lack of excitement about the new stuff we'd been doing from other areas of the company. Programmers seem to immediately "get" why exposing all of the CRUD functionality of your system through a web API is the sheer sweetness in terms of application integration. Sales and executive staff don't always see the possibilities. They saw the new version as architecture changes that improved performance and scalability while maintaining functional parity with the shipping version. That didn't scream sexy.

The programmers, ever fearing being a cost center with a very killable project, decided to start creating proof of concept integration examples that would create internal demand for the nearly finished version.

Pretty State Machine

As background, our system has an operational state that changes the way almost everything behaves. The change in this state is typically user driven. Someone using the software would decide to change the operational state due to an external event such as a Martian invasion. The software has a web interface but we had the idea that it'd be much cooler to change the operational state via a telephone.

We can change our operational state via the REST API by putting (as in HTTP PUT) an XML document representing the system status to the server. The system status has an element that represents the operational state. Now, our IT guy is always telling me the crazy stuff he can do with his home VOIP system. He runs a virtual machine with Asterisk and other assorted software an combination with an IP phone. I ran the idea of executing a shell script from a phone menu by him and he assured me it should be easy to do.

I was working under a deadline (of course) so I created three versions of the XML (one for each state I would be changing to) and got a copy of cURL installed in the Asterisk VM to PUT each document to our server, hopefully in response to someone pressing a button on a phone. After hitting a very serious dead end and wasting several hours trying to get an IVR working (that's what phone people call those voice menus) I finally found an example of an Asterisk iTunes controller. To use it you dial an extension, hear a beep, then press a button to do something. The script in Asterisk then calls a shell script in response to your button press. Five minutes after finding it I had a working example of changing operational state in our product via a telephone (actually a softphone but who's really keeping score at this point). Booyah! Sure it only took me 10 hours discover how to do an hour or so worth of work but then that's the plight of the knowledge worker.

Demo Day

The next day we invited additional people to our iteration demo meeting (really it's just the people that were supposed to be going all along but weren't) and showed the phone demo as our finale. Suddenly, people seemed as interested in our new version of the product as the developers are. There was a palpable sense of excitement. There were tears of joy. The developers were hoisted up onto the shoulders of sales and management and carried out into the streets (think about the scene in Dragon: The Bruce Lee Story).

I'm exaggerating a little bit perhaps. Of course there are all sorts of negatives preceding and following this event, but can't we just bask in the glory of this one little success for just a few moments? On the telephone.