Running as Root

XFire Mapping Files

The application I work on uses XFire as a Java SOAP framework (this project has been replaced by CXF which is why this isn't a bug report with a patch) that is launched via Web Start. At some point we decided to allow it to be run as an applet as well. One of the weird things I ran into when we did this was a ton of 404 errors in the access logs on the server. The applet was repeatedly trying to load these "*.aegis.xml" files. These are mapping files used to configure how XFire maps your types to XML. However, you don't necessarily need these files if you elect to configure the mappings another way. XFire apparently still looks for the mapping file to allow you a mechanism for overriding other mapping methods. Everybody still here? Good.

Repeatedly Loading Non-Existent Resources

Since we don't use the mapping files, they're not in any of the jars that would be loaded by the applet. So what happens is XFire calls XMLClassMetaInfoManager.getDocument whenever it needs to figure out how to map one of your objects to XML. The manager then executes a getResourceAsStream to see if you have a mapping file (either as a primary or backup configuration). This winds up in the classloader which in the case of an applet is the AppletClassLoader. The class loader first tries to find the requested resource locally and if it's not there it tries to load it from the codebase. In our case it's not on the server so you get a 404 in the logs. The fact that the resource isn't available anywhere isn't a problem but the fact that this information isn't cached anywhere means that the next time something looks for the mapping file the whole thing is going to happen all over again.

The code in XMLClassMetaInfoManager.getDocument could easily add and check for a key with a null value in the event that it had previously tried to find the mapping and couldn't. This is what it does instead:

	public Document getDocument(Class clazz) {
		if (clazz == null)
			return null;
		Document doc = (Document) documents.get(clazz.getName());
		if (doc != null) {
			return doc;
		}
		String path = '/' + clazz.getName().replace('.', '/') + ".aegis.xml";
		InputStream is = clazz.getResourceAsStream(path);
		if (is == null) {
			log.debug("Mapping file : " + path + " not found.");
			return null;
		}
		log.debug("Found mapping file : " + path);
		try {
			doc = new StaxBuilder().build(is);
			documents.put(clazz.getName(), doc);
			return doc;
		} catch (XMLStreamException e) {
			log.error("Error loading file " + path, e);
		}
		return null;
	}

(Incidentally, I'm really liking the syntaxhighlighter WordPress plugin.) The AppletClassLoader is off the hook because you can easily say that its behavior is a feature, not a bug.

The Quick Fix

Like I said, XFire is in maintenance mode. The prospect of getting the source code, creating a vendor branch in the SCM, fixing / testing, and finally internally hosting the patched artifact didn't appeal to me on the limited time scale on which I was working. The quick fix (that apparently only works in JRE 1.6) is to set the codebase_lookup parameter to false in the applet tag.

Sadly, the 1.6 restriction means I probably won't get to live with this fix for long and will likely wind up either patching XFire or upgrading to CXF. CXF may or may not still have the problem, but at least I'd be writing a patch for an actively developed library.

To further complicate things, there are other incidents of 404s with resource bundles and XML parsers. The nice thing about the codebase_lookup solution is that it "fixes" those issues as well. If I can't rely on JRE 1.6 as a requirement I'll have a lot more work to do than just patching XFire.

Applets. Yay.

Update

It's probably actually XMLTypeCreator that is causing my problem, but the code is pretty much identical. There are also issues with XMLDocumentationBuilder.loadDocument. I don't blame XFire in particular. There are a lot of projects out there with code that uses getResourceAsStream knowing that it could fail and then not caching the fact that it failed. They then repeat the operation multiple times. This assumes that getResourceAsStream and other such operations are very cheap performance-wise or that it's not worth the effort or memory to cache failed attempts. I'm not sure I buy it in the first place but it certainly goes out the window when you're in an applet and every such call that fails locally then gets executed across the network back to your codebase.

Within the product I work on we use OpenAds, which has been renamed to OpenX. Apparently OpenAds may have been somewhat of a pain to get installed since everyone cringed when I suggested we should make a test server for our ads. Since I'm always a fan of the latest and greatest, I decided to install OpenX 2.6.0 in a VM and see what the differences were. The great news is that it was a breeze to install and whatever was changed doesn't affect our product at all.

The really odd thing was that once I got it installed I couldn't see any images from its admin web site in Firefox (but I could in Internet Explorer). I searched around and found out from the internet just how stupid I am. I was trying to see images on an internally deployed online advertising server while I had AdBlock installed. The sad thing is this isn't the first time something like this has happened to me. Perhaps the third time will be the charm.

I found this problem and solution and then backtracked to some Google search results to make sure I wasn't being completely stupid. The issue was that the equals method on one of my objects was returning false when I knew the two objects were equal. A quick trip to the debugger showed me that the method was using some of the private member variables of the provided object to determine equality. Something along the lines of if(this.name.equals(other.name)) (no that's not the whole method, yes I know how to write an equals method, etc). The problem is that if the object provided to the method is proxied by something like a CGLIBLazyInitializer, those private fields may not have values. So, you're best off using the getter method. Also, as another blog and its comments point out, you need to be careful when calling certain methods on the provided object. One example is that you can't rely on using the getClass() method and should use instanceof instead.

As the other post also points out, this is one of the possible odd side effects from using Hibernate, even though it's really an issue with the CGLIB dynamic proxy class(es).

Today's weird problem had to do with a browser cookie not keeping the value I gave it. The cookie's value is an encrypted string. I noticed in the debugger that what was written out was not what was read in later. The decryption failed because the encrypted value retrieved from the cookie just didn't make sense. Apparently cookies are sent as HTTP headers and can only use US-ASCII characters minus whatever other special characters don't work (you can wade through several RFCs if you like). In my case it was trailing "="s that were being eaten.

The easiest solution (besides just not using those characters) is to encode the values. Something like base64 or using the URL encode/decode utility classes in Java would easily do the trick.

The more interesting thing is that we use other cookies to store all kinds of strings, sometimes internationalized funky character containing strings. Those Unicode characters also don't get handled very well in cookie values so there is a general need to encode/decode them. Maybe it's just safest to always encode on the way out and decode on the way in.

Chalk up another one for things I probably should have known but didn't. Yay!

Here's a quick note on migrating data from MS SQL Server to MySQL. I found out in a white paper entitled A Practical Guide to Migrating From Microsoft SQL Server to MySQL (free registration required) that the MySQL install (on Windows at least) includes a tool called the MySQL Migration Toolkit.

It has a simple wizard that steps you through all of the options and you're ready to migrate. You can even choose to migrate "live" or save the scripts to files. One minor annoyance is that you can't pick the name of the target database/schema. When migrating from a SQL Server database named "blahblah" with default schema of "dbo", the migration tool will create "blahblah_dbo" in MySQL. I thought I would just rename it only to find that support for RENAME DATABASE has been dropped.

Instead, I decided to just do a backup and restore said backup to a new schema. I then ran into a problem with one of our badly designed tables that has large (not huge) amounts of BLOB data. The restore died with an error saying "MySQL Server has gone away." I finally tracked down the issue which can be fixed with a small configuration change. Since I was testing this in MySQL for Windows, I just added max_allowed_packet=12M to the my.ini under the server directory and bounced the service. Everything worked nicely after that. Your size on that configuration option may vary, of course.

Yesterday at work I got a question from a co-worker about doing something under AIX. I don't really work with AIX, but I managed to answer the question. This is for another project that I don't work on. It got me to wondering if I could run AIX in a VM on PC hardware, probably using a PPC emulator. I'll save you the suspense and tell you that I couldn't find a way to do it. There are a couple of projects that are sort of trying to do it but they haven't done it successfully that I could find. Those would be PearPC and QEMU. I'm sure one of these projects will get it working someday, as soon as some really capable programmer wants it badly enough. I briefly thought about suggesting purchasing an RS6000 from eBay but decided it wasn't a project I needed to poke my nose into.

This is not a rant against AIX, although in a world gone mad with a billion distros of Linux, OpenSolaris, and PC hardware that is criminally cheap and available I don't feel the need to brush up on that incredibly rusty skill set. No, this is more about that fact that I don't think I would ever choose to work with any operating system and/or software product that I can't run in a VM on PC hardware–it's just too damn handy these days, especially when it comes to QA. That of course includes any non-hacked up version of MacOS, not that they'll miss my business. Also, sadly, I don't always get to pick what I want to work with. Still, it's definitely something to consider when picking your stack and deployment/hosting environment.

I ran across a post in my RSS feeds today that referenced a paper on Bypassing Web Authentication and Authorization with HTTP Verb Tampering. What an awesome title. I'm immediately adding "verb tampering" to the list of things I randomly exclaim in meetings. The short version of the paper (that's represented pretty well by the other blog post) is:

• some developers secure URLs in their web application by URL and method (POST, GET, etc). Everything else is allowed (for some strange reason)
• some servers when they receive an invalid HTTP method or often HEAD will perform a GET and then just discard the body of the response. This is fine and is part of the RFC apparently, since the headers have to match between the two. I just wasn't aware of the fact
• there are still programmers that have non-idempotent GETs in their applications

The scenario is you find these applications / servers and do something like send a HEAD to the URL "deleteUser?userId=27" and then the server does it, despite the fact that you're not logged in.

I'm amazed that this is a problem, for multiple reasons. Who are these people that still don't understand that you don't use GET to do things like delete records from your system? I'd hate to see what one of those crazy internet spiders could do to these guys.

This is also a reason why I'm a big advocate of pushing your security checks as close to the data as you can comfortably stand. Certainly you should have protection at a service or DAO layer to prevent users with inadequate permissions (unauthenticated users fall into this category) from performing most operations in your system. This is also a good practice to ensure that different types of potential front ends don't accidentally grant access to the wrong users. The URL level of security is just icing and fluffery to make the application a little more user friendly.

Of course, that being said I work on an application that relies on mostly on URL level restrictions (I didn't do it and I'm working on changing it) and, if I remember correctly, so does my favorite Java web stack. I should point out that neither of these suffer from the problem described in the paper.

Sure, I could have named the post something like "Laying Pipe" but that's a bit obvious, isn't it? As my two blog readers have no doubt noticed (with some annoyance), I've been playing around with Bitstrips and throwing together the occasional comic (complete with my first comment directed at how shitty my "artwork" is). One problem with Bitstrips (other than the fact they co-own everything you do) is that they don't have an RSS feed for either your comics or your series. I opened a bug on the matter but rather than sit back and wait for it, I decided it might be time to have some more fun with Yahoo! Pipes.

The Plan

Bitstrips provides a link to a pseudo feed for each series. The series ID is passed in as a parameter and you get a paginated view of all of the episodes. I figured I'd just grab that page in pipes, create a title off of the episode title, grab the associated URL for the episode, grab the image of the strip for the episode, and throw the whole thing together to make a feed for my series.

Execution

Numerous problems occurred after I start implementing that simple plan–most of the problems being in the Bitstrips layout. I won't go into detail about it, but here's a summary of the steps I went through to get my feed:

• Get the series "feed" page – http://bitstrips.com/feed.php?feed=s_9713 in this case
• Find the URL of whatever page is associated with the "go to last" button – The feed page outputs the series in order. For the feed, I want the most recent stuff. There is no option to just go to the last page, so I need to find that button, grab its associated URL, and then retrieve that URL as my new starting point.
• Regex some fields – I use the regular expression module in Pipes to pull out values for the title field, the publication date, and the link for the item.
• Clean up the date – The date isn't in the proper format, so I need to use the Date Formatter and re-assign the value to "item.y:published". This is how you get a pubDate into an RSS feed from Pipes (thank you Yahoo! discussion forums). This all happens inside of a loop.
• Sort – I want to include this feed into a spliced super feed, so I want to sort it by publication date. However, the publish date in the comic has no time stamp. Luckily, the URL for the comic uses an auto-incrementing ID, so I just sort by that.
• Get the image for the comic – Now we follow the link that we got earlier that goes to the individual episodes. We do this inside of a loop to get images for each item. After that loop, we use a regular expression to get rid of all the extra HTML around the image so we're just left with a nice img element.
• Ship it! – We now have the title, a link to the comic, and the image of the comic. We're done.

Results

Here's what the pipe looks like from way up here:

You can see the results of the pipe here or even examine the "source code" a little more closely by editing it (requires a Yahoo! login and you can't actually edit my version, so there).

Of course, this is a very brittle and ill-advised way of doing things. As soon as Bitstrips changes their site my pipe will burst into flames and spew all sorts of errors into the feed. There is also probably an easier way to do it either in or out of Yahoo! Pipes, but I had fun doing it this way. I haven't decided if I want to put this feed into a jumbo super feed yet or not, but that is most likely the way I'll do it. For now, subscribe if you want but it may go away or be duplicated in a feed to be named later.

I'm enamored with the idea of Google Android. I briefly looked at the development kit and was very impressed by it. And then I did nothing with it. Part of the problem is that there is no Android capable phone at the moment. That takes some of the sex appeal out of developing for it. More than that is the fact that I don't have any ideas for applications that I'm all that passionate about.

This weekend I heard that Google acquired Grand Central. They're a company I've never heard of but they've got an impressive list of features. Grand Central is a "web-based voice communications platform". It'll let you do all sorts of cool things with your phones via a single number.

This got me to thinking about cool things you could do with Grand Central and Android. Where I work, we use a product called Contactual to route calls for our support number to any other phone number. You can also use their web interface to place an outgoing call and have the resulting call sent to your phone. This is in place so support technicians don't inadvertently give their personal phone number to a customer via caller ID. Once a customer gets a personal number they think that they have a buddy in technical support that can't help them out whenever they run into a snag, rather than going through the proper channels.

What would be interesting would be to see an Android/Grand Central application that would make a similar scenario as easy as dialing an ordinary number. Perhaps allowing you to select any of your phone numbers from a drop down when making the call.

Many of the other features listed on Grand Central's page also look like they'd be an excellent fit with an Android phone.

• Call Record – Use the phone to signal to the server that you want to record your current call
• Block Callers – Why not use Android to let you mark an incoming call's number (or current call) as a blocked number?
• Call Switch – Use your phone to transfer your current call to any of your other numbers.

All of these things are possible with Grand Central and become that much cooler if you can use your phone to do them more directly. It seems like a great tie in for an upcoming product and a recent acquisition all in the same related problem space. Of course, I'm far too lazy and stupid to code any of it, but it still sounds cool.

I bought Lisa a new laptop a couple of months ago and the only OS option was Vista. I can't get her onto the Linux bandwagon unfortunately. Yesterday she was trying to print something only to find out that I hadn't set up her printer on the new laptop.

We have an HP Photosmart 1215 which sadly doesn't have (and apparently never will have) Vista drivers. I tried the XP driver only to find that it didn't quite work. Luckily, I found this post saying that the drivers for the 7200 should work. I tried them and they did. Yay, internet!

I might point out that I had no problem getting the printer working with Linux–you know, the hard to use operating system. Gee, I don't know why people are hating Vista so darn much.