Running as Root

I ran out of space on my home file server about 4 or 5 months ago. Since drives have become so cheap I went out and got a pair of 500GB Seagates with the intention of running a software RAID (level 1, two drive mirroring–let's have the 4/5 debate some other time). As long as you're not doing a software RAID using Windows, my understanding is that it'll be stable and reasonably performant. Besides, I'm not an IT shop, just a home network and I also didn't feel like purchasing and dicking around with new hardware for this project. Of course, being lazy I just now got around to actually doing the work.

My first idea was to just put good old Ubuntu server on there and set up Samba, SSH, etc myself. It's not all that difficult to do. Then, I ran across a project calls FreeNAS. It's a FreeBSD-ish network attached storage project that borrows some of the configuration pages / scripts from m0n0wall (another project I started using instead of Freesco or Coyote Linux for my routing and firewall needs).

After deciding to give FreeNAS a try I pulled down the live CD, currently version 0.685RC1, and burned a CD. I slapped the drives into the old box (400Mhz Pentium leftover), threw in a CD and booted up. Running in this configuration I would boot from the CD each time, read the specific configuration from a floppy, and use the full space of both drives for data. It took a few minutes to figure out the correct order of doing things (add drive to configuration, format drive, create raid volume, format raid, create mount point, configure CIFS, configure SSH, add users and groups) but the whole thing took well under an hour. You have the option to use the web interface to configure everything, back up your config floppy, and even add and rebuild new drives in the RAID.

Once I was done with all of that I had a file server I could access from both Windows and Linux. I moved some files out there and played around testing the performance. Everything seems good. Yay. Finally a home IT project that wasn't a huge pain in the ass. If nothing breaks I'm a convert. I'll just have to move over the old files at some point and I'll be fully done.

In the Future, Everything Will Be Glue

The entry title is from Weird Science, by the way:

Gary: I was crazy for this little eighth grade bitch.
Wyatt: Crazy, Insane!
Gary: I was dedicated to this girl, I called her every damn night!
Old Pimp Dude in the Bar: You called her every night? On the telephone?
Gary: On the telephone? What's he mean on the telephone, course it was on the telephone!

In my work life, the team I'm on has been working for quite some time fixing the architecture for an existing product. One of the things we've been doing while we've been fixing things is to add a REST API. You can save the REST vs WS-* debate for some other time and place.

Meet the New Product, Same as the Old Product

Unfortunately, there was a distinct lack of excitement about the new stuff we'd been doing from other areas of the company. Programmers seem to immediately "get" why exposing all of the CRUD functionality of your system through a web API is the sheer sweetness in terms of application integration. Sales and executive staff don't always see the possibilities. They saw the new version as architecture changes that improved performance and scalability while maintaining functional parity with the shipping version. That didn't scream sexy.

The programmers, ever fearing being a cost center with a very killable project, decided to start creating proof of concept integration examples that would create internal demand for the nearly finished version.

Pretty State Machine

As background, our system has an operational state that changes the way almost everything behaves. The change in this state is typically user driven. Someone using the software would decide to change the operational state due to an external event such as a Martian invasion. The software has a web interface but we had the idea that it'd be much cooler to change the operational state via a telephone.

We can change our operational state via the REST API by putting (as in HTTP PUT) an XML document representing the system status to the server. The system status has an element that represents the operational state. Now, our IT guy is always telling me the crazy stuff he can do with his home VOIP system. He runs a virtual machine with Asterisk and other assorted software an combination with an IP phone. I ran the idea of executing a shell script from a phone menu by him and he assured me it should be easy to do.

I was working under a deadline (of course) so I created three versions of the XML (one for each state I would be changing to) and got a copy of cURL installed in the Asterisk VM to PUT each document to our server, hopefully in response to someone pressing a button on a phone. After hitting a very serious dead end and wasting several hours trying to get an IVR working (that's what phone people call those voice menus) I finally found an example of an Asterisk iTunes controller. To use it you dial an extension, hear a beep, then press a button to do something. The script in Asterisk then calls a shell script in response to your button press. Five minutes after finding it I had a working example of changing operational state in our product via a telephone (actually a softphone but who's really keeping score at this point). Booyah! Sure it only took me 10 hours discover how to do an hour or so worth of work but then that's the plight of the knowledge worker.

Demo Day

The next day we invited additional people to our iteration demo meeting (really it's just the people that were supposed to be going all along but weren't) and showed the phone demo as our finale. Suddenly, people seemed as interested in our new version of the product as the developers are. There was a palpable sense of excitement. There were tears of joy. The developers were hoisted up onto the shoulders of sales and management and carried out into the streets (think about the scene in Dragon: The Bruce Lee Story).

I'm exaggerating a little bit perhaps. Of course there are all sorts of negatives preceding and following this event, but can't we just bask in the glory of this one little success for just a few moments? On the telephone.

How I waste my free time

In between trying to get a simple Arduino project thrown together and farting around with the Yahoo! UI Library, I took a timeout to play around with Firefox add-on (back in my day they were extensions) named Stylish.

The subject came up on Jyte that black text on a white background wasn't the friendliest design for some people. Unfortunately, most sites don't support user specific skins so if the site's design causes you problems, from a vision related impairment for instance, then you're just shit out of luck. That's where Stylish comes in. It lets you create custom CSS overrides on a per site basis.

Can you see me now?

Yet another thing to do when you're bored: see if you can completely change the color scheme of a site to make it hideously ugly for no good reason. It took a little work, but I managed to make a really nasty looking blue and purple on black style for Jyte.

Most of the work was just finding all the nasty little inherited background colors, background images, etc. One interesting problem came up because Jyte has a set of menus that are images of text. They have a white highlight built into them and look like complete crap on the new black background. I thought it wouldn't be a problem to turn links with images into links with the image alt attribute as the link text and then hide the image.

Let the CSS fun begin

I thought maybe some sweet CSS action like this would work:

.nav_item a img:before {
  content:attr(alt);
}

.nav_item a img {
  visibility:hidden;
  width:0;
}

Oh, how lovely that would be. Apparently it'll work in some builds of Firefox but got turned off in later versions. I think it was a feature request specifically to make my life more difficult. :before and :after pseudo elements for other elements work fine, just not images.

So, with a lot of trial and error I wound up with the less than ideal:

.nav_item a[href="/home"]:before {
  content:"Home";
}

.nav_item a[href="/home"] img {
  visibility:hidden;
  width:0;
}

Since I no longer had access to the alt attribute on the image I had to manually insert the correct link text for each menu item. I did this for each menu by using an attribute selector to find each link by its href attribute, then hard coded the content to the correct text and finally hid the image.

You nasty, nasty boy

Oh, it's nasty all right, but it works. The only problem is I was trying to make the CSS overrides generic and there's a dynamic menu that includes the user name. Luckily there's only one such dynamic menu so I just made an entry to handle all such menus as the dynamic one and let the more specific rules clean up after the fact. Ta da! A custom, site specific CSS override that no one will ever use because it looks like it was beaten with a bag of hammers and makes your eyes hurt to read.

The interesting thing is that there are now ways for users to make the sites they frequent more appealing and possibly more accessible via add-ons like Stylish and Greasemonkey. It's not ideal for non-technical users, but sites like Userscripts and Userstyles can help by acting as repositories with aids created by the more technical members of the community.

It was too simple to do, but here's a Greasemonkey script to add access keys to the claims page on Jyte. The keys are set to "n" and "p" for next and previous respectively. Under Firefox, access keys are accessed using Shit+Alt. So, Shift+Alt+N will take you to the next 10 claims on Jyte while you're on any page like http://jyte.com/claims. Another thing to do when you're bored. Write Greasemonkey scripts that will be made rapidly obsolete whenever a developer has a spare 30 seconds. Whee. As always, please excuse the crudity of this script as JavaScript is not my forte.

Here it is if you care.

There's a video of the complete Steve Jobs / Bill Gates joint interview at the D5 conference. It's long, but full of entertaining stuff, especially when Kara Swisher keeps her mouth shut.

The Dark Ages

Things have been dark here for a little too long because I've been busy living it up on Jyte and Pibb, two sites from JanRain. Jyte is a site that lets you make claims and have people either agree or disagree and discuss. In reality it's a bit more interesting than just that. The thing I like about Jyte is it is very effortless blogging. I can throw out a quick idea and get some feedback and discussion going.

Not so with this blog. I usually try to put a little more effort into the entries which result in somewhere near zero discussion. It's still useful for not only letting people know what I'm working on, but also for keeping notes on things I've found. It's not unusual for me to search my own blog to remember how I did something at some point. My memory leaks too much. Plus, I get to say I have a blog like all those cool kids.

Pibb is a web based messaging / chat application. It's just getting started but is good if for no other reason than it gives the people on Jyte a more direct way to communicate. Both of these use OpenID for authentication which is a very nice idea for a public web site. Those of you that watch my bookmarks (or this blog's feed) probably saw the quick bookmark on Acegi's possible inclusion of OpenID as an authentication provider (along with a couple of other Java OpenID libraries). Muy sexy.

That's Not How We Do Things Around Here

Both Jyte and Pibb, being community oriented sites, have felt some recent growing pains in terms of popularity and new members that immediately try to buck the de facto way of doing things. Tagging on Jyte is a good recent example. You can tag claims on Jyte to make them easier to find. New people inevitably either don't tag their claims or tag them with tags that have nothing at all to do with the claim. The tag centric portion of the community immediately tries to get the author to remedy the situation. Most of the time it ends peacefully, but occasionally people rebel against the very idea of someone trying to tell them what to do.

That's the way the community works. The users of the site give the site an identity. New people that don't agree cause friction. The more interesting thing is to see how the developers of the site try to make the community's customs more formalized in the inherent functionality of the site.

In our tag example, there used to be a "top tags" sidebar. Once a group started regularly abusing tags, their ill fitting tag became prominently placed on the top list. After the group refused to change their behavior the top tags list disappeared and was replaced by a set of commonly used tags. This is interesting in that it tries to provide some sort of structure to an otherwise unstructured mechanism–tagging. The short term solution of dealing with this minor abuse was to change the site's functionality to preserve the way the bulk of the community thought things should work.

The original motivation of the group was to use tags to easily find the claims made by members of the group. The JanRain developers eventually added a "find group's claims" link from the group membership page. Again, the functionality of the site changed based on community feedback. It was all very interesting to see how community (mis)behavior caused features to be added/removed to a public web site.

Have You Seen Goatse?

Our next example, and the subject of our subject, is what transpired on Pibb. Pibb got Dugg. This caused the barely opened site to receive an instant spike in traffic. The result was feedback from the new users ranging anywhere from the positive "this place sucks" to greater levels of naughtiness.

Public web site developers often think about what features would be cool for the community. Wouldn't it be great to be able to easily embed images into a chat using HTML? Wouldn't it be cool if users could create their own threads within a channel so that conversations about very specific topics could easily branch off from the original discussion? Gee, you bet it would.

But, that's where troll scalability (also referred to as cultural scaling) comes into play. When you're building a feature rich site for a well behaved community you don't have a lot to worry about in terms of abuse of features. As a site becomes more popular the number of misbehaving users (we'll use the term troll) increases. They may still be a constant percentage of the community, there are just more of them now.

Admittedly, this isn't a problem I routinely deal with since I develop corporate intranet web based applications. I can pretty much guarantee that the hosting corporation can control its own people. This is a luxury the public web site developer doesn't have.

What happens if users start putting images of goatse or tubgirl (links omitted to protect the innocent) in every place you allow the display of images? You let users create new channels? Great. What if I write a script to create a million channels? I'm not talking about the performance impact. Performance is another issue and is one that most people understand and plan for. The troll problem is one for which very few, if any, public web site developers plan.

Troll control is typically handled after you've gone public and gained some popularity. The solution is usually adding punitive measures to misbehavior and the elimination of features. Your community both wins and loses in this situation. It's great to get rid of the trolls, but where did my much beloved features go? If you're lucky, you lose very few of your treasured community contributors. Some will inevitably leave because they think your site is overrun by undesirables. Some will leave because the loss of functionality is unbearable (spoiled bastards).

So What?

The "so what" moment here is that, as a public web site, you may get one shot at getting popularity escape velocity. Your first large scale public exposure is important. You should work hard to ensure that it isn't a negative experience either to your potential new users or to your existing community.

You designed a site to perform well under load and to work in all the major browsers on all the major operating systems. You spent time tweaking your CSS to make it easy on the eyes. Why didn't you spend time thinking how every one of your cool features could be abused? You need to design your features around the "asshole" user. Don't assume everyone on your site wants to place nice nice. Are you safe from inputing script tags in form fields, SQL injection, omnipresent goatse images, animated GIFs, user created content spam, new user creation scripts, etc?

If any of these successful trolls were smart (I doubt that they are) they would open a business offering their devious skills to public web sites currently under development. If they ever do, I recommend hiring them. You'll save yourself and your community a lot of grief.

I wrote previously that I was playing around with creating a logo in Gimp. I see now the error of my ways. It's just not the right tool for the job. I still love Gimp, I'm just not in love with Gimp. A vector graphics program is what I need. I feel stupid in hindsight, not seeing the relationship was all wrong. This new thing I've got going on has made life richer. Behold, Inkscape! My next true software love.

I won't bother trying to do a full feature list (since I know around three of them), but it's definitely worth you looking into if you want to do something more like graphics / logo creation rather than photo retouching (still love ya, Gimp).

It's got a graphic import / convert to paths piece of functionality that I just can't get over. The laurel leaves in the logo above got lifted from a gif I found on the web. It took somewhere under a minute to import it, convert it, isolate just the leaves, and paste them into their own layer as an object. You can also view the XML version of the "image." All too good to be true? Well, it's also open source. I think I just came a little. Oh, and you can get it for Windows if you're into that sort of BDSM thing.

Someone asked for a Blockbuster version of the Netflix Greasemonkey script I wrote. I signed up for the free two week trial and got it working, I think. It's not very different from the original Netflix version, just some stripping of extra characters like "|WS|" or "|Unrated|" that Blockbuster adds to the title. Feel free to download it from here.

Update: BlockBuster made some minor changes that broke that version of the script. I've made some minor changes to fix things. I updated the link in the post to point to the new one. If you can't be bothered to find that link, you can get it here.

Update: I updated URLs to which the script applies and added quotes around the search title to get more exact matches. As always, you can get it here.

Update: Blockbuster started including the year of the disc in parentheses which would throw off the search so I stripped that information out before hitting the TiVo search site. I also changed the link to the TiVo site so that it would open in a new window/tab. I found myself always shift clicking the link so I just put it in the script.

I wasn't very satisfied with my last attempt at merging Netflix and Tivo. I'm sure it can be done using just Pipes eventually but I think it'll take a while for Pipes to mature enough.

Barring that, I got curious this weekend and decided to see if I could do the same kind of thing directly on the Netflix queue page via a Greasemonkey script. So I wrote a quick little script over the weekend:

The quick overview is that the script takes each movie title in your queue and searches for the text on TiVo's site. If it finds a match it puts a link after the movie to TiVo's search page. I used a link so you can open the matches in other tabs. This is especially handy since it takes a while to search TiVo for all the movies in your queue. I also tried to make the link stand out a little so it would be easy to spot while scrolling through the page. Besides being a slow loader, the other downside is that any match will cause the link to appear. This is a little bit annoying but seems bearable. In theory you would then schedule the recording online via TiVo's site.

If you're interested, go install Greasemonkey followed by this script. Feel free to modify the hell out of it if you like. I realize it's still not a perfect solution, but seems a bit useful for now. Also, please excuse the crudity of the code. Like I said, it was a quickie and Javascript is not my forte.

Update: Here's a Blockbuster version.

Update: I added the quotes to the embedded search. I no longer have a Netflix account, so it's not the easiest thing for me to test out. Let me know if there are any problems. The script is now hosted at userscripts.org.

Brandon had another cool idea which he posted on his blog. In short, he thought it'd be cool to be able to automatically schedule recordings on your DVR for the titles in your Netflix queue. I've been playing around with Yahoo! Pipes recently and this seemed like an ideal candidate project. Pipes is a GUI scripting tool for doing things with RSS feeds, like mashups.

After many false starts I finally arrived at the following:

1. Grab personalized Netflix queue RSS
2. Loop through each feed item
3. Construct a TiVo search URL
4. Grab the results via Feedity, which will make it an RSS feed (Pipes needs that)
5. Display the unique results, which should have been a link to upcoming showings in TiVo's guide for each matching title from your Netflix queue

Sounds good in theory. Unfortunately my final stumbling block was that Netflix currently includes the queue position in your queue RSS and Pipes provides no way to parse out the substring of the DVD title. This title gets sent to TiVo which finds no results. Feedity then includes the "Guru Guide FAQ" in the result set (which it seems to do whenever your search returned no results). Filtering that out would be trivial, by the way.

There seems to be some interest on the Pipes discussion boards for adding additional scripting power to the interface so I may revisit this in the future. Of course you could do this some other way, but I felt like playing with Pipes. In the meantime, here's a view of my final non-working composite pipe:

Update:

I decided to play around with this idea a little more (still using Pipes). If I use a feed that doesn't have numbered items, such as the Netflix recommendations or new releases, then the problem just moves a little upstream. Feedity isn't very good at extracting the relevant information from the TiVo search page. On top of that, most of the other free HTML to RSS hosted solutions have equal problems. So, I'm still thinking about it.