Long answer: Read on, code fiends. Read on.

Tonight I decided to earn some HusbandPoints™ by helping my wife get a large number of tagged photos off Picasa for a project that she’s working on. Downloading them by hand would’ve been a pain in the ass timewise and also would’ve been a big pain opportunity-cost wise as well, since she would’ve had to take time out from the main body of the project (a homemade cookbook for a friend’s wedding) to do a dumb photo-by-photo clickfest through the entire large Picasa album she’d assembled with her friends. Plus it gave me a reason to mess around w/ the Google APIs some — knowledge that would almost certainly come in handy later.

Now, the easiest way to go about scripting this w/ Ruby involves using open-uri to pass in the authorization token from Google into every request, per their ClientLogin authentication method. You do that with a piece of code like this:

Here we’re getting the content from Google (which will come as an Atom feed, as all of their various pieces of content do) and then parsing the result with Hpricot. We pass the http_header Hash to OpenURI’s open method to specify a set of HTTP header variables. This is supposed to be easy, but tonight it wasn’t, and my wife was treated to the inelegant sounds of me cursing at my laptop screen for 10 or 15 minutes until I figured out what the problem was.

‘feed://’ don’t go ’round here

The problem turmed out to be the “feed://” pseudo protocol. Safari likes it (because it fancies itself a feed reader), and decided to make the RSS link provided by Google for the tag set my wife wanted to download into a “feed://” URL. Of course, there’s no such protocol, and “feed://” itself is a pretty lame. People have been bitching about its lameness for a long, long time. It’s almost as lame as me not catching it.

But the lamest thing of all (which was causing the cursing) is how OpenURI handles this:

This is the same thing you get when you try to use open on a URL with a hash of header arguments and you’ve forgotten to require the OpenURI library in the first place.

The problem here seems to be with this part:

It’s not calling the part you might think — the piece where it asks if the name can be converted to a string and if it conforms to a loose URI regex pattern. It’s instead calling it with the original, version of open, the one that the Kernel class provides so you can easily open files and URLs (but without all the tasty options given you by OpenURI). This error gets thrown by Kernel when you try to use open outside the context of OpenURI (as this guy points out).

Since we can tell that a URL that starts with “feed://” should pass the first of the two tests in the “elsif” clause (the regex pattern), that means that it’s not passing some part of the the URI.parse test. Here’s what that URI.parse method looks like:

No great clues there. But if you run through the code in the OpenURI#open method’s elsif clause, it turns out that if you parse the offending “feed://”-based URI, you don’t get a “URI::HTTP” object. You get a “URI::Generic” object, which doesn’t respond to open. Obviously, the library doesn’t support this kind of URL, and if it weren’t overriding a Kernel method, it’d probably say so, but it can’t make assumptions about what you’re trying to do with open, so it instead falls through to the call to the overridden Kernel#open and you get the same error you’d get if you never used “require ‘open-uri’” in the first place.

Lesson learned, boys and girls — pseudo protocols aren’t supported by much at all other than self-important feed reading software.

Thanks to the Gimite Google Spreadsheet library for inspiration on the auth code

Say you start the debugger here:

Then you want to take a look at the “xml_obj” variable. If this were (for instance) Python’s pdb, we’d just type “xml_obj” and hit return and be done with it. Not so in ruby-debug:

This is because with default settings, the debugger needs a keyword (’p') to get it to actually evaluate your statement as Ruby and not a command to the debugger itself:

That gets really tedious, really fast. The debugger’s help function (’help p’) will helpfully tell you that this is because the “autoeval” option is not enabled. If you’re thick like me, you won’t see this and you’ll just continue doing “p <whatever>” until you get so frustrated you drop what you’re doing one day and go hunt down a fix.

Here is that fix from inside your code:

You can also do this inside the debugger:

Rails already does it via Rack middleware

You might be wondering why the debugging experience is different in Rails than in Ruby you’ve written elsewhere. I did too — remembering that this ‘p’ business isn’t necessary when I run the debugger as an option when I start up Mongrel in a Rails app. So I went digging for the code that Rails uses to set this stuff up. Those settings come from a piece of Rack middleware that lives in lib/rails/rack/debugger.rb. Here’s the class definition:

For more info on how Rails uses Rack, this is a pretty handy page from the Rails guides.

OK so first things first. This thing I’ve messed around with recently isn’t a service — it’s a client. We must make sure that we get the nomenclature correct so that we’ll know what we’re all talking about. The thing on the other end is the service. I was referring to it as a service in kind of a generic way, but it wasn’t too helpful because this is most definitely properly called a client.

So what kinds of changes have I made recently? Well first off, I decided that this snippet from my last post was just painful to the point of utter crappiness:

Not only is it ugly, it’s long-winded and hard to remember. As soon as I had another person working with this code at work, I felt the sting of shame and knew I had to re-factor. I decided that there was no reason not to take the opportunity to use one of Ruby’s vaunted metaprogramming features to try and make this a little shorter. Now it works like this:

Maybe #method_missing?

So how did I accomplish that? Simple, I used Ruby’s method_missing to enable arbitrary method calls to be made on the WebEx::Request object.

method_missing is a hook method that gets called when there’s no definition found for a method called on an object — i.e. the method is missing. When that happens, the Ruby interpreter automatically calls an instance method called method_missing. Usually, that’s not defined, so the interpreter will raise an exception, but if method_missing is defined, the interpreter does whatever it says to. Here, I’m defining it like this:

In this case, self is of course an object of the WebEx::Request class. When the interpreter tries to call #create_attendee on an instance of that class, it can’t find the method and method_missing is called. My implementation of method_missing assumes that the “method” being called on the WebEx::Request object is a real class method of the XMLObject class (which is where all my actual API calls are), so it calls it there, passing along the arguments it received in the first place.

Of course, if no such class method exists, the interpreter throws an exception like it should. But for legitimate calls, I’ve shortened my code by quite a bit.

<gasp> isn’t it bad to use method_missing?

Some might argue that this is a cavalier usage of method_missing. I don’t really think so. The entirety of my request class is only like three methods other than this. It exists only to make a request and process the result, but it does that dumbly because it actually uses XMLObjects for the real meat of both. WebEx::Request is just a dumb object acting on stuff at the direction of the WebEx::XMLObjects that it calls through XMLObject class methods. If the Request class had a lot more methods, using method_missing might have been an irresponsible design choice. But since the notion of the request and the thing being requested are necessarily intertwined, this approach made a lot of sense to me because it let me join the two classes up for usage (WebEx::Request and WebEx::XMLObject), improving readability and reducing verbosity but still letting me keep the two classes usefully separate in the code.

method_missing is one of those things that you’ll sometimes hear people talking about Ruby referring to as magic. Like other, more experienced folk, I hate the usage of the word “magic” to describe things that happen in Ruby. The concepts behind metaprogramming aren’t that difficult, and as Mr. Giles says in his blog post, metaprogramming is just programming. If not for method_missing, you wouldn’t have cool stuff like Builder and RSpec. Or maybe you would, but not as elegantly. method_missing underlies much of the powerful Doman-Specific Language stuff that is so popular and useful in Ruby, so as my dad used to say: “it’s not bad or scary — it’s just different.”

(Gist of the full module)

Ruby modules offer the programmer the ability to group classes together into like structures and to control namespaces for methods. They also function as a sort of pseudo-class, in that you can define “module methods” inside the module that feel like class methods. If you have a module Foo and a method self.bar, you can call it with Foo.bar.

That’s what I ended up doing with all the helper methods I’ve got in this new class definition — WebEx.time_from_string, WebEx.filter_by_session_name, etc.

I also abstracted the things I’d need into several classes. Now the XML client has:

• An Attendee class to describe the most common type of API data we need to create and manipulate
• An XMLObject class which describes a generic object containing XML and the recipe necessary to process it into Ruby data structures (implemented as a Proc object), as well as class methods for generating various types of such objects
• A Request class which is responsible for sending an XMLObject to the WebEX API, sending the retrieved data through the object’s processor, and exposing the processed data as well as the processed result strings from the API request.

Here’s how you’d create an attendee (I’m stipulating the existence of an initialized Attendee object — @attendee)

Gist of the new WebEx module

This is necessarily kind of a spare time project. It’s something for work, but it does what it’s meant to just fine right now, so re-factoring it is something I’m doing in spare cycles. However, I’m gratified that continued refactoring has brought me back to the original idea of having a simple class with class methods that mirror the API’s methods. I had to trip through the things that brought me to make the other classes and gather everything together into a module, but it was a helpful (if stumbling) process to go through.

Re-Arranging the WebEx Class

I figured it wouldn’t be long before I was back at the drawing board on my main architecture, and I was right. The earlier one I described turned out to be over-abstracted and hard to test.

It all started with my feeling that this looks really elegant:

But in practice it turns out to be a little strange. “Attendee” and “Event” were two classes with no attributes of their own, and none of my code ever instantiated objects of these classes. These are two pretty obvious signs over an over-abstracted implementation. I’d been thinking that I’d write logic later which would (for example) instantiate objects of the Event class inside Event.list’s processor, but as I got more and more into the implementation, it just didn’t seem like I was going to need to mess with WebEx’s return values as discreetly defined objects. After all, I already had Events and Attendees represented as hashes in an array, which was working just fine for this first use case and the ones I could see on the horizon. Having separate classes for Event and Attendee would give me maximum extensibility, but at the cost of having pieces of overlong, over-organized code with no (current) purpose.

So I moved the Event class’s code into the WebEx class. Same with Attendee — now the WebEx class’s code looks like this:

(Gist of the WebEx class)

As you can see, everything is now an instance method of the WebEx object. This means that the syntax for getting a list of Events is now:

This still looks a little weird to me. I had been thinking that I should make WebEx#request into a class method, so as to have:

But that would mean having WebEx.request instantiate and return a new object of the WebEx class. There’s nothing wrong with that, but given the fact that another WebEx object already needs to be created in order to call one of its instance methods (event_list), it felt like a case where two objects of the same class which weren’t being used at all in the same way. Because of that awkwardness, I decided to live with the clunky-but-servicable all-instance-method approach. After all – there’s a good chance that I’ll refactor it yet again as I go… :-p

Testing

I’m embarrassed to not have spotted this earlier: the class as it had been written before was very hard to test for a couple major reasons:

• There was no way to override the XML attribute of one of the WebExmlObjects being returned by class methods Event.list and Attendee.list_for_meeting
• The HTTP request happened within the WebEx.request method, making it difficult to stub the HTTP request’s response, which had to happen in order to ensure that calling that method during testing didn’t involve net calls.

I solved each of these easily enough: I abstracted the HTTP request into its own method and I added a “payload” argument to each method that returned a WebExmlObject so that I could override its request XML.

After that, it was time to set up some fixtures. I created directories for “request” and “response” in my fixtures dir and added files containing the well-formed XML samples I got from the WebEx docs. Then I wrote methods for opening/reading each of them in my WebExSpecHelper module (this testing is all in RSpec). Below is a test that ensures that WebEx#request is calling WebEx#request_post:

@w is the instance variable that is created before every spec, and lst_summary_event_response is the name of the spec helper method that returns the fixture of that XML response. There’s no particular reason I called this one as opposed to an attendee-related method – I just needed to assert that the call would happen and then stipulate the response it would give, so any of my helper methods would do.

Here’s that helper method doing what it’s meant to:

There are three events in the fixture, so the length should be three when nothing is passed to time limit. All the fixture data from WebEx was in the past, so I altered the dates in there to have one event in the past, one in the future, and one in the future at a more distant date. Here’s what happens when you pass a time limit past that first (earlier) future date

Only one result gets returned, because the fixture only has one event listing which has a start date after the date given.

Next Steps

So far, my unit tests have covered very little – basically just the processor portion of a WebExmlObject. For full coverage, I’ll need to test the ml attribute which means validating the generated XML against the XML Schema Definitions(XSDs) WebEx provides with their API docs. Ruby doesn’t provide have any all-native tools for doing validation of XML against a given XSD, but the libxml library (which is distributed as a gem and gets its power from C-bindings it compiles at install time) will let you pass in a schema as a string and then validate against it.

That felt the most natural to me. The idea is that you have a WebEx class that’s basically responsible for initiating the connection and handling the boilerplate security stuff that’s going to be at the top of any XML API request. The question then became how to structure the Event class so that it could do two things at once: pass in the XML for the request and process the XML from the response.

WebExmlObject and its Subclasses

I decided to define a WebExmlObject class with “xml” and “processor” attributes:

Then I could define subclasses for each of the major types of objects I’d be pulling from WebEx (Attendee and Event, to start). For right now, it doesn’t matter that the subclasses themselves don’t define any new attributes or instance methods – it’s worth it to me to do it this way because I believe that they probably will have to in the future. And in any case, I’m a sucker for aesthetics and simplicity, and I wanted the readability and elegance you get from the class method approach I outlined above. Here’s the Event class and its single class method:

(Event class Gist)

Notice that this method is basically in two halves: the first half creates the XML and the second creates the processor as a Proc object. After creating the XML (using the super-handy Builder library), I create a new Event object and load the XML into it. Then I create a Proc and set it to the event object’s :processor attribute. The processor assumes one argument will be passed to it – the XML response as a REXML document – and it returns an array of hashes, each one representing an event. Only the stuff I care about is in each event hash, an easily extendable list represented by the event_keys variable.

The WebEx Class

Next up is the implementation of the request part. The WebEx class has two instance methods: request and response. Request takes a WebExmlObject, adds its XML in as the body of the request, makes the request, parses the response, and returns an object of class WebEx. Response holds the response header. Here’s the full implementation of the class:

(WebEx class Gist)

Putting it Together

I’m still working on the integration part of this, but it’s happening in Sinatra, so I can easily show the short piece that renders the form:

It’s obviously still a work in progress, but you can see how the XML-to-object abstraction feels pretty well hidden, and the class structure provides the level of elegance and terseness that I was hoping for. If you’re unfamiliar with the way Sinatra works, the above is called when an HTTP GET request is made to the URL “<WEBROOT>/online_demo”. Once the request is made and parsed into the events variable, I create the @events instance variable to hold the actual events array. The last thing I do is call the ERb template “online_demo”. Instance variables created in this block are available to the template in much the same way as the controller/view relationship works in Rails. The view is responsible for iterating over the array and inserting the proper attributes into the proper places:

I’ll post more about this as I move forward, just to document the process of learning how to structure this stuff. I’m still not entirely satisfied with the semantics of the WebEx class’s methods – especially since calling request will set up your object with the data you need and response only gives you access to the header of the response. I pacify myself by reasoning that since one is a class method and one is an instance method, there’s no philosophical/structural problem, but something about it still bugs me.

Big thanks to bona fide code wizard Collin VanDyck for suggestions in this process – as always, I’m proud to be able to call CCV my homie.

Sound familiar?

I was in this boat until I found out about the profiling command:

The double “spec” here can be a bit confusing. The first is the command, and the second is the directory (RAILS_ROOT/spec) that you’re running it on. The -f is the format flag. You can see a list of all formats with:

Basically you’re just passing the “profile” argument to the formatting command. It gives you a readout of the 10 most time-intensive specs at the top and then a nicely verbose list of all pending and failed tests. Using this list, I was able to go straight to the most obnoxious offenders and eliminate the external dependencies (these specs happened to be fetching feeds) that were slowing my test suite down. In less than 15 minutes, I had chopped my execution time down to 10% of what it was before I started.

Huzzah!

• Everyone calls it autotest, but it’s actually ZenTest that you need to install. ZenTest is a larger testing package, of which autotest is merely one part.
• Autotest runs as a separate process (which you should background — the ‘&’ is your friend!) and watches your Rails dir after you have it turned on. This last bit took me some time to figure out, as the otherwise excellent RSpec Peepcode screencast completely glosses over this point. I kept waiting for the magic to happen when I hit “save” in TextMate, but I didn’t have the process running from the command line. Magic doesn’t work. I should stop expecting it to.
• Do yourself a favor and vastly simplify your autotest–>Growl connection by using Collin VanDyck’s GrowlGlue gem as the basis for your .autotest config file.

As of right now, I’m sitting here with just the basic GG config. I’m sure I’ll expand/customize tomorrow, but I’m just happy for now that I was able to get a working setup with just one line of code, as the normal way to get the connection going involves a decent amount of crap. After you get that going, GrowlGlue lets you set up all sorts of customization of the notifications — including custom messages, custom notification icons, and even speaking your test results through OS X’s built-in text-to-speech support — in nice, neat Ruby statements.

It’s nice to have something get in my face about my tests, and it’s also nice to have something that automatically makes you want to get your tests passing and running faster. Autotest provides you with the nagging voice that says “get your tests running right, holmes!” It’s good to strike this longstanding to-do off my list.

Late update: Turns out that running autotest in the background isn’t really ideal since you might sometimes want to cancel a running suite or check out the time elapsed since you started a given test set. What I’m looking for now is a way to see how long it takes any individual test to run so that I can find the offenders that are kicking up the aggregate time.