Sorting a Rails log file by SQL duration

Rails’ ActiveRecord logger writes log files like:

Post Load (735.8ms) SELECT posts.* FROM posts where post.title = 'foo'

You may want to know the longest SQL queries for performance optimsation purposes, and general troubleshooting. To list recent queries in order of duration, with longest queries shown last, use this:

head -10000 development.log | grep '([0-9.]+ms)' | sed 's/.(([[:digit:].]+)ms./1ms &/g' | sort -n

(The sed expression was a little more work than I’d bargained for as sed regular expressions are always lazy; even with GNU/Posix extensions, non-lazy just doesn’t exist.)

The Weirdness of Ajax Redirects: Some Workarounds

I’ve been dealing with the situation when your server redirects after a DELETE. Which is actually a fairly common use case. User deletes a blog post, redirect them to the index of all posts. The article below has some Rails workarounds, but is relevant to anyone using XHR and perhaps HTML5 History, ie many single-page web apps.

It’s important stuff to know if you’re HTML5ing in earnest, because there was a recent change to Chrome which makes it standard-compliant, but fundamentally different to other browsers and also makes it handle DELETE and other verbs different to how it handles POST and GET.

DELETE redirection

This redirection becomes troublesome if you want to do this in a way which will seamlessly support Ajax calls, as well as standard web requests. It’s troublesome because of a lethal combination of two browser facts.

Firstly, XHR automatically follows redirects. Your JavaScript has no way to jump in and veto or alter any redirect that comes from the server. XHR call fires off to server, server responds with 302 status code and a location header, the browser automatically issues a new request to the specified location.

Secondly, a DELETE request, when it’s redirected, will actually cause another DELETE request to the new location. Yes, if you’re redirecting to the index, congratulations … your server now receives a request to DELETE the fraking index. You lose!!! So the very normal thing to do for a non-Ajax app suddenly becomes an epic tragedy. It’s completely unintuitive, but it’s actually the standard! I learned as much by filing an erroneous Chrome bug. Turns out Chrome’s unintuitive behaviour is actually correct and Firefox and Opera were wrong. Even more confusing, I was seeing POST requests being converted to GETs, and it turns out this is also part of the 302 standard – POSTs can be converted to GETs, but not DELETEs (or PUTs etc I assume).

Just now, I made a little workaround which seems to be working nicely in my case. In my AppController, which is my app’s own subclass of ActionController, I simply convert any 302 response (the default) to a 303, for redirect_to calls. I should maybe do this for XHR calls only, to be “pure” when a normal call comes in, but no real difference anyway.

class ApplicationController < ActionController::Base

# see src at https://github.com/rails/rails/blob/master/actionpack/lib/action_controller/metal/redirecting.rb def redirect_to(options = {}, response_status = {}) super(options, response_status) self.status = 303 if self.status == 302 end

end [/ruby]

Finding the new location

While I’m blogging this, let me tell you about another related thing that happens with redirects (this section applies to any type of request – GET, POST, DELETE, whatever). As I said above, XHR will automatically follow requests. The problem is, how do you know where the ultimate response actually came from? XHR will only tell you where the initial request went; it refuses to reveal what it did afterwards. It won’t even tell you if there was any redirect. Knowing the ultimate location is important for HTML5 apps using the History API, because you may want to pushState() to that location. The solution is to have the web service output its own location in a header. Weird that you have to, but gets the job done.

class ApplicationController < ActionController::Base … before_filter proc { |controller| controller.response.headers[‘x-url’] = controller.request.fullpath }

end [/ruby]

Rails Default Date Gotcha

This had me going for a while!

I wanted to default a field to the epoch, i.e. some decades ago, or older. That’s because a batch process will work on the records with the oldest such field, so setting the field to epoch guarantees they’ll get priority initially.

I had a migration like such:

  1. t.datetime :acted_on, :default => 0  ### BAD

After some debugging, I notice FactoryGirl is setting the field to null. The fix is:

  1. t.datetime :acted_on, :default => DateTime.new ### GOOD

Note it’s DateTime.new for epoch, not DateTime.now, which will be the present time. (And you probably wouldn’t want that, because it would be the time you run the migration, not the time a given record is actually created. If you want creation time, use a before_create callback.)

Preventing a Rails (Mongrel App) from Crashing

I’ve had a Rails website which works fine for about 12-18 hours, then starts giving out intermittent 500 errors because the mongrels die.

After searching around, I ended up fixing it on two levels.

(a) Direct solution – Fix MySQL config One reason Mongrels die is MySQL connections not timing out, leading to starvation. Apparently there’s a bug here which means you have to set “ActiveRecord::Base.verification_timeout = 14400” in environment.rb, the figure must be less than MySQL’s interactive timeout, which is 8 hours (28800 secs, so this is half of that). But as this thread points out, it doesn’t seem like that will achieve a whole lot on its own, so there’s also a tiny Strac hack you can include in the Rails code. Basically the hack is to set a reconnect flag when establishing a connection. The code is shown in the aforementioned thread.

(b) Risk mitigation – Automate monitoring and automatically redeploy when it fails I’ve always done silly things with cronjobs to automate redeployment, got the job done okay, but is definitely an admin smell. Nagios seemed too complicated. I just noticed this Monit tool seems to be gaining traction in the Rails community and turned out to be pretty easy to set up. It wakes up every three minutes (by default) and runs a specified command if specified conditions are(n’t) met. I hope Cap or Deprec will introduce support for Monit in the future.

Rails JSON, JSON Rails

<

p style=”clear:both; margin-top: 2em; “> Man, JSON has come out of nowhere! I first came across it amid the Javascript hype in early ’05, now it’s everywhere. Not only in your Ajax apps, but in the On-Demand Javascript APIs of the world.

Sharing an ethic of simplicity with Rails, it’s not surprising these technologies have come closely together, with JSON support now baked into the Rails core. (They would have been even closer together, if only anyone had noticed JSON==YAML early on and merged the two completely.) And yet, the current implementation misses a couple of tricks.

  • Use of “attributes”. According to Rails, a JSON object generally has just a single key, “attributes”. Instead of the simple { :name=>”Pac-Man”, :creator=>”Namco” }, we get (and have to give) { :attributes => { :name=>”Pac-Man”, :creator=>”Namco” }}. That’s not DRY. Not that it matters, but the reason for this is because ActiveRecords internally store persistable data fields in a hashmap called “attributes”. The JSON serialization is therefore faithful to Rails’ internal implementation, while being unfaithful to intuition. Maybe you could argue for “attributes” on the basis that you might one day have a transient attribute with the same name as a persistent attribute, in which case you’d get a clash. But Convention Over Configuration rules this argument out; you can always override JSON serialization behavior in boundary situations.
  • Child attributes This is probably an issue with XML as well. Basically, there is no attention paid to child attributes, i.e. JSON/XML serialization of a HABTM or has_many relationship. You want something like { :name=>”Pac-Man”, :creator=>”Namco”, :high_scores=>[{ :player_id=>1, :score=>99999 }, {:player_id=>2, :score=>88888 }] }, but all you get is the top-level attributes :name and :creator.

What to do?

First, I just discovered this extremely helpful library – Jsonifier, which also ships as a Rails plugin. It deals with both of the above problems. There’s no “attributes” chaff and it deals with associated records, even going recursive and letting you navigate from end of a many-to-many into the other. (In my example, you could show the names and ages of Pac-Man high scorers.) You also get to whitelist or blacklist attributes matches Rails’ associations syntax, i.e. :only, :except. Highly recommended, check it out.

Jsonifier, however, is pure to_json – it doesn’t (yet) handle deserialization. To my knowledge, there’s no Rails library that persists both parent and children in one go. There are lots of forum posts about handling multiple checkboxes and the like, and the answer usually involves overriding “children=” or “child_ids=”. I want to do this automatically!

I’ve been working out how to extend ActiveRecord to persist automatigically. I have the following very rough code working in basic situations, although it would need to do some more validation to work in production, among other things. It works on data structures such as the recurisve one above. You must first use a before_filter to convert the incoming JSON string into a Rails structure.

class << ActiveRecord::Base

def recursive_create(params) transaction do collections_by_children = pluck_hash_collections_by_children params new_model = self.create(params) populate_children new_model, collections_by_children new_model end end

def pluck_hash_collections_by_children(params) hash_collections_by_children = Hash.new params.each_pair { |name, value| if value.class==Array hash_collections_by_children[name] = value params.delete(name) end } hash_collections_by_children end

def populate_children(new_model, hash_collections_by_children) hash_collections_by_children.each_pair { |children_name, hash_collection| new_model.send(“#{children_name}”).destroy_all child_class_name = children_name.singularize.camelize child_class = Object.const_get(child_class_name) hash_collection.each { |child_hash| child = child_class.create(child_hash) new_model.send(“#{children_name}”) << child } } end

end

module ActiveRecord class Base

def recursive_update_attributes(params)
  transaction do
    collections_by_children = self.class.pluck_hash_collections_by_children params
    self.update_attributes(params)
    self.class.populate_children self, collections_by_children
    save
  end
end

end

end [/ruby]

Ruby/Rails: Overriding NilClass

Rails uses “whiny nil”, which means if you call a method on an object that happens to be nil (null), you get an exception. This is good. But with strings in a web app (in any language), you often don’t know if an empty string will be nil or simply zero-length (“”). That’s because some code will see a form submitted with a whitespace string and make it nil, while others will make it “”. These might be my libraries or the web framework’s libraries or plugin libraries. This leads to ugly code like this:

  1. @message = "Oi! Enter your name fella" if name or name.empty?

When we really just want

  1. @message = "Oi! Enter your name fella" if name.empty?

So I derive great pleasure from overriding NilClass like so:

  1. class NiClass
  2.   def empty?
  3.     true
  4.   end
  5. end

If HTML Used Convention Over Configuration …

Browsers would automatically pull in CSS and JS according to the filename and I would no longer have to look for an example every time I need a link or script tag.

In the absence of any other spec, /abc/def.html would cause the browser to look for /abc/def.css and /abc/site.css and /site.css. And then it would look for the same, but in JS – /abc/def.js, /abc/site.js, and /site.js. If I had all the time in the world, I’d make an Apache filter to do just that.

This would slow things down a little, but have you noticed – the world is becoming faster? It’s an another example of abundance thinking. Also, if it was a standard, it would not really slow things down as browsers and servers would develop protocols for speeding things up.

So I created a little Ajax web app template to put all the skeleton Ajax code on one page.

HTML

<html>
  <head>
     <title>Hello World!</title>

 &lt;script type=&quot;text/javascript&quot; src=&quot;app.js&quot;&gt;&lt;/script&gt;
 &lt;link rel=&quot;stylesheet&quot; type=&quot;text/css&quot; href=&quot;app.css&quot;/&gt;

</head> <body> <h1>Hello World!</h1> <input id="search" name="search" />

</body> </html>

CSS

body { background-color: white; }
div.error { font-color: red; }

search { width: 200px; }

Javascript

function $(id) { return document.getElementById(id); }

window.onload = function() { $("search").onclick = function() { alert("The search begins!"); } }

Ruby is Rails is … REST

Will the peripheral IT community come to view REST and Rails as equivalent? It might sound ridiculous, but consider: Unix==Linux, Wiki==Wikipedia, Ajax=Web 2.0, blogging==RSS, podcast==spoken MP3. Last but not least, every knows that

Ruby==Rails

So it only stands to reason that the REST equivalence shall come to pass, as REST hops on for a free ride on the Rails worthy-hype machine.

REST==Rails

 

(For the sake of completion, yes, Ruby==REST.)

The possibility of Rails==REST didn’t occur to me until I read this – probably quite necessary – clarification:

I want also take a step back for a moment, and mention that REST is not a Rails-only thing. REST was developed long before Rails, and is simply an approach that can be applied to many kinds of software applications.

If Rails can bring REST to the fore of mainstream development, in addition to other goodies like Convention Over Configuration and Migrations, that’s one more reason to heartily <heart> Rails.