As a (positive) side-effect of Ruby’s and RoR’s current popularity, we can witness the mushrooming of Ruby/Rails blogs of different size, shape, color and quality. While this is a great thing in general, it is easy to get overflooded with all this information pouring in from every side - much harder to pick a smaller set which you can monitor in your time devoted for reading Ruby blogs.

Fortunately there are some great Ruby/Rails blog aggregators like http://rubycorner.com/, http://www.planetrubyonrails.org/ - not to be confused with http://www.planetrubyonrails.com/ (here is why) or http://planet.caboo.se/. The problem is that it’s clumsy to monitor all these aggregators - they contain hundreds of blogs each with a lot of overlapping.

This issue was partially solved by this Ruby on Rails Blog Aggregator Dupe Prevention yahoo pipe. However, there is a fundamental problem with that: You still have several hundreds of blogs - nearly 500 of them, and new ones coming out day by day. If you have time to read all these periodically, all the power to you - however, the rest of us would need a way to pick the top n ones in order to be able to cope with their reading.

I am proposing a solution for this problem, by performing these steps:

1. Collecting all the blog URLs from the above 4 aggreagators
2. Performing some data cleaning (e.g. http://www.site.com, http://site.com, http://site.com/blog and other variants, with and without trailing slashes are pointing to the same site) and dupe removal
3. Querying http://www.alexa.com and http://www.technorati.com for the popularity of each of the blogs on the list (this already generated two rankings: “Alexa top 30″ and “Technorati top 30″.
Alexa required some intervention by hand (more on that later).
4. Putting it all together: based on the two top 30 lists, generate a final, all-in-one Ruby/Rails blog top 10!

Let’s see an overview of the process - then I will explain each step in detail!

Let’s begin with scraping http://www.planetrubyonrails.com/. This extractor was the easiest to construct of all the four:

    planet_ror_com_data = Scrubyt::Extractor.define do
      fetch("http://www.planetrubyonrails.com/pages/channels")

      link "http://weblog.jamisbuck.org/"
    end

This wasn’t too hard, was it? We have just extracted all the blogs URLs from planetrubyonrails.com! To observe the result, all we need is to run this snippet:

planet_ror_com_data.to_hash.each{|h| puts h[:link]}

http://weblog.jamisbuck.org/
http://www.therailsway.com/
http://weblog.rubyonrails.com/
http://blogs.relevancellc.com/
http://www.anarchogeek.com/
http://www.bencurtis.com/
http://slash7.com/
http://blog.lathi.net/
http://convergentarts.com/
http://blog.innerewut.de/
http://codefluency.com/rails-views
http://www.busyashell.com/blog/
http://curthibbs.wordpress.com/
http://blog.bleything.net/
http://www.loudthinking.com/
http://railstips.org/
http://null.in/
http://www.jvoorhis.com/
http://www.puneruby.com/blog
http://sitekreator.com/satishtalim/rubyblog.html
http://www.rubyinside.com/
http://www.chadfowler.com/index.cgi
http://dibya.wordpress.com/
http://rubymerchant.blogspot.com/index.html
http://www.rubyonrailsblog.com/
http://blog.caboo.se/
http://www.ryandaigle.com/
http://vinsol.com/
http://jroller.com/page/obie
http://www.infoq.com/
http://drnicwilliams.com/
http://www.danwebb.net/
http://www.lukeredpath.co.uk/
http://blog.testingrails.com/
http://blog.vixiom.com/
http://robertrevans.com/
http://redhanded.hobix.com/
http://benmyles.com/
http://weblog.techno-weenie.net/
http://www.clarkware.com/cgi/blosxom
http://mir.aculo.us/
http://www.oreillynet.com/ruby/blog/
http://www.robbyonrails.com/
http://blog.grayproductions.net/
http://www.rubyquiz.com/
http://www.codyfauser.com/
http://railsexpress.de/blog/
http://www.softiesonrails.com/
http://blog.ericgoodwin.com/
http://iamrice.org/
http://blog.hasmanythrough.com/
http://david.goodlad.ca/
http://nubyonrails.topfunky.com/
http://onrails.org/
http://www.railtie.net/
http://brainspl.at/
http://podcast.rubyonrails.org/
http://blog.leetsoft.com/
http://errtheblog.com/
http://jystewart.net/process
http://www.pluginaweek.org/
http://mephistoblog.com/
http://encytemedia.com/
http://evil.che.lu/
http://technomancy.us/
http://poocs.net/
http://www.opensoul.org/
http://www.mathewabonyi.com/
http://www.railslivecd.org/
http://blog.methodmissing.com/
http://sporkmonger.com/breakpoint_client
http://typo.onxen.info/
http://blog.inquirylabs.com/
http://rashkovskii.com/
http://www.infused.org/
http://interblah.net/
http://streamlinedframework.com/
http://toolmantim.com/
http://webonrails.wordpress.com/
http://www.sapphiresteel.com/
http://eigenclass.org/hiki.rb
http://www.pjhyett.com/
http://jonathan.tron.name/
http://amazing-development.com/
http://merubyyoujane.com/
http://www.fallenrogue.com/
http://troubleseeker.com/
http://blog.nicksieger.com/
http://blog.methodmissing.com/
http://schf.uc.org/
http://www.brianketelsen.com/
http://www.simplisticcomplexity.com/
http://fluctisonous.com/
http://www.pinupgeek.com/
http://web2withrubyonrails.gauldong.net/
http://www.koziarski.net/
http://www.notsostupid.com/
http://www.shanesbrain.net/
http://blog.evanweaver.com/
http://www.zerosum.org/devblog
http://blog.zenspider.com/
http://townx.org/taxonomy/term/5/0
http://www.fromdelhi.com/
http://myles.eftos.id.au/blog
http://www.jeremyhubert.com/
http://www.jason-palmer.com/
http://seanicus.blogspot.com/
http://ajaxonrails.wordpress.com/
nil
http://www.matthewbass.com/blog
http://rails.co.za/
http://www.shifteleven.com/
http://peepcode.com/
http://blog.wolfman.com/
http://blog.dannyburkes.com/
http://jayfields.blogspot.com/
http://www.freeonrails.com/
http://techcheatsheets.com/tag/ruby%20or%20rails
http://rails-engines.org/rss/news/
http://www.railsenvy.com/
http://www.eribium.org/
http://hobocentral.net/blog
http://maintainable.com/articles
http://railspikes.com/
http://radiantcms.org/
http://hackety.org/
http://adam.blogs.bitscribe.net/
http://www.rubyfleebie.com/
http://www.stephenbartholomew.co.uk/
http://izumi.plan99.net/blog
http://blog.davidchelimsky.net

Let’s move on to http://planet.caboo.se/ which is just slightly harder to scrape:

    planet_caboose_data = Scrubyt::Extractor.define do
      fetch("http://planet.caboo.se/")

      link_row do
        link "al3x/@href"
      end
    end

The next site is http://rubycorner.com/. The new thing here compared to the first two scrapers is that the blog URLs can be found on different pages - ergo we need to crawl to all of those pages. Fortunately this is no problem for scRUBYt!:

    rubycorner_data = Scrubyt::Extractor.define do
      fetch 'http://rubycorner.com/blogs/lang/en'

      link "Atlantic Dominion Solutions/@href"
      next_page ">"
    end

Last but not least, here is the scraping code for http://www.planetrubyonrails.org/

      fetch "http://www.planetrubyonrails.org/"

      link_container "div[Feeds]" do
        link "a work on process/@href", :generalize => true
      end
    end

I don’t have too much to say about the data cleaning part - if you are interested, you can download the code used for generating this tutorial and check it out there.

Scraping alexa was great fun - mainly because they don’t want to be scraped! However, if they would like to stop a halfway serious script junkie, they will have to do better than this. Here is the alexa extractor:

  def scrape(site)
    @hidden = open("http://client.alexa.com/common/css/scramble.css").read.scan(/^\.(.+) /).flatten unless @hidden
    alexa_data = Scrubyt:: Extractor.define do
        fetch "http://www.alexa.com/data/details/traffic_details?url=#{site}"

        td "td[Traffic Rank for]" do
          span '/span[3]' do
            html :type => :html_subtree
          end
        end
    end
    data = alexa_data.to_hash[0][:html]
    data ? descramble(data) : nil
  end

  def descramble(mess)
    data = mess.gsub!(//,'').split(//)
    data = data.reject{|d| d=="" || (@hidden.include? d.scan(/"(.+?)"/).flatten[0])}
    data.map!{|d| d.sub(/".+">/,'') }
    data.join('')
  end

Here is the extractor I have used to scrape technorati:

  def scrape(site)
    techno_data = Scrubyt::Extractor.define do
      fetch "http://www.technorati.com/blogs/#{site}?reactions"
      rank "/html/body/div/div/div/div/div/div" do
        rank_number /Rank: (.+)$/
      end
    end
    techno_data.to_hash.reject{|h| h.empty?}[0][:rank_number]
  end

Nothing special here - what you did not see yet in the other extractors is the use of a regular expression pattern (rank_number), which gets the number out of the corresponding element and leaves the trash behind.

The only manual step I had to make in this whole process was to tweak the alexa results a bit - because according to alexa.com, “Alexa’s traffic rankings are for top level domains only (e.g. domain.com). We do not provide separate rankings for subpages within a domain (e.g. www.domain.com/subpage.html) or subdomains (e.g. subdomain.domain.com)“. This in practice meant that I have checked all blogger.com, wordpress.org, jroller.com and similar domains whether they concrete pages are really that high ranked, or just their top-level domain is. This blog also made it to the top 30, but I had to throw it away because about half of the traffic goes to the scRUBYt! forum, and the remaining visits were not enough to stay inside the top 30.

If you are still with me, I guess you had enough of scraping, screwing, cleaning and assembling by now - and your ordeal is over! Here are the results:

There were a few blogs which performed equally well on both lists - they were assembled into a top 10 list, which you can check out here.

I would really like to hear your opinion on this little experiment - whether you think it makes sense or it is completely off, how could it be improved in the future, what features could be added etc. If I’ll receive some positive feedback, I think I will work on the algorithm a bit more, and run it once in say every 3 months to see what’s happening around the Ruby/Rails blogosphere. Let me know what do you think!

Related Links

• The code used in the article
• The final result: Ruby/Rails blog top 10

We have decided it’s time to start a scRUBYt! wiki (thanks to Whitefolkz for coming up with the idea and starting to work on it). You can find the wiki here. Whitefolkz already posted some guidelines and additional info.

Please post your experience, extractors, guides, tutorials or any other nuggets of knowledge you think the others can benefit from! Thanks for your help in advance.