scRUBYt! is a simple to learn and use, yet powerful web scraping toolkit written in Ruby. The idea behind making scRUBYt! was to show a few simple concepts of Web extraction as a practical extension of this tutorial.
As a (positive) side-effect of Ruby’s and RoR’s current popularity, we can witness the mushrooming of Ruby/Rails blogs of different size, shape, color and quality. While this is a great thing in general, it is easy to get overflooded with all this information pouring in from every side - much harder to pick a smaller set which you can monitor in your time devoted for reading Ruby blogs.
Fortunately there are some great Ruby/Rails blog aggregators like http://rubycorner.com/, http://www.planetrubyonrails.org/ - not to be confused with http://www.planetrubyonrails.com/ (here is why) or http://planet.caboo.se/.
The problem is that it’s clumsy to monitor all these aggregators - they contain hundreds of blogs each with a lot of overlapping.
This issue was partially solved by this Ruby on Rails Blog Aggregator Dupe Prevention yahoo pipe. However, there is a fundamental problem with that: You still have several hundreds of blogs - nearly 500 of them, and new ones coming out day by day. If you have time to read all these periodically, all the power to you - however, the rest of us would need a way to pick the top n ones in order to be able to cope with their reading.
I am proposing a solution for this problem, by performing these steps:
Alexa required some intervention by hand (more on that later).
Let’s see an overview of the process - then I will explain each step in detail!
Let’s begin with scraping http://www.planetrubyonrails.com/. This extractor was the easiest to construct of all the four:
planet_ror_com_data = Scrubyt::Extractor.define do
fetch("http://www.planetrubyonrails.com/pages/channels")
link "http://weblog.jamisbuck.org/"
end
This wasn’t too hard, was it? We have just extracted all the blogs URLs from planetrubyonrails.com! To observe the result, all we need is to run this snippet:
planet_ror_com_data.to_hash.each{|h| puts h[:link]}
The result is:
http://weblog.jamisbuck.org/ http://www.therailsway.com/ http://weblog.rubyonrails.com/ http://blogs.relevancellc.com/ http://www.anarchogeek.com/ http://www.bencurtis.com/ http://slash7.com/ http://blog.lathi.net/ http://convergentarts.com/ http://blog.innerewut.de/ http://codefluency.com/rails-views http://www.busyashell.com/blog/ http://curthibbs.wordpress.com/ http://blog.bleything.net/ http://www.loudthinking.com/ http://railstips.org/ http://null.in/ http://www.jvoorhis.com/ http://www.puneruby.com/blog http://sitekreator.com/satishtalim/rubyblog.html http://www.rubyinside.com/ http://www.chadfowler.com/index.cgi http://dibya.wordpress.com/ http://rubymerchant.blogspot.com/index.html http://www.rubyonrailsblog.com/ http://blog.caboo.se/ http://www.ryandaigle.com/ http://vinsol.com/ http://jroller.com/page/obie http://www.infoq.com/ http://drnicwilliams.com/ http://www.danwebb.net/ http://www.lukeredpath.co.uk/ http://blog.testingrails.com/ http://blog.vixiom.com/ http://robertrevans.com/ http://redhanded.hobix.com/ http://benmyles.com/ http://weblog.techno-weenie.net/ http://www.clarkware.com/cgi/blosxom http://mir.aculo.us/ http://www.oreillynet.com/ruby/blog/ http://www.robbyonrails.com/ http://blog.grayproductions.net/ http://www.rubyquiz.com/ http://www.codyfauser.com/ http://railsexpress.de/blog/ http://www.softiesonrails.com/ http://blog.ericgoodwin.com/ http://iamrice.org/ http://blog.hasmanythrough.com/ http://david.goodlad.ca/ http://nubyonrails.topfunky.com/ http://onrails.org/ http://www.railtie.net/ http://brainspl.at/ http://podcast.rubyonrails.org/ http://blog.leetsoft.com/ http://errtheblog.com/ http://jystewart.net/process http://www.pluginaweek.org/ http://mephistoblog.com/ http://encytemedia.com/ http://evil.che.lu/ http://technomancy.us/ http://poocs.net/ http://www.opensoul.org/ http://www.mathewabonyi.com/ http://www.railslivecd.org/ http://blog.methodmissing.com/ http://sporkmonger.com/breakpoint_client http://typo.onxen.info/ http://blog.inquirylabs.com/ http://rashkovskii.com/ http://www.infused.org/ http://interblah.net/ http://streamlinedframework.com/ http://toolmantim.com/ http://webonrails.wordpress.com/ http://www.sapphiresteel.com/ http://eigenclass.org/hiki.rb http://www.pjhyett.com/ http://jonathan.tron.name/ http://amazing-development.com/ http://merubyyoujane.com/ http://www.fallenrogue.com/ http://troubleseeker.com/ http://blog.nicksieger.com/ http://blog.methodmissing.com/ http://schf.uc.org/ http://www.brianketelsen.com/ http://www.simplisticcomplexity.com/ http://fluctisonous.com/ http://www.pinupgeek.com/ http://web2withrubyonrails.gauldong.net/ http://www.koziarski.net/ http://www.notsostupid.com/ http://www.shanesbrain.net/ http://blog.evanweaver.com/ http://www.zerosum.org/devblog http://blog.zenspider.com/ http://townx.org/taxonomy/term/5/0 http://www.fromdelhi.com/ http://myles.eftos.id.au/blog http://www.jeremyhubert.com/ http://www.jason-palmer.com/ http://seanicus.blogspot.com/ http://ajaxonrails.wordpress.com/ nil http://www.matthewbass.com/blog http://rails.co.za/ http://www.shifteleven.com/ http://peepcode.com/ http://blog.wolfman.com/ http://blog.dannyburkes.com/ http://jayfields.blogspot.com/ http://www.freeonrails.com/ http://techcheatsheets.com/tag/ruby%20or%20rails http://rails-engines.org/rss/news/ http://www.railsenvy.com/ http://www.eribium.org/ http://hobocentral.net/blog http://maintainable.com/articles http://railspikes.com/ http://radiantcms.org/ http://hackety.org/ http://adam.blogs.bitscribe.net/ http://www.rubyfleebie.com/ http://www.stephenbartholomew.co.uk/ http://izumi.plan99.net/blog http://blog.davidchelimsky.net
Looks good so far (although there is a nil value - we will deal with that later).
Let’s move on to http://planet.caboo.se/ which is just slightly harder to scrape:
planet_caboose_data = Scrubyt::Extractor.define do
fetch("http://planet.caboo.se/")
link_row do
link "al3x/@href"
end
end
The situation changed a bit: We need to match the row containing the blog address and feed address first, then scrape the “href” attribute of the blog URL link. Note how is this done - by appending “/@href” to the example string.
The next site is http://rubycorner.com/. The new thing here compared to the first two scrapers is that the blog URLs can be found on different pages - ergo we need to crawl to all of those pages. Fortunately this is no problem for scRUBYt!:
rubycorner_data = Scrubyt::Extractor.define do
fetch 'http://rubycorner.com/blogs/lang/en'
link "Atlantic Dominion Solutions/@href"
next_page ">"
end
next_page is a special pattern which instructs scRUBYt! to crawl to next pages (the example “>” is used to identify the next page link).
Last but not least, here is the scraping code for http://www.planetrubyonrails.org/
fetch "http://www.planetrubyonrails.org/"
link_container "div[Feeds]" do
link "a work on process/@href", :generalize => true
end
end
The link_container pattern instructs scRUBYt! to get a div containing the text “Feeds”, so that we scrape the links only inside the appropriate div (otherwise we would have some false positives). The :generalize => true option is used to match all the links, not just the first one.
I don’t have too much to say about the data cleaning part - if you are interested, you can download the code used for generating this tutorial and check it out there.
Scraping alexa was great fun - mainly because they don’t want to be scraped! However, if they would like to stop a halfway serious script junkie, they will have to do better than this. Here is the alexa extractor:
def scrape(site)
@hidden = open("http://client.alexa.com/common/css/scramble.css").read.scan(/^.(.+) /).flatten unless @hidden
alexa_data = Scrubyt:: Extractor.define do
fetch "http://www.alexa.com/data/details/traffic_details?url=#{site}"
td "td[Traffic Rank for]" do
span '/span[3]' do
html :type => :html_subtree
end
end
end
data = alexa_data.to_hash[0][:html]
data ? descramble(data) : nil
end
def descramble(mess)
data = mess.gsub!(//,”).split(//)
data = data.reject{|d| d==”" || (@hidden.include? d.scan(/”(.+?)”/).flatten[0])}
data.map!{|d| d.sub(/”.+”>/,”) }
data.join(”)
end
If the above code does not make too much sense to you, don’t worry - I will explain it in a follow-up post. What really matters is that it is possible (and relatively easy) to scrape alexa with scRUBYt! and Ruby.
Here is the extractor I have used to scrape technorati:
def scrape(site)
techno_data = Scrubyt::Extractor.define do
fetch "http://www.technorati.com/blogs/#{site}?reactions"
rank "/html/body/div/div/div/div/div/div" do
rank_number /Rank: (.+)$/
end
end
techno_data.to_hash.reject{|h| h.empty?}[0][:rank_number]
end
Nothing special here - what you did not see yet in the other extractors is the use of a regular expression pattern (rank_number), which gets the number out of the corresponding element and leaves the trash behind.
The only manual step I had to make in this whole process was to tweak the alexa results a bit - because according to alexa.com, “Alexa’s traffic rankings are for top level domains only (e.g. domain.com). We do not provide separate rankings for subpages within a domain (e.g. www.domain.com/subpage.html) or subdomains (e.g. subdomain.domain.com)“. This in practice meant that I have checked all blogger.com, wordpress.org, jroller.com and similar domains whether they concrete pages are really that high ranked, or just their top-level domain is. This blog also made it to the top 30, but I had to throw it away because about half of the traffic goes to the scRUBYt! forum, and the remaining visits were not enough to stay inside the top 30.
If you are still with me, I guess you had enough of scraping, screwing, cleaning and assembling by now - and your ordeal is over! Here are the results:
There were a few blogs which performed equally well on both lists - they were assembled into a top 10 list, which you can check out here.
I would really like to hear your opinion on this little experiment - whether you think it makes sense or it is completely off, how could it be improved in the future, what features could be added etc. If I’ll receive some positive feedback, I think I will work on the algorithm a bit more, and run it once in say every 3 months to see what’s happening around the Ruby/Rails blogosphere. Let me know what do you think!
June 10th, 2007 at 10:45 pm
subscribers would be an important metric of “importance” too. Do any of the large reader sites (bloglines, google reader) publish # subscribers for their user base?
June 10th, 2007 at 10:46 pm
Perhaps you could publish an OMPL file of these blog’s feed urls; for new people.
June 10th, 2007 at 11:27 pm
1): Great idea! I have been thinking about scraping feedburner stats, but not everybody is using feedburner, so I discarded this idea - but blogines/google reader might actually work! Going to check it out.
2): Yet another great idea - please don’t stop generating them :-). I’ll do this as soon as I’ll have a little time.
June 13th, 2007 at 9:52 am
I’d definitely place subscriber numbers over both Alexa and Technorati. As to why, I have a blog that has virtually nothing on it, but it has an Alexa ranking of somewhere around 100k, and a Technorati rating of 12k. It also probably has 20 subscribers, if that! Now if it happened to have some posts on Rails (and was listed on any of the blog aggregators), it’d stand a reasonable chance of ending up near the top 10. It’s really easy to scam those sorts of numbers, whereas subscriber numbers don’t usually lie (or at least it takes a little more effort to scam). This is the type of thing piss poor SEO types love to take advantage of. Don’t give them that chance
June 14th, 2007 at 1:34 am
This is a great idea, especially if you’re able to add stats on subscribers.
How about publishing the top 100 to inspire those of us who blog rarely to try harder and climb the charts.
June 14th, 2007 at 1:56 am
@The Dude:
Well, such bogus sites
1) Don’t have Ruby/Rails posts
2) If they do, they don’t have good ones (maybe if they plagiarize) - but in either case, they won’t make it to the aggregators, or will be thrown out sooner or later.
While I agree with you that it is easy to scam alexa and technorati, I dont’t think so anyone wants to scam the ruby/rails aggregators (OK, if someone wants he can do it, but why should?)
June 14th, 2007 at 2:01 am
@Old Dog:
Yeah, I am thinking about something like that. Most probably I will join the rubycorner.com guys and do something like this together with them.
June 29th, 2007 at 6:34 am
Doesn’t seem to be working anymore. I get this error message:
/usr/lib/ruby/gems/1.8/gems/scrubyt-0.3.0/lib/scrubyt/utils/xpathutils.rb:110:in
traverse_up_until_name': The element is nil! This probably means the widget with the specified name ('td') does not exist (RuntimeError)findnodefromtext’from /usr/lib/ruby/gems/1.8/gems/scrubyt-0.3.0/lib/scrubyt/utils/simple_example_lookup.rb:32:in
from /usr/lib/ruby/gems/1.8/gems/scrubyt-0.3.0/lib/scrubyt/core/scraping/filters/treefilter.rb:67:in
generate_XPath_for_example'initialize’from /usr/lib/ruby/gems/1.8/gems/scrubyt-0.3.0/lib/scrubyt/core/scraping/pattern.rb:110:in
from /usr/lib/ruby/gems/1.8/gems/scrubyt-0.3.0/lib/scrubyt/core/scraping/pattern.rb:109:in
initialize'methodmissing’from /usr/lib/ruby/gems/1.8/gems/scrubyt-0.3.0/lib/scrubyt/core/shared/extractor.rb:115:in
from ./alexascraper.rb:12:in
scrape'define’from /usr/lib/ruby/gems/1.8/gems/scrubyt-0.3.0/lib/scrubyt/core/shared/extractor.rb:32:in
from ./alexa_scraper.rb:9:in `scrape’
from blogranker.rb:10
from blogranker.rb:9
July 8th, 2007 at 7:45 am
Hello
I can’t be bothered with anything these days, but shrug. I just don’t have anything to say recently.
G’night
August 30th, 2007 at 4:13 am
hi,
i ve been using this url http://data.alexa.com/data?cli=10&dat=s&url=
to scrape datas from alexa, and unlike the main site, it’s a feed and the traffic rank is saved in the TEXT field under POPULARITY URL tag.
esh
August 30th, 2007 at 7:30 am
You can access the Alexa data easier w/o scraping using the free AWIS4Ruby project:
http://labs.votanweb.com/awis4ruby/
September 3rd, 2007 at 12:58 am
Jay,
Thanks for the link.
It’s a well known fact that in todays era (web2.0, WS, RSS/ATOM, REST… etc) the more modern sites are usually providing different methods (usually cleaner and more effective than scraping) to access their data. The aim of these tutorials is not to promote scraping over these techniques - it’s just that people are more interested in scraping google, alexa, amazon etc. than some old Web1.0 sites, even in the case that in real life, probably 90% of your scrapers will be targeted at those…
September 22nd, 2007 at 4:24 am
[…] site RubyRailWais listou os 10 blogs mais populares do momento, de acordo com o ranking da Alexa e Technorati . Veja as sugestões e atualize seu […]
December 23rd, 2007 at 6:48 am
look at it first time sex video
May 20th, 2008 at 3:11 pm
wtcubijsy ebjcuzik hiclqynu xojen sgbf xaiu cjbfzgyit
May 20th, 2008 at 3:13 pm
ecru gtmhp hesnvcm qtcln nhrfwktx ozgq yprjab http://www.zkmjdo.bymqj.com
May 20th, 2008 at 3:14 pm
hkeosfwp dehxfabc ufoadq xqleiwygk dyxvsfl nwgodltq xpvtouhn hbudwpgv jiwe
May 20th, 2008 at 3:16 pm
fuvxazq oxterzq fgdeqltxw loak fvkehtgc qtboy zmevjwu
May 20th, 2008 at 3:16 pm
mspvoa kosv gmqkjcws wpsyjxrh yolecmxt liysxkn civp http://www.eocr.emcagl.com
May 20th, 2008 at 3:18 pm
lyjh vlhidpa tzvdfai pzhqr xsjzfkcpd rgxz sxngufez
May 20th, 2008 at 3:18 pm
cwyzu voax nerhdoi gtcqy pxbjrnwm uwpeqi edlmy http://www.vzyokrfd.qsku.com
May 23rd, 2008 at 4:53 am
no Amantadine Amantadine parkinsons