scRUBYt! is a simple to learn and use, yet powerful web scraping toolkit written in Ruby. The idea behind making scRUBYt! was to show a few simple concepts of Web extraction as a practical extension of this tutorial.
Judging from your mails and bug reports, the biggest missing thing at the moment is that the cookies are lost during the navigation (which causes for example that after logging in to a site, you can not scrape the page). More people have been asking if this is the intended behavior, and if not, how easily can it be fixed.
Of course this is *not* the intended behavior, but a bug in the present version which will be fixed in the next release. The problem at the moment is that before the scraping, Hpricot loads the page through open-uri (rather than taking it over from Mechanize) and obviously open-uri does not know anything about Mechanize’s cookies.
Fetching every page with Mechanize should automatically fix the https problems as well. I am planning to introduce new options like possibility to set a custom proxy, custom user-agent string and more to further tweak the page loading.
Please note that even these improvements will still not allow to login to javascript sites - like google analytics. I will need to add FireWatir to scRUBYt! for that. This is also on the TODO list, but not for the next release.
February 17th, 2007 at 12:00 am
any time estimates on this?
scRUBYt feels awesome from what I’ve seen this far, but the site I’m interested in seems to rely on cookies. So I’m eagerly awaiting the next version.
February 17th, 2007 at 12:13 am
Time estimate for the next release: next Monday… also a lot of other chunky stuff to be expected!
April 15th, 2007 at 11:12 pm
did cookies make it into the latest release? scrubyt-0.2.8?
April 15th, 2007 at 11:30 pm
Sure! Actually they made it to 0.2.6 already and I guess they should be still there…
April 16th, 2007 at 6:50 am
is there a set cookies call… or is it automatic. i dont see any documentation on it. thanks.
April 16th, 2007 at 6:54 am
No, it should be absolutely automatic - for example when you are logging in to a site, all you need to do is to fill the two textfields and submit the form - and you can navigate further or scrape the page where you have arrived, without caring about cookies or anything.
If you have further questions, please sign up to the forum (http://agora.scrubyt.org)! It’s more convenient to discuss stuff there