scRUBYt! is a simple to learn and use, yet powerful web scraping toolkit written in Ruby. The idea behind making scRUBYt! was to show a few simple concepts of Web extraction as a practical extension of this tutorial.
We have just released the new version, 0.2.6 with some great new features, tons of bugfixes and lot of changes overall which should greatly affect the reliability of the system.
A lot of long-awaited features have been added: most notably, automatic crawling to the detail pages, which was the most requested feature in scRUBYt!’s history ever. I will add a tutorial and detailed example on how to use this feature, which enables you to easily crawl a whole site.
Another great addition is the improved example generation - you don’t have to use the whole text of the element you would like to match anymore - it is enough to specify a substring, and the first element that contains the string will be returned. Moreover, you can use also regular expressions, in which case the first element with text content matching the regexp will be returned. If this still won’t be enough, it is possible to create a compound example like this:
flight :begins_with => 'Arrival', :contains /d{4}/, :ends_with => '20:00'
I guess it’s quite intuitive how should this work.
We have finished to fix an enormous amount of bugs and tested the whole system thoroughly, so the overall reliability should be improved a lot as opposed to the previous releases.
If you have any comments, questions, suggestions, please visit the brand new forum!
Last, but not least the full changelog:
* [NEW] Automatically crawling to and extracting from detail pages
* [NEW] Compound example specification: So far the example of a pattern
had to be a string. Now it can be a hash as well, like
{:contains => /dd-d/, :begins_with => 'Telephone'}
* [NEW] More sophisticated example specification: Possible to use regexp
as well, and need not (but still possible of course) to specify the whole
content of the node - nodes that contain the string/match the regexp
will be returned, too
* [NEW] Possibility to force writing text in case of non-leaf nodes
* [NEW] Crawling to the next page now possible via image links as well
* [NEW] Possibility to define examples for any pattern (before it did not
make sense for ancestors)
* [NEW] Implementation of crawling to the next page with different methods
* [NEW] Heuristics: if something ends with _url, it is a shortcut for:
some_url 'href', :type => :attribute
* [FIX] Crawling to the next page (the broken google example): if the next
link text is not an <a>, traverse down until the <a> is found; if it is
still not found, traverse up until it is found
* [FIX] Crawling to next pages does not break if the next link is greyed out
(or otherwise present but has no href attribute (Credit: Robert Au)
* [FIX] DRY-ed next link lookup - it should be much more robust now as it
uses the 'standard' example lookup
* [NEW] Correct exporting of detail page extractors
* [NEW] Added more powerful XPath regexp (Credit: Karol Hosiawa)
* [NEW] New examples for the new featutres
* [FIX] Tons of bugfixes, new blackbox and unit tests, refactoring and
stabilization
Please keep the feedback coming - your contributions are a key factor to scRUBYt!’s succes. This is not an exaggeration or a feeble attempt at flattery - since we (obviously) can not test everything on every possible page, we can make scRUBYt! truly powerful only if you send us all the quirks and problems you encounter during scraping, as well as your suggestions and ideas. Thanks everyone!
March 26th, 2007 at 2:53 am
Great job, Peter.
One thing I’d like to suggest is the use of a proxy to record interactions with a webpage and automatically allow the user to select items to parse or record a user’s interactions, generate the right script to get the user to that page (including POSTs / GETS) and let the user then select items to parse out of the target page.
April 9th, 2007 at 5:53 am
Thanks!
About the proxy: yeah, that’s a great idea!
ATM we are working on refactoring the whole codebase as it has grown far beyond the limits ever planned and a complete rewrite is needed to prepare it for the new additions. Further we are adding FireWatir support and completely rewriting the output system which will result in a lot of output formats and a lot of flexibility overall.
This means a lot of work - however, after this will be done we will be adding new features and this idea sounds great!