Visual Web Ripper makes it possible to extract information from an entire website using only a point-and-click interface. With the interface and the built-in browser, it is possible to build a template to completely scrape the contents of a web site. The template is then set up to run and the job completed.

However, this simple job isn’t the extent of the capabilities of Visual Web Ripper and, for some users, manipulating the captured or extracted information is just as important as the original data. The solution for this lies in the advanced options.

Websites, in particular forums or catalogues, are not static. Changes to these sites are a concern when considering data extraction. Once the template has been set up and the project run, it may take an hour or more for very large sites to be scraped. Because catalogues may contain numerous images, these sites can take longer to process. Running the entire scraping template each turns in to a long, needless process when the only items of interest are the changes and updates made to the site.

Visual Web Ripper deals with this efficiently in the Advanced Options menu. Each project is different, but the template creator will normally set up the template to run at a specified interval. A catalogue might be ripped once a day while a forum might be checked hourly.

In the template, there are options to detect duplicate content. For example, if pulling data from a forum, the title and last post date of a forum topic could be used as the check for duplicate data. The project creator selects the element title and last post date, and then selects the advanced options for this element.

In the Advanced Options menu under Duplicate Data, users can instruct Visual Web Ripper how to handle duplicate content. The default action is to take no action, which causes the template to download all data no matter what changes are found. However, Visual Web Ripper can be instructed to retrieve only new content by changing the setting.

In a forum situation, there are usually one or more sticky topics (such as Administrator notes or forum rules) and these might be dealt with by using a minimum number of rows to scan before extraction is canceled. That can take care of new sticky topics without downloading unnecessary duplicate data. Some advanced Visual Web Ripper users have created scripts that determine if a topic is sticky before cancelling the template.

Once those changes are made, the template can run regularly but will only extract data which is not duplicated, in other words new posts, and new topics. The ability to do this can be a significant timesaver.