FelSpider: Web Spider - Felix John COLIBRI.

• abstract : Download a complete web site, with custom or gui filtering and selection of the downloaded files.
  
• key words : web spider, downloader, off line reader, HTTP, HTML, TCP IP, virtual path
  
• software used : Windows XP, Delphi 6
  
• hardware used : Pentium 1.400Mhz, 256 M memory, 140 G hard disc
  
• scope : Delphi 1 to 8 for Windows, Kylix
  
• level : Delphi developer
  
• plan :
  
  • Introduction
    
  • Web Spidering
    
  • The Delphi Sources
    
  • Improvements
    
  • Download the Sources
    

1 - Introduction

To automate the download of the pages from a site, we use a Web Spider. The basic idea is very old: start with some .HTML page, extract all its links, and recurse by downloading those links.

We already presented a single page downloader, which downloads a page, analyzes the anchor tags, and downloads the corresponding image tags. So all the features are already present. The spider then only adds the page selection.

2 - Web Spidering

2.1 - The objective

The answer is simple: file swamping. If you donwload, say, the page from a learned teacher at MIT or Waterloo about "UML and Design Patterns", chances are that this page will also reference:

• other courses taught by the same professor which might not interest you (java programming, computer 101 etc)
  
• coleague's and budies pages, which might work in other areas (chemistry or psychiatry)
  
• personal pages (resume, photo galleries of last conventions, favorite songs)
  
• links to other universities with similar courses. And from there, the flood starts anew.
  

2.2 - Spidering Strategies

• we can forbid some pages that we know will seldom interest us. In our case, all multi-media stuff is banned right from the start: no .AVI, .MOV, .WAV and the like. Note that since the advent of BDN (Borland community movies) this might change.
  
• the second most effective technique was to limit the pages to the request's local path. If we request Bertrand MEYER's page:

  http://www.inf.ethz.ch/personal/meyer/

  we are usually also interested to load the sub directories referenced in this page:

  http://www.inf.ethz.ch/personal/meyer/events/2001/ubs.html http://www.inf.ethz.ch/personal/meyer/events/talks.html http://www.inf.ethz.ch/personal/meyer/images/sounds/pronunciation.html http://www.inf.ethz.ch/personal/meyer/publications/ http://www.inf.ethz.ch/personal/meyer/publications/computer/dotnet.pdf http://www.inf.ethz.ch/personal/meyer/publications/inaugural/inaugural.pdf http://www.inf.ethz.ch/personal/meyer/publications/joop/overloading.pdf http://www.inf.ethz.ch/personal/meyer/publications/online/bits/index.html

  but would be reluctant to download pages with a complete different domain:

  http://www.csse.monash.edu.au http://se.inf.ethz.ch/jobs/

  Some references should interest us (page aliases or pages with quite similar content):

  http://www.inf.ethz.ch/~meyer/ http://www.inf.ethz.ch/~meyer/events/ http://www.inf.ethz.ch/~meyer/ongoing/ http://www.inf.ethz.ch/~meyer/ongoing/covariance/recast.pdf http://www.inf.ethz.ch/~meyer/ongoing/etl/ http://www.inf.ethz.ch/~meyer/ongoing/etl/agent.pdf http://www.inf.ethz.ch/~meyer/ongoing/references/ http://www.inf.ethz.ch/~meyer/publications/ http://www.inf.ethz.ch/~meyer/publications/events/events.pdf http://www.inf.ethz.ch/~meyer/publications/extraction/extraction.pdf http://www.inf.ethz.ch/~meyer/publications/ieee/trusted-icse.pdf http://www.inf.ethz.ch/~meyer/publications/proofs/references-proofs.pdf http://www.inf.ethz.ch/~meyer/publications/reviews/csharp-review.html

  and some could interest us (his books, I guess):

  http://www.inf.ethz.ch/ http://vig.pearsoned.com/store/product/1,3498,store-562_isbn-0130622966,00.html http://www.amazon.com/exec/obidos/ASIN/0130331155/qid=1003330798/sr=8-1/ref=sr_8_7_1/104-4

  Based on this example, we see that:

  • pages in sub directories are very often of interest
    
  • pages in parent directories are of increasing generality, and the risk of overflooding increases
    
  • pages in other domain cannot easily be automatically selected, even if they have a close relation to the requested domain
    
• some site offer links to files available for download. We frequently encountered this for "reading lists" of some university courses. Some teachers place those files in a local (sometimes private) directory, but some other simply list the URL where a copy of the file is available. So, in this case, the strategy should be "download all the non-html files listed in this page". Access to those domains should be allowed, but parsing of the corresponding parent page and pathes refused.
  

• the three sites

• we forbid some pages (.AVI, .MOV, .DOC etc)

• here is an example of page and sub pathes:

• here is an example of page+ any pages of some kind (reading list .PDF):

2.3 - File organization

• by topic (database, web programming, accounting ...)
  
• by request domain and path name (c:\www.cs.berkeley.edu\..., c:\www.ece.uwaterloo.ca\... etc)
  

We cannot save the files under the requested page name only, since a page often contains links to some parent page (for an organization logo, for instance). And even if we adopt a strict "only sub directory" structure, we might want to later download sibbling directories, or upper directories.

Here is an example:

• we want to download the class diagram page of ModelMaker:

  http://www.modelmakertools.com/modelmaker/screenshots/page2.html

• we could save this page as:

  c:\www.modelmakertools.com\page2.html

• however, we could later want to also look at any page1.html, page3.html, or the parent http://www.modelmakertools.com/modelmaker/ page, even the other tools, like http://www.modelmakertools.com/code-explorer/

  This is why we save the page2.html as:

  c:\www.modelmakertools.com\modelmaker\screenshots\page2.html

  
  

We also include an "__doc.txt" file at the root of all our downloade sites (in our case, this would be c:\www.modelmakertools.com\__doc.txt), and we usually place some key words in this text. All the __doc.txt can then be automatically parsed for some expressions to tell which sites contain some topic.

Note that our naming scheme is not totally alias free. Some sites can be renamed, split or otherwise reorganized under different names. This is even the case for our own site, which can be reached with www.felix-colibri.com OR www.colibrix.com. Removing such alias would need the computation of unique signature for each page and an indexing scheme. We considered that it would require too much work to simply remove some redundancy.

2.4 - The Spidering algorithm

and:

• "fill the requested URL list": we usually start with a seed page. However we could type a list of URLs in a tMemo, load the list from a "todo" file, or paste some URLs from previously rejected pages
  
• "select the first URL" : this starting page is either an URL ending with .HTML, .PHP, .ASP etc, or the domain name, or some path in the target domain.

  http://download.oracle.com/otn/nt/oracle10g/10g_win32_db.zip   /downloads/index.html /ww/ aboutMe.html#contactInfo bliki ../../super_search.htm ./files/tables.zip   http://cc.borland.com/beta/Item.aspx?id=23191

• "donwload the selected URL" : the socket sends a GET, and the Server answers either with the requested explicit page, or with a default page (index.html), or the page dynamically build from a CGI request. The socket the receives the HTTP answer
  
• "save the page" : in the usual "domain download" case, we save the page in the path derived from the URL. In the case of a "reading list" download, we save the files in the requested page folder. And if the name of a file is a relative name, we must add the absolute path.
  
• "IF this is an HTML containing page" : we cannot use the request extension as a reliable way to trigger the .HTML parsing.
  So, when the extension is not an .HTML like exention or an explicit non-HTML exention (.ZIP, .PDF, .PNG ...), we look whether the file contain some .HTML tag. We chose <HTML> and </HTML> as the test criterion, although this is not bullet proof:
  • some binary file might contain those tags, and still conaint only binary data
    
  • some web pages do not contain <HTML> and </HTML> (only <BODY>, or even only the text with some text tags and some anchors
    
• "extract the anchor tags" if the page is a declared or detected .HTML page, we parse the content and build the anchor tag list
  
• "filter the tags" : this is the filtering stage. Here we can
  
  • eliminate unwanted extensions (.WAV ...)
    
  • eliminate some links based on textual patterns (no URL containing "resume", or "photo gallery" etc
    
  • eliminate links which do not contain the same domain as the initially requested page
    
• "select the next URL" : this step will determine the dowload order. If we download all files up to the end, the download order has no importance whatsoever. But if we might abort the process after some time (bug, overflooding, timeout ...), then it is obviously better to first download the "best" pages. The ranking function obviously depends on the user needs. In our case
  • we usually first donwload the non-HTML files (the .PDD, .ZIP etc)
    
  • then we donwload the .HTML pages
    
    • first the sub pages
      
    • then the parent pages, gradually moving from the current path to the site root
      
      
  This can be presented in the following figure.
  • In Red the unwanted pages (.MOV) and in blue the non-HTML pages (.PDF):

  • we download the target page:

  • then the non HTML pages:

  • and the sub pages:

  • and we gradually fetch the parent pages:

    and

• a customizable filtering callback where one can add some condition to remove some pages
  
• a customizable next page selection procedure
  

• the page level (parent, the page itself, child pages)
  
• the type of pages (none, all, only .HTML-like, image, zip and .PDF like)
  

3 - The Delphi full Source code

3.1 - The Overall organization

The spider only introduces filtering and page ordering functionalities.

So we simply derived the c_site_spider from the c_spider, adding the needed attributes and methods. The main form was also slightly changed.

The filtering and selection additions are:

• a default management
  
• a custom callback possibility (with some requested coding on the main Form)
  
• a gui handling
  

3.2 - The Guid management

The snapshot shows the selection of

• all pages at the requested URL level
  
• the .HTML, .ZIP and .PDF from the children
  
• only the images from the parent pages
  

and:

• we use two enumerated types for our levels and page types
  
• the c_site_spider contains the m_selection_array where the boolean are stored by the user
  
• the setting of the booleans is performed using the set_selection method
  

To select which pages must be kept for download, we test each anchor tag:

where

• m_selection_array[p_page_level_type, l_page_type] tests what the user selected
  
• m_page_level_fo_array[l_page_level_type](p_c_url) test the actual page level. The m_page_level_fo_array was initialized in the CONSTRUCTOR to point to page level test functions. So testing:

  if m_page_level_fo_array[e_child_level_page](p_c_url)   then

  is the same as

  if f_is_child_page(p_c_url)   then

  And the same is done with the m_page_type_fn_array test
  

3.3 - The Custom Filtering and Selection

For the filtering, we simply created a callback in the c_site_spider CLASS, and if this method is initialized by the developer, the filtering procedure uses this callback to decide if a pages should be kept or not:

• here is the filtering hook:

  t_po_accept_url= procedure(p_c_url: c_url; var pv_accept: Boolean) of Object; c_site_spider= class(c_spider)                  m_on_filter_url: t_po_accept_url;                  // ...                end; // c_site_spider

• the developper adds the method in tForm1 CLASS:

  TForm1= class(TForm)           // ...           private             procedure filter_page(p_c_url: c_url; var pv_accept: Boolean);         end; // tForm1

• with, for instance, this check:

  procedure TForm1.filter_page(p_c_url: c_url; var pv_accept: Boolean);   begin     with g_c_spider do       pv_accept:=           (              f_is_child_page(p_c_url)           and              f_is_html_extension(p_c_url)           )         or           (              f_is_domain_page(p_c_url)           and              not f_is_child_page(p_c_url)           and             (p_c_url.m_extension= '.png')           );   end; // filter_page

• and the callback pointer is initialized when the c_site_spider object is created:

  c_site_spider= class(c_spider)                  m_oa_on_select_url: array of t_po_accept_url;                  // -- ...                end; // c_site_spider

  
  

• first find a page on the current level
  
• then look at the child page level
  
• then search the parent level
  

• in the c_site_spider CLASS we have the callback array:

  c_site_spider= class(c_spider)                  m_oa_on_select_url: array of t_po_accept_url;                  // -- ...                end; // c_site_spider

• the user adds the methods definitions in the tForm CLASS:

  TForm1= class(TForm)           // ...           private             procedure select_next_page_child(p_c_url: c_url; var pv_accept: Boolean);             procedure select_next_page_parent_images(p_c_url: c_url; var pv_accept: Boolean);         end; // tForm1

• with, for instance, the following implementation:

  procedure TForm1.select_next_page_child(p_c_url: c_url; var pv_accept: Boolean);   begin     with g_c_spider do       pv_accept:= f_is_child_page(p_c_url) and f_is_html_extension(p_c_url);   end; // select_next_page_child procedure TForm1.select_next_page_parent_images(p_c_url: c_url;     var pv_accept: Boolean);   begin     with g_c_spider do       pv_accept:= f_is_domain_page(p_c_url) and not f_is_child_page(p_c_url)           and (p_c_url.m_extension= '.png');   end; // select_next_page_parent_images

• and the callbacks are initialzes with:

  procedure TForm1.goClick(Sender: TObject);   begin     // ...     with g_c_spider do     begin       append_selection_procedure(select_next_page_child);       append_selection_procedure(select_next_page_parent_images);       // -- ...

  
  

3.4 - The delfault strategy

• everything from the current page
  
• the .HTML from the children
  
• the images from the parent
  

3.5 - The main form

• a c_find_memo, which can sort the pages which will not be downloaded. This can be used to perform a first analysis:
  • we fist donwload the the main target URL
    
  • the "other pages" displays all anchor tags in this page
    
  • when clicking "sort", we easily see what is in the page itself, the child pages, the other domains etc
    
• the speed button filter selection, which have been displayed above.
  

4 - Improvements

• start the filtering at the tag-analysis level (not after adding the tags to the file list)
  
• improve the GUI selection functionality: buiding the usual expression building technique (the kind found in Quick Report, for instance)
  
• start the download in parallel instead of using a page after page limit
  
• build a more abstract and generic "spidering API"
  

5 - Download the Sources

• web_spider.zip: creating (83 K)
  

• the main program (.DPR, .DOF, .RES), the main form (.PAS, .DFM), and any other auxiliary form
  
• any .TXT for parameters
  
• all units (.PAS) for units
  

• are self-contained: you will not need any other product (unless expressly mentioned).
  
• can be used from any folder (the pathes are RELATIVE)
  
• will not modify your PC in any way beyond the path where you placed the .ZIP (no registry changes, no path creation etc).
  

• create or select any folder of your choice
  
• unzip the downloaded file
  
• using Delphi, compile and execute
  

As usual:

• please tell us at fcolibri@felix-colibri.com if you found some errors, mistakes, bugs, broken links or had some problem downloading the file. Resulting corrections will be helpful for other readers
  
• we welcome any comment, criticism, enhancement, other sources or reference suggestion. Just send an e-mail to fcolibri@felix-colibri.com.
  
• or more simply, enter your (anonymous or with your e-mail if you want an answer) comments below and clic the "send" button

  Name :
  E-mail :
  Comments * :

• and if you liked this article, talk about this site to your fellow developpers, add a link to your links page ou mention our articles in your blog or newsgroup posts when relevant. That's the way we operate: the more traffic and Google references we get, the more articles we will write.
  

6 - Conclusion

7 - The author