Latest step by Step Installation guide for dummies: Nutch 0.9

By Peter P. Wang, Zillionics Inc.

Try the search engine I developed: Malachi Search

Please support my effort by using the best free/low price web hosting: 1&1 Inc

peterwang@zillionics.com

To add your comments, please go to: http://nutchtube.blogspot.com/2008/02/latest-step-by-step-installation-guide.html

 

  1. Download software
    1. Nutch 0.9: http://www.apache.org/dyn/closer.cgi/lucene/nutch/
    2. JAVA JDK 6 update 3: http://java.sun.com/javase/downloads/index.jsp
    3. Apache web server 6: http://tomcat.apache.org/download-60.cgi
    4. Cygwin: http://www.cygwin.com/

 

  1. Install software one by one
    1. First, install cygwin: run cygwinSetup.exe. You should see this when run it.

 

    1. Second, install JAVA: run dk-6u3-windows-i586-p.exe

 

    1. Third, install Apache: run apache-tomcat-6.0.14.exe.

Run it by clicking the Configure Tomcat icon below.

 

Click the Start button below to start Apache Tomcat Service.

 

Then you will be able to see the following screen in the browser if you go to http://localhost:8080

 

    1. Fourth, unzip nutch-0.9.tar.gz to any directory you like, e.g. c:\nutch.

 

 

  1. Setup the crawler
    1. In Cygwin window, go to the directory of your nutch, and set your JAVA_HOME as follows..

 

    1. Create a directory called urls to hold the a text file with urls inside of it.

 

    1. In this directory, create the text file with any name you like. Put any URL’s line by line. This is the crawler’s “shopping list”. J

 

    1. Edit the file conf/crawl-urlfilter.txt and replace MY.DOMAIN.NAME with the name of the domain you wish to crawl. For example, if you wished to limit the crawl to the apache.org domain, the line should read: 
                      +^http://([a-z0-9]*\.)*apache.org/
 

 

    1. Edit the file conf/nutch-site.xml. insert at minimum following properties into it and edit in proper values for the properties: 
 
<?xml version="1.0"?>
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
 
<!-- Put site-specific property overrides in this file. -->
 
<configuration>
 
<property>
  <name>http.agent.name</name>
  <value>Peter Wang</value>
  <description>Peter Pu Wang
  </description>
</property>
 
<property>
  <name>http.agent.description</name>
  <value>Nutch spiderman</value>
  <description> Nutch spiderman
  </description>
</property>
 
<property>
  <name>http.agent.url</name>
  <value>http://peterpuwang.googlepages.com </value>
  <description>http://peterpuwang.googlepages.com
  </description>
</property>
 
<property>
  <name>http.agent.email</name>
  <value>MyEmail</value>
  <description>peterpuwang@yahoo.com
  </description>
</property>
 
</configuration>
 
  1. Run the crawler

Once things are configured, running the crawl is easy. Just use the crawl command. Its options include:

  • -dir dir names the directory to put the crawl in.
  • -threads threads determines the number of threads that will fetch in parallel.
  • -depth depth indicates the link depth from the root page that should be crawled.
  • -topN N determines the maximum number of pages that will be retrieved at each level up to the depth.

For example, a typical call might be:

bin/nutch crawl urls -dir crawl -depth 3 -topN 50

Typically one starts testing one's configuration by crawling at shallow depths, sharply limiting the number of pages fetched at each level (-topN), and watching the output to check that desired pages are fetched and undesirable pages are not. Once one is confident of the configuration, then an appropriate depth for a full crawl is around 10. The number of pages per level (-topN) for a full crawl can be from tens of thousands to millions, depending on your resources.

  1. Web Searching based on the crawling result above:
    1. Go to http://localhost:8080/manager/html in the browser. In the “WAR file to deploy” section. Select the nutch0.9.war file to upload. It is in your nutch directory. Then you will see the /nutch-0.9 is in the list.

 

    1. Go to your Apache tomcat directory\webapps, e.g. C:\Program Files\Apache Software Foundation\Tomcat 6.0\webapps, and you will see the nutch-0.9.war is already copied there.

 

    1. In the browser’s page http://localhost:8080/manager/html, click “Start” link in the /nutch-0.9 row. Then you will see a folder called “nutch-0.9” will be created in the webapps folder shown above.

d.      Set Your Searcher Directory

Next, navigate to your nutch webapp folder then WEB-INF/classes. Edit the nutch-site.xml file and add the following to it (make sure you don't have two sets of <configuration></configuration> tags!): 

<configuration>
  <property>
    <name>searcher.dir</name>
    <value>your_crawl_folder_here</value>
  </property>
</configuration>

For example, if your nutch directory resides at C:\nutch-0.9.0 and you specified crawl as the directory after the -dir command, then enter C:\nutch-0.9.0\crawl\ instead of your_crawl_folder_here.

e.      Reload

Reload the Application. Use the Tomcat Manager and simply click the "Reload" command for nutch, or restart Tomcat using the windows services tool.

Open up a browser and enter the url http://localhost:8080. The nutch search page should appear. As long as you've defined the correct location of your nutch index directory (as shown above), clicking search should yield results.

 

Congratulations! It rocks!

 

Peter P. Wang

peterpuwang@gmail.com