Also see:  
http://www.getafreelancer.com/projects/Perl-Python/Web-Spider.html
 
Budget: $30-250
 
Description
Web spider require to collect UK urls from the internet and add to a MYSQL database after checking the WHOIS and adding the expiry date and removing duplicates
 
Spider needs a web interface that I log into and restart see progress etc. and connect to the database.
 
Database need to have functions of search by expiry date and extraction
 
Applicant needs to be familiar with Web Spiders or Web Scrapers and search engines and how they relate AND have example of previous similar work.
 
How or in what language the spider is written in only that the script is reliable and robust. Start Pages, number of child process is up to the developer
 
Linux based server