All About Search Indexing Robots and Spiders
Search Tools explains how the search engine programs called "robots" or "spiders" work, and reviews related sites.
Http User Agent Index
An alphabetical list of user agents and the deployer behind them.
List of Robot Agent Strings
A list of Web robots with the identifying data they leave in Web site logs.
Psychedelix List of User-Agents
Large list of search engine spiders, similar Web robots, and Web browsers: their web-log identification and links to their originators.
Robots, Spiders and Other User Agents: a Resource for WebMasters
Searchable database of robots, spiders and other user agents for programs that surf the web.
Robots.txt Syntax Checker
Online tool for syntax verification to robots.txt files.
Search Engine IP Addresses
Lists IP addresses of search engine spiders. Can be searched by IP address. Also links to resources on spiders.
Spiders, Crawlers and Indexers Database
Searchable database of information on Web robots: name, useragent, status, description, and owner's name URL and email address.
The User-Agent Database
A list of spiders (robots or crawlers), browsers and harvesters, sorted into a searchable database.
The Web Robots Pages
FAQs about the programmes that index the Web, how to direct visiting robots, a database of robots, an archived mailing list, background reading, and related links.