Wikipedia:Spiders and bots policy

Search engine spiders
A huge portion of our traffic comes from referrals from search engines. Obviously, we want search engines to spider over the site and index us! These spiders are usually well-behaved; they take their time, obey the robots.txt file, and don't stress the server too much.

Other spiders
Other well-behaved spiders are also welcome.

But from time to time spiders are not friendly -- they don't obey the restrictions in robots.txt and overuse database-stressing dynamic functions, and/or request pages at so great a rate that they're preventing legitimate human users from getting at the database. If a spider is found to be causing that kind of trouble, the IP address will be banned from access to the site. Currently so banned are:
 * 144.167.21.15
 * 192.153.22.246
 * 202.69.76.19
 * 212.27.33.00/24
 * 24.130.248.43
 * 194.209.152.200
 * 80.192.68.91
 * 80.2.170.93
 * 209.208.186.2
 * 68.62.88.211
 * 203.175.70.118
 * 65.60.161.156
 * 81.86.203.137
 * 63.164.242.215
 * 66.147.154.3
 * 62.101.126.224
 * 209.125.45.130
 * 172.189.3.245
 * 61.30.127.4
 * 61.30.14.26
 * 137.230.1.11
 * 194.228.168.42

(These don't show up in Special:Ipblocklist, which blocks IPs from saving edits to the wiki.)

Bots that make edits to the wiki
Well-behaved bots are acceptable under some circumstances. Please see Bots.