I had put together a list of search engine and other automated crawlers and bots together with their IP addresses which I was to not include in visitor usage statistics for a website, so only human visitors were tracked.
The list was very useful both for showing only human visitors, but also to minimise the database table entries as I noticed that often more crawlers and bots visited sites than human and database tables can become necessaries huge, unless you actually have a use for crawler and bot access.
I kept the list of crawlers and bots after the site was discontinued, as it may be useful and it is listed below.
If you use it and especially if you believe that it should be kept up to date and can add new crawlers and bots, then please get in touch.
crawl-66-249-71-248.googlebot.com, 66.249.71.248 crawl-66-249-66-38.googlebot.com, 66.249.66.38 crawl-66-249-65-142.googlebot.com, 66.249.65.142 crawl-66-249-65-210.googlebot.com, 66.249.65.210 crawl-66-249-66-201.googlebot.com, 66.249.66.201 crawl-66-249-65-4.googlebot.com, 66.249.65.4 crawl-66-249-65-102.googlebot.com, 66.249.65.102 livebot-65-55-208-24.search.live.com, 65.55.208.24 livebot-65-55-208-21.search.live.com, 65.55.208.21 livebot-65-55-213-39.search.live.com, 65.55.213.39 livebot-65-55-208-19.search.live.com, 65.55.208.19 livebot-65-55-208-20.search.live.com, 65.55.208.20 livebot-65-55-208-23.search.live.com, 65.55.208.23 livebot-65-55-208-22.search.live.com, 65.55.208.22 livebot-65-55-208-25.search.live.com, 65.55.208.25 livebot-65-55-212-19.search.live.com, 65.55.212.19 crawl-66-249-66-196.googlebot.com, 66.249.66.196 crawl-66-249-66-66.googlebot.com, 66.249.66.66 crawl-66-249-65-241.googlebot.com, 66.249.65.241 crawl-66-249-66-51.googlebot.com, 66.249.66.51 crawl-66-249-65-48.googlebot.com, 66.249.65.48 crawl-66-249-65-98.googlebot.com, 66.249.65.98 livebot-65-54-188-14.search.live.com, 65.54.188.14 livebot-65-55-208-18.search.live.com, 65.55.208.18 livebot-65-55-208-17.search.live.com, 65.55.208.17 livebot-65-54-188-15.search.live.com, 65.54.188.15 livebot-65-54-188-13.search.live.com, 65.54.188.13 livebot-65-54-188-16.search.live.com, 65.54.188.16 crawl13.exabot.com, 193.47.80.49 livebot-65-55-212-232.search.live.com, 65.55.212.232 livebot-65-55-213-36.search.live.com, 65.55.213.36 livebot-65-54-188-17.search.live.com, 65.54.188.17 livebot-207-46-98-77.search.live.com, 207.46.98.77 crawl-66-249-65-12.googlebot.com, 66.249.65.12 livebot-65-55-208-16.search.live.com, 65.55.208.16 crawl-66-249-65-11.googlebot.com, 66.249.65.11 crawl-66-249-65-45.googlebot.com, 66.249.65.45 crawl-66-249-66-48.googlebot.com, 66.249.66.48 crawl-66-249-66-229.googlebot.com, 66.249.66.229 crawl-66-249-65-227.googlebot.com, 66.249.65.227 crawl-66-249-65-112.googlebot.com, 66.249.65.112 livebot-207-46-98-76.search.live.com, 207.46.98.76 livebot-207-46-98-78.search.live.com, 207.46.98.78 crawl-66-249-65-106.googlebot.com, 66.249.65.106 crawl11.springright.com, 66.135.44.43 livebot-65-55-208-145.search.live.com, 65.55.208.145 crawler4.irl.cs.tamu.edu, 128.194.135.94 crawl14.springright.com, 66.135.44.46 livebot-65-55-212-233.search.live.com, 65.55.212.233 livebot-65-55-212-234.search.live.com, 65.55.212.234 crawl-66-249-66-142.googlebot.com, 66.249.66.142 crawl-66-249-65-34.googlebot.com, 66.249.65.34 crawl-66-249-65-39.googlebot.com, 66.249.65.39 livebot-65-55-208-124.search.live.com, 65.55.208.124 livebot-65-55-208-132.search.live.com, 65.55.208.132 livebot-65-55-208-125.search.live.com, 65.55.208.125 crawl-66-249-73-170.googlebot.com, 66.249.73.170 crawl-66-249-66-129.googlebot.com, 66.249.66.129 crawl-66-249-66-133.googlebot.com, 66.249.66.133 crawl-66-249-66-244.googlebot.com, 66.249.66.244 crawl-66-249-66-115.googlebot.com, 66.249.66.115 crawl-66-249-66-14.googlebot.com, 66.249.66.14 crawl-66-249-66-71.googlebot.com, 66.249.66.71 crawl-66-249-66-18.googlebot.com, 66.249.66.18 crawl-66-249-72-99.googlebot.com, 66.249.72.99 crawl-66-249-65-235.googlebot.com, 66.249.65.235 38.98.120.69, 38.98.120.69 crawl-66-249-66-10.googlebot.com, 66.249.66.10 crawl-66-249-66-131.googlebot.com, 66.249.66.131 crawl-66-249-72-83.googlebot.com, 66.249.72.83 crawl-66-249-65-145.googlebot.com, 66.249.65.145 crawl-66-249-66-102.googlebot.com, 66.249.66.102 livebot-65-55-209-122.search.live.com, 65.55.209.122 livebot-65-55-209-128.search.live.com, 65.55.209.128 livebot-65-55-209-121.search.live.com, 65.55.209.121 crawl-66-249-66-180.googlebot.com, 66.249.66.180 crawl-66-249-72-129.googlebot.com, 66.249.72.129 crawl-66-249-66-80.googlebot.com, 66.249.66.80 crawl-66-249-72-3.googlebot.com, 66.249.72.3 crawler100.ask.com, 65.214.45.100 38.99.44.106, 38.99.44.106 crawl-66-249-66-110.googlebot.com, 66.249.66.110 crawl-66-249-72-193.googlebot.com, 66.249.72.193 crawl-66-249-72-46.googlebot.com, 66.249.72.46 crawl-66-249-66-143.googlebot.com, 66.249.66.143 crawl-66-249-66-161.googlebot.com, 66.249.66.161 srv07.id-search.org, 67.159.44.232 srv03.id-search.org, 66.90.101.75 spider10.picsearch.com, 217.212.224.159 spider6.picsearch.com, 217.212.224.146 spider7.picsearch.com, 217.212.224.147 spider3.picsearch.com, 217.212.224.143 spider4.picsearch.com, 217.212.224.144 spider20.picsearch.com, 217.212.224.170 spider18.picsearch.com, 217.212.224.168 spider21.picsearch.com, 217.212.224.177 domU-12-31-37-00-02-81.usma3.compute.amazonaws.com, 72.44.62.118 domU-12-31-37-00-06-0B.usma3.compute.amazonaws.com, 72.44.58.245 crawl-66-249-65-173.googlebot.com, 66.249.65.173 crawl-16.cuill.com, 208.36.144.7 crawl-13.cuill.com, 38.99.44.104 208.36.144.6.ptr.us.xo.net, 208.36.144.6 crawl-66-249-72-205.googlebot.com, 66.249.72.205 xcrawl103.alexa.com, 64.208.172.175 crawl-66-249-72-43.googlebot.com, 66.249.72.43 egspd42135.ask.com, 65.214.44.116 livebot-65-55-212-235.search.live.com, 65.55.212.235 crawl-66-249-72-40.googlebot.com, 66.249.72.40 egspd42100.ask.com, 65.214.44.81 crawl-66-249-72-50.googlebot.com, 66.249.72.50 speedyspider.entireweb.com, 88.131.153.91 crawler2010.ask.com, 63.123.238.50 balin.seekbot.net, 195.27.115.51 crawler4026.ask.com, 65.214.45.135 ss110.dal0.gigablast.com, 38.114.104.120 livebot-65-55-212-46.search.live.com, 65.55.212.46 livebot-65-55-212-237.search.live.com, 65.55.212.237 crawl-66-249-72-163.googlebot.com, 66.249.72.163 ec2-67-202-22-47.compute-1.amazonaws.com, 67.202.22.47 ec2-67-202-26-120.compute-1.amazonaws.com, 67.202.26.120 s15252624.onlinehome-server.info, 87.106.129.54 crawl-66-249-72-135.googlebot.com, 66.249.72.135 xcrawl106.alexa.com, 64.208.172.178 crawl-66-249-73-111.googlebot.com, 66.249.73.111 crawl-66-249-65-205.googlebot.com, 66.249.65.205 crawl-66-249-66-208.googlebot.com, 66.249.66.208 crawl-66-249-66-112.googlebot.com, 66.249.66.112 crawl-66-249-65-38.googlebot.com, 66.249.65.38 crawl-66-249-65-42.googlebot.com, 66.249.65.42 crawl-66-249-72-18.googlebot.com, 66.249.72.18 crawl-66-249-72-195.googlebot.com, 66.249.72.195 spider25.picsearch.com, 217.212.224.181 crawl-66-249-72-106.googlebot.com, 66.249.72.106 crawl-66-249-72-198.googlebot.com, 66.249.72.198 crawl-66-249-66-177.googlebot.com, 66.249.66.177 crawl-66-249-66-145.googlebot.com, 66.249.66.145 crawl-66-249-66-3.googlebot.com, 66.249.66.3 spider23.picsearch.com, 217.212.224.179 crawl-66-249-66-228.googlebot.com, 66.249.66.228 crawl-66-249-66-168.googlebot.com, 66.249.66.168 crawl-66-249-65-105.googlebot.com, 66.249.65.105 crawl-66-249-66-19.googlebot.com, 66.249.66.19 crawl-66-249-66-33.googlebot.com, 66.249.66.33 crawl-66-249-66-11.googlebot.com, 66.249.66.11 192.168.1.41, 192.168.1.41 crawl-66-249-66-236.googlebot.com, 66.249.66.236 crawl-66-249-65-164.googlebot.com, 66.249.65.164 AAmiens-151-1-49-17.w83-192.abo.wanadoo.fr, 83.192.36.17 lns-bzn-23-82-248-100-152.adsl.proxad.net, 82.248.100.152 201-25-108-130.cbace301.ipd.brasiltelecom.net.br, 201.25.108.130 crawl-66-249-65-82.googlebot.com, 66.249.65.82 crawl-66-249-72-45.googlebot.com, 66.249.72.45 crawl-66-249-66-194.googlebot.com, 66.249.66.194 crawl-66-249-72-212.googlebot.com, 66.249.72.212 crawl-66-249-66-70.googlebot.com, 66.249.66.70 crawl-66-249-65-197.googlebot.com, 66.249.65.197 per92-7-82-236-196-90.fbx.proxad.net, 82.236.196.90 crawl-66-249-72-111.googlebot.com, 66.249.72.111 crawl-66-249-65-141.googlebot.com, 66.249.65.141 host220-59-dynamic.0-87-r.retail.telecomitalia.it, 87.0.59.220 host123-196-dynamic.58-82-r.retail.telecomitalia.it, 82.58.196.123 crawl-66-249-72-35.googlebot.com, 66.249.72.35 crawl-66-249-65-5.googlebot.com, 66.249.65.5 crawl-66-249-67-181.googlebot.com, 66.249.67.181 pool-72-90-71-132.syrcny.fios.verizon.net, 72.90.71.132 crawl-66-249-65-67.googlebot.com, 66.249.65.67 pool-71-168-122-164.cncdnh.fios.verizon.net, 71.168.122.164 77-101-62-103.cable.ubr03.craw.blueyonder.co.uk, 77.101.62.103 89-149-227-78.internetserviceteam.com, 89.149.227.78 crawl-66-249-65-208.googlebot.com, 66.249.65.208 p15201384.pureserver.info, 212.227.102.33 crawl-66-249-71-8.googlebot.com, 66.249.71.8 crawl-66-249-71-9.googlebot.com, 66.249.71.9 crawl-66-249-71-7.googlebot.com, 66.249.71.7 rbi0342.giga-dns.com, 193.37.152.242 77.91.224.12, 77.91.224.12 crawl-2.cuill.com, 38.99.13.123 crawl-66-249-72-48.googlebot.com, 66.249.72.48 223.Red-79-153-27.staticIP.rima-tde.net, 79.153.27.223 49.Red-83-44-60.dynamicIP.rima-tde.net, 83.44.60.49 crawler5017.ask.com, 66.235.124.17 crawler.kalooga.com, 195.210.57.83 c-76-24-48-135.hsd1.nh.comcast.net, 76.24.48.135 crawl-66-249-71-77.googlebot.com, 66.249.71.77 crawl-66-249-71-78.googlebot.com, 66.249.71.78 crawl-66-249-71-79.googlebot.com, 66.249.71.79 crawl-66-249-71-179.googlebot.com, 66.249.71.179 crawl-66-249-71-180.googlebot.com, 66.249.71.180 crawl-66-249-71-181.googlebot.com, 66.249.71.181 38.103.63.62, 38.103.63.62 205.209.170.18, 205.209.170.18 82-44-90-16.cable.ubr03.craw.blueyonder.co.uk, 82.44.90.16 crawl-66-249-71-152.googlebot.com, 66.249.71.152 crawl-66-249-71-151.googlebot.com, 66.249.71.151 crawl-66-249-71-150.googlebot.com, 66.249.71.150 ukspider3.wise-guys.nl, 82.94.176.143 llf520195.crawl.yahoo.net, 74.6.17.169 llf531267.crawl.yahoo.net, 72.30.79.99 llf520055.crawl.yahoo.net, 74.6.22.168 llf531151.crawl.yahoo.net, 72.30.142.88 77-96-83-75.cable.ubr03.craw.blueyonder.co.uk, 77.96.83.75 llf520051.crawl.yahoo.net, 74.6.22.180 llf531331.crawl.yahoo.net, 72.30.78.247 ANancy-257-1-131-51.w90-40.abo.wanadoo.fr, 90.40.162.51 llf531251.crawl.yahoo.net, 72.30.79.85 crawl-66-249-71-209.googlebot.com, 66.249.71.209 ec2-75-101-218-215.compute-1.amazonaws.com, 75.101.218.215 llf531049.crawl.yahoo.net, 72.30.142.243 crawl-66-249-71-85.googlebot.com, 66.249.71.85 crawl-66-249-71-138.googlebot.com, 66.249.71.138 ec2-174-129-100-129.compute-1.amazonaws.com, 174.129.100.129 crawl-66-249-71-3.googlebot.com, 66.249.71.3 llf520032.crawl.yahoo.net, 74.6.8.125 llf520009.crawl.yahoo.net, 74.6.8.115 crawl-66-249-71-238.googlebot.com, 66.249.71.238 llf520197.crawl.yahoo.net, 74.6.17.171 llf520073.crawl.yahoo.net, 74.6.22.183 llf520049.crawl.yahoo.net, 74.6.22.190 crawl10.dotnetdotcom.org, 208.115.111.251 llf520048.crawl.yahoo.net, 74.6.22.185 llf520017.crawl.yahoo.net, 74.6.8.121 llf531333.crawl.yahoo.net, 72.30.78.236 crawl-66-249-71-231.googlebot.com, 66.249.71.231 llf531378.crawl.yahoo.net, 72.30.65.36 crawl-66-249-71-247.googlebot.com, 66.249.71.247 llf531379.crawl.yahoo.net, 72.30.65.25 crawl-2c.cuil.com, 216.129.119.12 crawl-15c.cuil.com, 216.129.119.45 llf520109.crawl.yahoo.net, 74.6.22.107 crawl-15c.cuil.com, 38.99.13.116 llf531117.crawl.yahoo.net, 72.30.142.189 llf520101.crawl.yahoo.net, 74.6.22.122 crawl2.nat.svl.searchme.com, 208.111.154.16 crawl-66-249-66-45.googlebot.com, 66.249.66.45 crawl-66-249-71-182.googlebot.com, 66.249.71.182 crawl-66-249-71-146.googlebot.com, 66.249.71.146 b3091252.crawl.yahoo.net, 67.195.112.36 llf320020.crawl.yahoo.net, 67.195.37.93 crawl-66-249-71-47.googlebot.com, 66.249.71.47 crawl-13c.cuil.com, 216.129.119.43 llf531250.crawl.yahoo.net, 72.30.79.119 crawl-66-249-71-173.googlebot.com, 66.249.71.173 b3090794.crawl.yahoo.net, 67.195.115.88 crawl-66-249-72-132.googlebot.com, 66.249.72.132 crawl-66-249-72-130.googlebot.com, 66.249.72.130 crawl-15c.cuil.com, 67.218.116.162 94-169-190-143.cable.ubr06.craw.blueyonder.co.uk, 94.169.190.143 llf531069.crawl.yahoo.net, 72.30.142.218 llf531355.crawl.yahoo.net, 72.30.78.253 crawl-66-249-71-27.googlebot.com, 66.249.71.27 llf531050.crawl.yahoo.net, 72.30.142.24 llf531285.crawl.yahoo.net, 72.30.79.30 llf520173.crawl.yahoo.net, 74.6.17.177 llf531010.crawl.yahoo.net, 72.30.161.218 94-169-191-32.cable.ubr06.craw.blueyonder.co.uk, 94.169.191.32 llf531011.crawl.yahoo.net, 72.30.161.235 llf531050.crawl.yahoo.net, 72.30.142.248 crawl-66-249-65-73.googlebot.com, 66.249.65.73 crawl-66-249-65-76.googlebot.com, 66.249.65.76 b3091312.crawl.yahoo.net, 67.195.111.237 crawl-66-249-65-71.googlebot.com, 66.249.65.71 crawl-66-249-65-108.googlebot.com, 66.249.65.108 crawl-66-249-65-153.googlebot.com, 66.249.65.153 crawl-66-249-65-154.googlebot.com, 66.249.65.154 b3091289.crawl.yahoo.net, 67.195.111.252 llf531301.crawl.yahoo.net, 72.30.79.32 crawl-66-249-71-38.googlebot.com, 66.249.71.38 llf531207.crawl.yahoo.net, 72.30.81.184 b3091059.crawl.yahoo.net, 67.195.110.157 b3091131.crawl.yahoo.net, 67.195.112.152 crawl-66-249-71-207.googlebot.com, 66.249.71.207 crawl-66-249-65-83.googlebot.com, 66.249.65.83 crawl-66-249-65-21.googlebot.com, 66.249.65.21 crawl-66-249-65-7.googlebot.com, 66.249.65.7 crawl-66-249-65-177.googlebot.com, 66.249.65.177 crawl-66-249-71-153.googlebot.com, 66.249.71.153 spider08.yandex.ru, 87.250.255.242 crawl-66-249-65-37.googlebot.com, 66.249.65.37 crawl-66-249-65-183.googlebot.com, 66.249.65.183 b3090862.crawl.yahoo.net, 67.195.115.222 crawl-66-249-65-201.googlebot.com, 66.249.65.201 crawl-66-249-65-231.googlebot.com, 66.249.65.231 b3090847.crawl.yahoo.net, 67.195.115.241 crawl-66-249-71-193.googlebot.com, 66.249.71.193 b3090841.crawl.yahoo.net, 67.195.115.252 crawl-66-249-65-199.googlebot.com, 66.249.65.199 crawl-66-249-65-225.googlebot.com, 66.249.65.225 crawl4.dotnetdotcom.org, 208.115.111.245 b3090777.crawl.yahoo.net, 67.195.115.93 crawl1.dotnetdotcom.org, 208.115.111.242 cpc1-craw6-2-0-cust171.croy.cable.virginmedia.com, 94.169.190.172 cpc4-craw6-2-0-cust117.croy.cable.virginmedia.com, 92.233.66.118 crawl-66-249-65-217.googlebot.com, 66.249.65.217 llf320037.crawl.yahoo.net, 67.195.37.92 crawl-6c.cuil.com, 67.218.116.131 crawl-0c.cuil.com, 216.129.119.10 crawl-8c.cuil.com, 67.218.116.133 crawl-17c.cuil.com, 67.218.116.164 crawl-7c.cuil.com, 67.218.116.132 crawl-5c.cuil.com, 67.218.116.130 crawl-11c.cuil.com, 216.129.119.41 crawl-1c.cuil.com, 216.129.119.11 crawl-14c.cuil.com, 216.129.119.44 crawl-9c.cuil.com, 67.218.116.134 crawl-16c.cuil.com, 67.218.116.163 crawl-18c.cuil.com, 67.218.116.165 crawl-4c.cuil.com, 216.129.119.14 crawl-19c.cuil.com, 67.218.116.166 crawl-12c.cuil.com, 216.129.119.42 crawl-10c.cuil.com, 216.129.119.40 crawl-3c.cuil.com, 216.129.119.13 crawl-66-249-65-113.googlebot.com, 66.249.65.113 crawl-66-249-65-120.googlebot.com, 66.249.65.120 crawl-66-249-65-65.googlebot.com, 66.249.65.65 crawl-66-249-71-213.googlebot.com, 66.249.71.213 b3091151.crawl.yahoo.net, 67.195.112.179 crawl-66-249-65-124.googlebot.com, 66.249.65.124 crawl-66-249-65-87.googlebot.com, 66.249.65.87 crawl-66-249-65-122.googlebot.com, 66.249.65.122 crawl-66-249-71-54.googlebot.com, 66.249.71.54 crawl-66-249-65-121.googlebot.com, 66.249.65.121 crawl-66-249-65-167.googlebot.com, 66.249.65.167 crawl-66-249-65-77.googlebot.com, 66.249.65.77 crawl-66-249-65-103.googlebot.com, 66.249.65.103 88.131.106.30, 88.131.106.30 crawl-66-249-65-168.googlebot.com, 66.249.65.168 88.131.106.30, 88.131.106.31 b3090831.crawl.yahoo.net, 67.195.115.176 b3091116.crawl.yahoo.net, 67.195.112.223 b3091052.crawl.yahoo.net, 67.195.110.158 crawl-66-249-65-69.googlebot.com, 66.249.65.69 crawl-66-249-65-68.googlebot.com, 66.249.65.68 crawl-66-249-71-217.googlebot.com, 66.249.71.217 crawler-220.hurricane.ny.shopwiki.com, 209.51.162.220
Leave a Reply