I had put together a list of search engine and other automated crawlers and bots together with their IP addresses which I was to not include in visitor usage statistics for a website, so only human visitors were tracked.
The list was very useful both for showing only human visitors, but also to minimise the database table entries as I noticed that often more crawlers and bots visited sites than human and database tables can become necessaries huge, unless you actually have a use for crawler and bot access.
I kept the list of crawlers and bots after the site was discontinued, as it may be useful and it is listed below.
If you use it and especially if you believe that it should be kept up to date and can add new crawlers and bots, then please get in touch.
crawl-66-249-71-248.googlebot.com, 66.249.71.248
crawl-66-249-66-38.googlebot.com, 66.249.66.38
crawl-66-249-65-142.googlebot.com, 66.249.65.142
crawl-66-249-65-210.googlebot.com, 66.249.65.210
crawl-66-249-66-201.googlebot.com, 66.249.66.201
crawl-66-249-65-4.googlebot.com, 66.249.65.4
crawl-66-249-65-102.googlebot.com, 66.249.65.102
livebot-65-55-208-24.search.live.com, 65.55.208.24
livebot-65-55-208-21.search.live.com, 65.55.208.21
livebot-65-55-213-39.search.live.com, 65.55.213.39
livebot-65-55-208-19.search.live.com, 65.55.208.19
livebot-65-55-208-20.search.live.com, 65.55.208.20
livebot-65-55-208-23.search.live.com, 65.55.208.23
livebot-65-55-208-22.search.live.com, 65.55.208.22
livebot-65-55-208-25.search.live.com, 65.55.208.25
livebot-65-55-212-19.search.live.com, 65.55.212.19
crawl-66-249-66-196.googlebot.com, 66.249.66.196
crawl-66-249-66-66.googlebot.com, 66.249.66.66
crawl-66-249-65-241.googlebot.com, 66.249.65.241
crawl-66-249-66-51.googlebot.com, 66.249.66.51
crawl-66-249-65-48.googlebot.com, 66.249.65.48
crawl-66-249-65-98.googlebot.com, 66.249.65.98
livebot-65-54-188-14.search.live.com, 65.54.188.14
livebot-65-55-208-18.search.live.com, 65.55.208.18
livebot-65-55-208-17.search.live.com, 65.55.208.17
livebot-65-54-188-15.search.live.com, 65.54.188.15
livebot-65-54-188-13.search.live.com, 65.54.188.13
livebot-65-54-188-16.search.live.com, 65.54.188.16
crawl13.exabot.com, 193.47.80.49
livebot-65-55-212-232.search.live.com, 65.55.212.232
livebot-65-55-213-36.search.live.com, 65.55.213.36
livebot-65-54-188-17.search.live.com, 65.54.188.17
livebot-207-46-98-77.search.live.com, 207.46.98.77
crawl-66-249-65-12.googlebot.com, 66.249.65.12
livebot-65-55-208-16.search.live.com, 65.55.208.16
crawl-66-249-65-11.googlebot.com, 66.249.65.11
crawl-66-249-65-45.googlebot.com, 66.249.65.45
crawl-66-249-66-48.googlebot.com, 66.249.66.48
crawl-66-249-66-229.googlebot.com, 66.249.66.229
crawl-66-249-65-227.googlebot.com, 66.249.65.227
crawl-66-249-65-112.googlebot.com, 66.249.65.112
livebot-207-46-98-76.search.live.com, 207.46.98.76
livebot-207-46-98-78.search.live.com, 207.46.98.78
crawl-66-249-65-106.googlebot.com, 66.249.65.106
crawl11.springright.com, 66.135.44.43
livebot-65-55-208-145.search.live.com, 65.55.208.145
crawler4.irl.cs.tamu.edu, 128.194.135.94
crawl14.springright.com, 66.135.44.46
livebot-65-55-212-233.search.live.com, 65.55.212.233
livebot-65-55-212-234.search.live.com, 65.55.212.234
crawl-66-249-66-142.googlebot.com, 66.249.66.142
crawl-66-249-65-34.googlebot.com, 66.249.65.34
crawl-66-249-65-39.googlebot.com, 66.249.65.39
livebot-65-55-208-124.search.live.com, 65.55.208.124
livebot-65-55-208-132.search.live.com, 65.55.208.132
livebot-65-55-208-125.search.live.com, 65.55.208.125
crawl-66-249-73-170.googlebot.com, 66.249.73.170
crawl-66-249-66-129.googlebot.com, 66.249.66.129
crawl-66-249-66-133.googlebot.com, 66.249.66.133
crawl-66-249-66-244.googlebot.com, 66.249.66.244
crawl-66-249-66-115.googlebot.com, 66.249.66.115
crawl-66-249-66-14.googlebot.com, 66.249.66.14
crawl-66-249-66-71.googlebot.com, 66.249.66.71
crawl-66-249-66-18.googlebot.com, 66.249.66.18
crawl-66-249-72-99.googlebot.com, 66.249.72.99
crawl-66-249-65-235.googlebot.com, 66.249.65.235
38.98.120.69, 38.98.120.69
crawl-66-249-66-10.googlebot.com, 66.249.66.10
crawl-66-249-66-131.googlebot.com, 66.249.66.131
crawl-66-249-72-83.googlebot.com, 66.249.72.83
crawl-66-249-65-145.googlebot.com, 66.249.65.145
crawl-66-249-66-102.googlebot.com, 66.249.66.102
livebot-65-55-209-122.search.live.com, 65.55.209.122
livebot-65-55-209-128.search.live.com, 65.55.209.128
livebot-65-55-209-121.search.live.com, 65.55.209.121
crawl-66-249-66-180.googlebot.com, 66.249.66.180
crawl-66-249-72-129.googlebot.com, 66.249.72.129
crawl-66-249-66-80.googlebot.com, 66.249.66.80
crawl-66-249-72-3.googlebot.com, 66.249.72.3
crawler100.ask.com, 65.214.45.100
38.99.44.106, 38.99.44.106
crawl-66-249-66-110.googlebot.com, 66.249.66.110
crawl-66-249-72-193.googlebot.com, 66.249.72.193
crawl-66-249-72-46.googlebot.com, 66.249.72.46
crawl-66-249-66-143.googlebot.com, 66.249.66.143
crawl-66-249-66-161.googlebot.com, 66.249.66.161
srv07.id-search.org, 67.159.44.232
srv03.id-search.org, 66.90.101.75
spider10.picsearch.com, 217.212.224.159
spider6.picsearch.com, 217.212.224.146
spider7.picsearch.com, 217.212.224.147
spider3.picsearch.com, 217.212.224.143
spider4.picsearch.com, 217.212.224.144
spider20.picsearch.com, 217.212.224.170
spider18.picsearch.com, 217.212.224.168
spider21.picsearch.com, 217.212.224.177
domU-12-31-37-00-02-81.usma3.compute.amazonaws.com, 72.44.62.118
domU-12-31-37-00-06-0B.usma3.compute.amazonaws.com, 72.44.58.245
crawl-66-249-65-173.googlebot.com, 66.249.65.173
crawl-16.cuill.com, 208.36.144.7
crawl-13.cuill.com, 38.99.44.104
208.36.144.6.ptr.us.xo.net, 208.36.144.6
crawl-66-249-72-205.googlebot.com, 66.249.72.205
xcrawl103.alexa.com, 64.208.172.175
crawl-66-249-72-43.googlebot.com, 66.249.72.43
egspd42135.ask.com, 65.214.44.116
livebot-65-55-212-235.search.live.com, 65.55.212.235
crawl-66-249-72-40.googlebot.com, 66.249.72.40
egspd42100.ask.com, 65.214.44.81
crawl-66-249-72-50.googlebot.com, 66.249.72.50
speedyspider.entireweb.com, 88.131.153.91
crawler2010.ask.com, 63.123.238.50
balin.seekbot.net, 195.27.115.51
crawler4026.ask.com, 65.214.45.135
ss110.dal0.gigablast.com, 38.114.104.120
livebot-65-55-212-46.search.live.com, 65.55.212.46
livebot-65-55-212-237.search.live.com, 65.55.212.237
crawl-66-249-72-163.googlebot.com, 66.249.72.163
ec2-67-202-22-47.compute-1.amazonaws.com, 67.202.22.47
ec2-67-202-26-120.compute-1.amazonaws.com, 67.202.26.120
s15252624.onlinehome-server.info, 87.106.129.54
crawl-66-249-72-135.googlebot.com, 66.249.72.135
xcrawl106.alexa.com, 64.208.172.178
crawl-66-249-73-111.googlebot.com, 66.249.73.111
crawl-66-249-65-205.googlebot.com, 66.249.65.205
crawl-66-249-66-208.googlebot.com, 66.249.66.208
crawl-66-249-66-112.googlebot.com, 66.249.66.112
crawl-66-249-65-38.googlebot.com, 66.249.65.38
crawl-66-249-65-42.googlebot.com, 66.249.65.42
crawl-66-249-72-18.googlebot.com, 66.249.72.18
crawl-66-249-72-195.googlebot.com, 66.249.72.195
spider25.picsearch.com, 217.212.224.181
crawl-66-249-72-106.googlebot.com, 66.249.72.106
crawl-66-249-72-198.googlebot.com, 66.249.72.198
crawl-66-249-66-177.googlebot.com, 66.249.66.177
crawl-66-249-66-145.googlebot.com, 66.249.66.145
crawl-66-249-66-3.googlebot.com, 66.249.66.3
spider23.picsearch.com, 217.212.224.179
crawl-66-249-66-228.googlebot.com, 66.249.66.228
crawl-66-249-66-168.googlebot.com, 66.249.66.168
crawl-66-249-65-105.googlebot.com, 66.249.65.105
crawl-66-249-66-19.googlebot.com, 66.249.66.19
crawl-66-249-66-33.googlebot.com, 66.249.66.33
crawl-66-249-66-11.googlebot.com, 66.249.66.11
192.168.1.41, 192.168.1.41
crawl-66-249-66-236.googlebot.com, 66.249.66.236
crawl-66-249-65-164.googlebot.com, 66.249.65.164
AAmiens-151-1-49-17.w83-192.abo.wanadoo.fr, 83.192.36.17
lns-bzn-23-82-248-100-152.adsl.proxad.net, 82.248.100.152
201-25-108-130.cbace301.ipd.brasiltelecom.net.br, 201.25.108.130
crawl-66-249-65-82.googlebot.com, 66.249.65.82
crawl-66-249-72-45.googlebot.com, 66.249.72.45
crawl-66-249-66-194.googlebot.com, 66.249.66.194
crawl-66-249-72-212.googlebot.com, 66.249.72.212
crawl-66-249-66-70.googlebot.com, 66.249.66.70
crawl-66-249-65-197.googlebot.com, 66.249.65.197
per92-7-82-236-196-90.fbx.proxad.net, 82.236.196.90
crawl-66-249-72-111.googlebot.com, 66.249.72.111
crawl-66-249-65-141.googlebot.com, 66.249.65.141
host220-59-dynamic.0-87-r.retail.telecomitalia.it, 87.0.59.220
host123-196-dynamic.58-82-r.retail.telecomitalia.it, 82.58.196.123
crawl-66-249-72-35.googlebot.com, 66.249.72.35
crawl-66-249-65-5.googlebot.com, 66.249.65.5
crawl-66-249-67-181.googlebot.com, 66.249.67.181
pool-72-90-71-132.syrcny.fios.verizon.net, 72.90.71.132
crawl-66-249-65-67.googlebot.com, 66.249.65.67
pool-71-168-122-164.cncdnh.fios.verizon.net, 71.168.122.164
77-101-62-103.cable.ubr03.craw.blueyonder.co.uk, 77.101.62.103
89-149-227-78.internetserviceteam.com, 89.149.227.78
crawl-66-249-65-208.googlebot.com, 66.249.65.208
p15201384.pureserver.info, 212.227.102.33
crawl-66-249-71-8.googlebot.com, 66.249.71.8
crawl-66-249-71-9.googlebot.com, 66.249.71.9
crawl-66-249-71-7.googlebot.com, 66.249.71.7
rbi0342.giga-dns.com, 193.37.152.242
77.91.224.12, 77.91.224.12
crawl-2.cuill.com, 38.99.13.123
crawl-66-249-72-48.googlebot.com, 66.249.72.48
223.Red-79-153-27.staticIP.rima-tde.net, 79.153.27.223
49.Red-83-44-60.dynamicIP.rima-tde.net, 83.44.60.49
crawler5017.ask.com, 66.235.124.17
crawler.kalooga.com, 195.210.57.83
c-76-24-48-135.hsd1.nh.comcast.net, 76.24.48.135
crawl-66-249-71-77.googlebot.com, 66.249.71.77
crawl-66-249-71-78.googlebot.com, 66.249.71.78
crawl-66-249-71-79.googlebot.com, 66.249.71.79
crawl-66-249-71-179.googlebot.com, 66.249.71.179
crawl-66-249-71-180.googlebot.com, 66.249.71.180
crawl-66-249-71-181.googlebot.com, 66.249.71.181
38.103.63.62, 38.103.63.62
205.209.170.18, 205.209.170.18
82-44-90-16.cable.ubr03.craw.blueyonder.co.uk, 82.44.90.16
crawl-66-249-71-152.googlebot.com, 66.249.71.152
crawl-66-249-71-151.googlebot.com, 66.249.71.151
crawl-66-249-71-150.googlebot.com, 66.249.71.150
ukspider3.wise-guys.nl, 82.94.176.143
llf520195.crawl.yahoo.net, 74.6.17.169
llf531267.crawl.yahoo.net, 72.30.79.99
llf520055.crawl.yahoo.net, 74.6.22.168
llf531151.crawl.yahoo.net, 72.30.142.88
77-96-83-75.cable.ubr03.craw.blueyonder.co.uk, 77.96.83.75
llf520051.crawl.yahoo.net, 74.6.22.180
llf531331.crawl.yahoo.net, 72.30.78.247
ANancy-257-1-131-51.w90-40.abo.wanadoo.fr, 90.40.162.51
llf531251.crawl.yahoo.net, 72.30.79.85
crawl-66-249-71-209.googlebot.com, 66.249.71.209
ec2-75-101-218-215.compute-1.amazonaws.com, 75.101.218.215
llf531049.crawl.yahoo.net, 72.30.142.243
crawl-66-249-71-85.googlebot.com, 66.249.71.85
crawl-66-249-71-138.googlebot.com, 66.249.71.138
ec2-174-129-100-129.compute-1.amazonaws.com, 174.129.100.129
crawl-66-249-71-3.googlebot.com, 66.249.71.3
llf520032.crawl.yahoo.net, 74.6.8.125
llf520009.crawl.yahoo.net, 74.6.8.115
crawl-66-249-71-238.googlebot.com, 66.249.71.238
llf520197.crawl.yahoo.net, 74.6.17.171
llf520073.crawl.yahoo.net, 74.6.22.183
llf520049.crawl.yahoo.net, 74.6.22.190
crawl10.dotnetdotcom.org, 208.115.111.251
llf520048.crawl.yahoo.net, 74.6.22.185
llf520017.crawl.yahoo.net, 74.6.8.121
llf531333.crawl.yahoo.net, 72.30.78.236
crawl-66-249-71-231.googlebot.com, 66.249.71.231
llf531378.crawl.yahoo.net, 72.30.65.36
crawl-66-249-71-247.googlebot.com, 66.249.71.247
llf531379.crawl.yahoo.net, 72.30.65.25
crawl-2c.cuil.com, 216.129.119.12
crawl-15c.cuil.com, 216.129.119.45
llf520109.crawl.yahoo.net, 74.6.22.107
crawl-15c.cuil.com, 38.99.13.116
llf531117.crawl.yahoo.net, 72.30.142.189
llf520101.crawl.yahoo.net, 74.6.22.122
crawl2.nat.svl.searchme.com, 208.111.154.16
crawl-66-249-66-45.googlebot.com, 66.249.66.45
crawl-66-249-71-182.googlebot.com, 66.249.71.182
crawl-66-249-71-146.googlebot.com, 66.249.71.146
b3091252.crawl.yahoo.net, 67.195.112.36
llf320020.crawl.yahoo.net, 67.195.37.93
crawl-66-249-71-47.googlebot.com, 66.249.71.47
crawl-13c.cuil.com, 216.129.119.43
llf531250.crawl.yahoo.net, 72.30.79.119
crawl-66-249-71-173.googlebot.com, 66.249.71.173
b3090794.crawl.yahoo.net, 67.195.115.88
crawl-66-249-72-132.googlebot.com, 66.249.72.132
crawl-66-249-72-130.googlebot.com, 66.249.72.130
crawl-15c.cuil.com, 67.218.116.162
94-169-190-143.cable.ubr06.craw.blueyonder.co.uk, 94.169.190.143
llf531069.crawl.yahoo.net, 72.30.142.218
llf531355.crawl.yahoo.net, 72.30.78.253
crawl-66-249-71-27.googlebot.com, 66.249.71.27
llf531050.crawl.yahoo.net, 72.30.142.24
llf531285.crawl.yahoo.net, 72.30.79.30
llf520173.crawl.yahoo.net, 74.6.17.177
llf531010.crawl.yahoo.net, 72.30.161.218
94-169-191-32.cable.ubr06.craw.blueyonder.co.uk, 94.169.191.32
llf531011.crawl.yahoo.net, 72.30.161.235
llf531050.crawl.yahoo.net, 72.30.142.248
crawl-66-249-65-73.googlebot.com, 66.249.65.73
crawl-66-249-65-76.googlebot.com, 66.249.65.76
b3091312.crawl.yahoo.net, 67.195.111.237
crawl-66-249-65-71.googlebot.com, 66.249.65.71
crawl-66-249-65-108.googlebot.com, 66.249.65.108
crawl-66-249-65-153.googlebot.com, 66.249.65.153
crawl-66-249-65-154.googlebot.com, 66.249.65.154
b3091289.crawl.yahoo.net, 67.195.111.252
llf531301.crawl.yahoo.net, 72.30.79.32
crawl-66-249-71-38.googlebot.com, 66.249.71.38
llf531207.crawl.yahoo.net, 72.30.81.184
b3091059.crawl.yahoo.net, 67.195.110.157
b3091131.crawl.yahoo.net, 67.195.112.152
crawl-66-249-71-207.googlebot.com, 66.249.71.207
crawl-66-249-65-83.googlebot.com, 66.249.65.83
crawl-66-249-65-21.googlebot.com, 66.249.65.21
crawl-66-249-65-7.googlebot.com, 66.249.65.7
crawl-66-249-65-177.googlebot.com, 66.249.65.177
crawl-66-249-71-153.googlebot.com, 66.249.71.153
spider08.yandex.ru, 87.250.255.242
crawl-66-249-65-37.googlebot.com, 66.249.65.37
crawl-66-249-65-183.googlebot.com, 66.249.65.183
b3090862.crawl.yahoo.net, 67.195.115.222
crawl-66-249-65-201.googlebot.com, 66.249.65.201
crawl-66-249-65-231.googlebot.com, 66.249.65.231
b3090847.crawl.yahoo.net, 67.195.115.241
crawl-66-249-71-193.googlebot.com, 66.249.71.193
b3090841.crawl.yahoo.net, 67.195.115.252
crawl-66-249-65-199.googlebot.com, 66.249.65.199
crawl-66-249-65-225.googlebot.com, 66.249.65.225
crawl4.dotnetdotcom.org, 208.115.111.245
b3090777.crawl.yahoo.net, 67.195.115.93
crawl1.dotnetdotcom.org, 208.115.111.242
cpc1-craw6-2-0-cust171.croy.cable.virginmedia.com, 94.169.190.172
cpc4-craw6-2-0-cust117.croy.cable.virginmedia.com, 92.233.66.118
crawl-66-249-65-217.googlebot.com, 66.249.65.217
llf320037.crawl.yahoo.net, 67.195.37.92
crawl-6c.cuil.com, 67.218.116.131
crawl-0c.cuil.com, 216.129.119.10
crawl-8c.cuil.com, 67.218.116.133
crawl-17c.cuil.com, 67.218.116.164
crawl-7c.cuil.com, 67.218.116.132
crawl-5c.cuil.com, 67.218.116.130
crawl-11c.cuil.com, 216.129.119.41
crawl-1c.cuil.com, 216.129.119.11
crawl-14c.cuil.com, 216.129.119.44
crawl-9c.cuil.com, 67.218.116.134
crawl-16c.cuil.com, 67.218.116.163
crawl-18c.cuil.com, 67.218.116.165
crawl-4c.cuil.com, 216.129.119.14
crawl-19c.cuil.com, 67.218.116.166
crawl-12c.cuil.com, 216.129.119.42
crawl-10c.cuil.com, 216.129.119.40
crawl-3c.cuil.com, 216.129.119.13
crawl-66-249-65-113.googlebot.com, 66.249.65.113
crawl-66-249-65-120.googlebot.com, 66.249.65.120
crawl-66-249-65-65.googlebot.com, 66.249.65.65
crawl-66-249-71-213.googlebot.com, 66.249.71.213
b3091151.crawl.yahoo.net, 67.195.112.179
crawl-66-249-65-124.googlebot.com, 66.249.65.124
crawl-66-249-65-87.googlebot.com, 66.249.65.87
crawl-66-249-65-122.googlebot.com, 66.249.65.122
crawl-66-249-71-54.googlebot.com, 66.249.71.54
crawl-66-249-65-121.googlebot.com, 66.249.65.121
crawl-66-249-65-167.googlebot.com, 66.249.65.167
crawl-66-249-65-77.googlebot.com, 66.249.65.77
crawl-66-249-65-103.googlebot.com, 66.249.65.103
88.131.106.30, 88.131.106.30
crawl-66-249-65-168.googlebot.com, 66.249.65.168
88.131.106.30, 88.131.106.31
b3090831.crawl.yahoo.net, 67.195.115.176
b3091116.crawl.yahoo.net, 67.195.112.223
b3091052.crawl.yahoo.net, 67.195.110.158
crawl-66-249-65-69.googlebot.com, 66.249.65.69
crawl-66-249-65-68.googlebot.com, 66.249.65.68
crawl-66-249-71-217.googlebot.com, 66.249.71.217
crawler-220.hurricane.ny.shopwiki.com, 209.51.162.220