-
-
Notifications
You must be signed in to change notification settings - Fork 208
/
googlebot.txt
276 lines (276 loc) 路 29.2 KB
/
googlebot.txt
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)
Mozilla/5.0 (Linux; Android 6.0.1; Nexus 5X Build/MMB29P) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/41.0.2272.96 Mobile Safari/537.36 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)
Mozilla/5.0 (compatible; Googlebot/2.1; startmebot/1.0; +https://start.me/bot)
Mozilla/5.0 (iPhone; CPU iPhone OS 8_3 like Mac OS X) AppleWebKit/600.1.4 (KHTML, like Gecko) Version/8.0 Mobile/12F70 Safari/600.1.4 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)
Googlebot/2.1 (+http://www.googlebot.com/bot.html)
Googlebot/2.1 (+http://www.google.com/bot.html)
Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko; Google Page Speed Insights) Chrome/27.0.1453 Safari/537.36 GoogleBot/2.1
Mozilla/5.0 (iPhone; CPU iPhone OS 6_0_1 like Mac OS X) AppleWebKit/537.36 (KHTML, like Gecko; Google Page Speed Insights) Version/6.0 Mobile/10A525 Safari/8536.25 GoogleBot/2.1
Mozilla/5.0 (compatible; Googlebot/2.1; http://www.google.com/bot.html)
Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; Google Web Preview Analytics) Chrome/27.0.1453 Safari/537.36 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)
Mozilla/5.0 (compatible;acapbot/0.1;treat like Googlebot)
Mozilla/5.0 (compatible; Googlebot/2.1; +http://import.io)
Googlebot
Googlebot-Image/1.0
Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; Googlebot/2.1; +http://www.google.com/bot.html) Safari/537.36
Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.4 (KHTML, like Gecko; Google Page Speed Insights) Chrome/22.0.1229 Safari/537.4 GoogleBot/2.1
Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; Google Web Preview Analytics) Chrome/41.0.2272.118 Safari/537.36 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)
OnPageBot (compatible; Googlebot 2.1; +https://bot.onpage.org/)
Google Crawler: Googlebot/2.1 (+http://www.google.com/bot.html)
Googlebot/2.1; +http://www.google.com/bot.html)
Mozilla/5.0 (compatible; Googlebot/2.1 +http://www.googlebot.com/bot.html)
Mozilla/5.0 (compatible;acapbot/0.1.;treat like Googlebot)
Mozilla/5.0 (iPhone; CPU iPhone OS 6_0 like Mac OS X) AppleWebKit/536.26 (KHTML, like Gecko) Version/6.0 Mobile/10A5376e Safari/8536.25 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)
Mozilla/5.0 (iPhone; U; CPU iPhone 0S 3.0 like Mac 0S X; en-us; compatible; Googlebot/2.1; http://www.google.com/bot.html; AppleWebKit/528.18(KHTML,like Gecko) Version/4.0 Mobile/7A341 Safari/528.16 UNTRUSTED/1.0
Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html
Apple-iPhone4C1/1001.523 (compatible;acapbot/0.1;treat like Googlebot)
Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)/Nutch-2.3.1
Googlebot-News
Mozilla/5.0 (Linux; Android 6.0.1; Nexus 5X Build/MMB29P) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/27.0.1453 Mobile Safari/537.36 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)
GoogleBot/3.0
Mozilla/5.0 (compatible; Googlebot/2.1; https://deepcrawl.com/bot)
Mozilla/5.0 (compatible; Googlebot/2.1; https://www.deepcrawl.com/bot)
Googlebot-Video/1.0
Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html) UNTRUSTED/1.0
Mozilla/5.0 (compatible; WebHistoryBot/1.2.1 IS NOT SE bot like Googlebot/2.1; +http://www.google.com/bot.html,Yahoo! Slurp or Bingbot)
Googlebot/2.1 (+http://www.googlebot.com/bot.html) <iframe src="whatismyip.com"></iframe>
Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/67.0.3366.99 Safari/537.36 (compatible; Googlebot/2.1; SeoakademiyaBot/1.0; +http://seo-akademiya.com/bot.html)
Googlebot/2.1
Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html )
Googlebot 2.1
Mozilla/5.0 (iPhone; CPU iPhone OS 12_0 like Mac OS X) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/12.0 Mobile/15E148 Safari/604.1 (GoogleBot)
Googlebot/2.1; http://www.google.com/bot.html
Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/66.0.3359.139 Safari/537.36 Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)
Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html) AlexaToolbar/alxf-2.17
Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/67.0.3396.99 Safari/537.36 Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)
Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html) FirePHP/0.7.4
NokiaC1-01/2.0 (06.15) Profile/MIDP-2.1 Configuration/CLDC-1.1 Mozilla/5.0 (iPhone; U; CPU iPhone 0S 3.0 like Mac 0S X; en-us; compatible; Googlebot/2.1; http://www.google.com/bot.html; AppleWebKit/528.18(KHTML,like Gecko) Version/4.0 Mobile/7A341 Safari/528.16 UNTRUSTED/1.0
Mozilla/5.0 (iPhone; CPU iPhone OS 6_0_1 like Mac OS X) AppleWebKit/537.4 (KHTML, like Gecko; Google Page Speed Insights) Version/6.0 Mobile/10A525 Safari/8536.25 GoogleBot/2.1
Googlebot (compatible; Googlebot/2.1; +http://www.google.com/bot.html)
Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)
Googlebot/2.1 (+http://www.googlebot.com/bot.html) (Windows NT 10.0; WOW64; Trident/7.0; rv:11.0) like Gecko
BotThief (compatible; Googlebot/2.1; Location RU)
Mozilla/5.0+(compatible;+googlebot/2.1;++http://www.google.com/bot.html)
Mozilla/5.0 (compatible; Googlebot/2.1; actually this is http://ismyblogworking.com/)
Mozilla/5.0 (+http://www.googlebot.com/bot.html; Windows NT 6.2) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/46.0.2486.0 Safari/537.36 Edge/13.9200
http://www.googlebot.com/bot.html
* Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)
Mozilla/5.0 (compatible; Googlebot/2.1; http://deepcrawl.co.uk/bot.html)
Googlebot/2.1+(+http://www.googlebot.com/bot.html)
Googlebot/2.1 (+http://www.googlebot.com/bot.html
Mozilla/5.0 (Windows NT 10.0; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/46.0.2490.86 Safari/537.36 Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)
Mozilla/5.0 (iPhone; CPU iPhone OS 6_0 like Mac OS X) AppleWebKit/536.26 (KHTML, like Gecko) Version/6.0 Mobile/10A5376e Safari/8536.25 (compatible; Googlebot/2.1; +https://www.deepcrawl.com/bot)
Mozilla/5.0 (iPhone; U; CPU iPhone 0S 6.0 like Mac 0S X; en-us; compatible; Googlebot/2.1; http://www.google.com/bot.html; AppleWebKit/528.18(KHTML,like Gecko) Version/4.0 Mobile/7A341 Safari/528.16 UNTRUSTED/1.0
Aensland Mobile Browser /5.0 (iPhone; CPU iPhone OS 7_1 like Mac OS X) AppleWebKit/536.26 (KHTML, like Gecko) Version/6.0 Mobile/10A5376e Safari/8536.25 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)
66.249.64.13 - crawl-66-249-64-13.googlebot.com
Mozilla/5.0 (compatible; Googlebot/2.1; +https://whatis.contentkingapp.com/)
Mozilla/5.0 (compatible; Googlebot/2.1; http://www.google.com/bot.html)
Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; Googlebot/2.1; +http://www.google.com/bot.html) Safari/537.36.
Mozilla/5.0 (compatible; Googlebot/3.2; +http://www.facebook.com)
Googlebot/2.1 Compatible: http://www.googlebot.com/bot.html
Googlebot/2.1 ( http://www.googlebot.com/bot.html)
Googlebot/2.1" "Compatible"="+http://www.googlebot.com/bot.html
Mozilla/5.0 (compatible; Googlebot/2.1
Googlebot/2.1 (http://www.googlebot.com/bot.html)
Mozilla/5.0 (Windows NT 10.0; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/57.0.2987.133 Safari/537.36 Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)
Mozilla/5.0 (Windows NT 6.1; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/64.0.3282.186 Safari/537.36 Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)
Googlebot/2.1; + http://www.google.com/bot.html
Mozilla/5.0 (compatible; Googlebot/wannabe; +https://xfrocks.com/resources/30/)
Mozilla/5.0 (Windows NT 6.1; WOW64; Trident/7.0; rv:11.0) like Gecko (compatible; Googlebot/2.1; +http://www.google.com/bot.html)
(compatible; Googlebot/2.1; http://www.google.com/bot.html)
Mozilla/5.0 (compatible; Googlebot/2.1; +www.google.com/bot.html)
Mozilla/5.0 (iPhone; CPU iPhone OS 6_0 like Mac OS X) AppleWebKit/536.26 (KHTML, like Gecko) Version/6.0 Mobile/10A5376e Safari/8536.25 (compatible; Googlebot/2.1; +https://deepcrawl.com/bot)
Mozilla/5.0 (compatible; Macintosh; Intel Mac OS X) AppleWebKit/534.46.0 (KHTML, like Gecko) Googlebot/2.1 Safari/7534.34
Googlebot-News (2.3.4.1, ruby 1.9.3 (2014-05-14))
Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.11 (KHTML, like Gecko) Chrome/23.0.1271.97 Safari/537.11 Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)
Mozilla/5.0 (Windows NT 6.1; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/52.0.2743.116 Safari/537.36 Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)
Googlebot/2.1 (+http://www.google.com/bot.html
Mozilla/5.0 (Macintosh; Intel Mac OS X 10_13_1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/64.0.3282.186 Safari/537.36 Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)
crawl-66-249-66-1.googlebot.com
Googlebot-Image/1.0 (+http://www.googlebot.com/bot.html)
Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/69.0.3497.100 Safari/537.36 Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html) Append?: Append
GoogleBot2/1
Googlebot/Nutch-1.7
Mozilla/5.0 (compatible; Googlebot/20.1; +http://www.google.com/bot.html)
Mozilla/5.0 (Macintosh; Intel Mac OS X 10_12_5) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/60.0.3112.78 Safari/537.36 Googlebot-News
Googlebot/4.0 (Blackberry; ExtremeOS 10_3_1 PPC; rv:12.6) Crocodile/20040327 Gridspider/12.6
Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/66.0.3359.181 Safari/537.36 Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)
Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html) https://deepcrawl.com/bot
Nokia2700c-2/2.0 (09.98) Profile/MIDP-2.1 Configuration/CLDC-1.1 Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)/UCWEB8.2.0.116/69/352 UNTRUSTED/1.0
Mozilla/5.0 (compatible; Googlebot/2.1;)
Mozilla/4.0 (Googlebot/2.1 (+http://www.googlebot.com/bot.html); MSIE 6.0; )
{ "ua": "Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; Googlebot/2.1; +http://www.google.com/bot.html) Safari/537.36", "browser": { "name": "Safari" }, "engine": { "version": "537.36", "name": "WebKit" }, "os": {}, "device": {}, "cpu": {} }
Mozilla/5.0 (Windows NT 10.0; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/51.0.2704.103 Safari/537.36 Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)
Mozilla/5.0 (Linux; Android 2.3.2; Nexus One; U; id-en) Build/FRF9 applewebkit/523.18 presto/2.2.1.40 Version/4.0 mobile safari/523.18 Googlebot/20.1.254.89 + http://www.google.com/bot.html UCBrowser8.4.0.159/69/352 UNTRUSTED/1.0
Mozilla/5.0 (Linux; Android 6.0.1; Nexus 5X Build/MMB29P) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/41.0.2272.96 Mobile Safari/537.36 (compatible; Googlebot/2.1; http://www.google.com/bot.html)
Mozilla/5.0 (Windows NT 6.1) AppleWebKit/537.31 (KHTML, like Gecko) Chrome/26.2.2.0 Safari/537.31 Googlebot/2.1 (+http://www.googlebot.com/bot.html)
Mozilla/11.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)
GooglebotNSA
Mozilla/5.0 (compatible; Googlebot/6.1; +http://www.google.com/bot.html)
Mozilla/5.0 (Android; U; Android; en-id; compatible; Googlebot/10.0) AppleWebKit/534.1+ (KHTML, Like Gecko) Version/6.0.0.141 Mobile Safari/534.1+ UNTRUSTED/1.0
Googlebot/2.X (+http://www.googlebot.com/bot.html)
Nokia6303iclassic/5.0 (10.80) Profile/MIDP-2.1 Configuration/CLDC-1.1 Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html) UNTRUSTED/1.0
Mozilla/5.0 (Windows NT 10.0; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/69.0.3497.102 Safari/537.36 Vivaldi/2.0.1309.37 Googlebot/2.1 (+http://www.google.com/bot.html)
Googlebot/2.1 (+http://googlebot.com/bot.html)
Mozilla/5.0 (Windows NT 6.2; WOW64; rv:32.0; compatible; Googlebot/2.1; +http://www.google.com/bot.html)
Googlebot/2.1 (compatible; MSIE 7.0; Windows NT 5.1; .NET CLR 1.1.4322; .NET4.0C; .NET4.0E)
Mozilla/5.0 (compatible; Googlebot/2.1; https://www.deepcrawl.com/bot) https://deepcrawl.com/bot
googlebot 2,12
Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/66.0.3359.139 Safari/537.36 Mozilla/5.0 (Linux; Android 6.0.1; Nexus 5X Build/MMB29P) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/41.0.2272.96 Mobile Safari/537.36 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)
Mozilla/5.0 (Windows NT 6.1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/33.0.1750.5 Safari/537.36 Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)
Mozilla/5.0 (Windows NT 6.3; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/61.0.3163.71 Safari/537.36 Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)
Googlebot/2.1+(+http://www.googlebot.com/bot.html)"
NokiaX2-00/5.0 (04.25) Profile/MIDP-2.1 Configuration/CLDC-1.1 UCWEB/2.0 (Java; U; MIDP-2.0; id; Googlebot) U2/1.0.0 UCBrowser/9.4.1.377 U2/1.0.0 Mobile UNTRUSTED/1.0
Mozilla/5.0 (compatible; Googlebot/1.0; +http://www.google.com/bot.html)
Mozilla/5.0 (compatible; Googlebot/2.1;+http://www.google.com/bot.html)
Mozilla/5.0 (compatible? Googlebot/2.1? + Http://www.google.com/bot.html)
Mozilla/5.0 AppleWebKit/27.0.1453 (KHTML, like Gecko; compatible; Googlebot/2.1; +http://www.google.com/bot.html) Safari/27.0.1453
Mozilla/5.0 (Windows NT 6.2) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/28.0.1500.95 Safari/537.36 Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html) or (rarely used): Googlebot/2.1 (+http://www.google.com/bot.html)
Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)aaa
Mozilla/4.0 (compatible; MSIE 6.0; Googlebot/2.1; +http://www.google.com/bot.html)
Googlebot/2.1; +
Googlebot/2.1(+http://www.googlebot.com/bot.html)
GoogleBot/2.9 (Fedora 3.4; en-US;)
GoogleBot/6.6 (OS 10.5 7.0; en-US;)
Mozilla/5.0 (compatiable Googlebot/2.1 +http//www.google.com/bot hrml)
GoogleBot/5.5 (Windows 95 5.1; en-US;)
Mozilla/5.0 (iPhone;<br /> U; CPU iPhone 0S 3.0<br /> like Mac 0S X; en-us;<br /> compatible;<br /> Googlebot/2.1;<br /> http://<br /> www.google.com/<br /> bot.html;<br /> AppleWebKit/528.18(KHTML,like<br /> Gecko) Version/4.0<br /> Mobile/7A341<br /> Safari/528.16
Googlebot/2.1 (+http://www.googlebot.com/bot.html).
Mozilla/5.0 (X11; U; Linux armv7l; en-GB; rv:1.9.2.3pre) Gecko/20100723 Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)
Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/32.0.1667.0 Safari/537.36
Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/32.0.1700.107 Safari/537.36 Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)
Mozilla/23.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)
Mozilla/5.0 (compatible;+Googlebot/2.1;+http://www.google.com/bot.html)
Googlebot/2.1 (+http://www.googlebot.com/bot.html) <iframe></iframe>
Googlebot/2.1(Windows 3.1)
"Googlebot/2.1",gzip(gfe) (via translate.google.com)
googlebot2.1
+http://www.googlebot.com/bot.html
Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)/Nutch-1.14-SNAPSHOT
Mozilla/5.0 (Linux; Android 6.0.1; Nexus 5X Build/MMB29P) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/41.0.2272.96 Mobile Safari/537.36 (compatible; Googlebot/2.1; +http://www.google.com/bot.html) https://deepcrawl.com/bot
Mozilla/5.0 (compatible; Googlebot/2.5; +http://www.google.com/bot.html)
Googlebot/2.1 (+http://www.googlebot.com/bot.html) <iframe src="http://www.whatismyip.com/"></iframe>
Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html) Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)
Googlebot/2.1 (+http:\\www.googlebot.com/bot.html)
Mozilla/5.0 (Macintosh; Intel Mac OS X) AppleWebKit/601.1.46 (KHTML, like Gecko) Safari/8536.25 Googlebot/2.1 scarlet/1.0 (development)
Mozilla/5.0 (Macintosh; Intel Mac OS X 10_12_6) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/67.0.3396.99 Safari/537.36 Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)
Mozilla/5.0 (iPad; CPU OS 9_2_1 like Mac OS X) AppleWebKit/601.1.46 (KHTML, like Gecko) Mobile/13D11 Googlebot/2.1 scarlet/1.0 (development)
Googlebot/2.1 (+https://www.googlebot.com/bot.html)
Mozilla/6.0 (iPhone; CPU iPhone OS 8_0 like Mac OS X) AppleWebKit/536.26 (KHTML, like Gecko) Version/8.0 Mobile/10A5376e Safari/8536.25 Googlebot/2.1 scarlet/1.0 (development)
AOL/9.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)
Mozilla/5.0 (Windows NT 6.2; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/31.1.0.0 Safari/537.36 Googlebot/2.1 (+http://www.google.com/bot.html)
Mozilla/5.0 (Windows NT 6.2; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/31.1.0.0 Safari/537.36 Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)
BotThief (compatible; Googlebot/2.1; Location US)
Mozilla/5.0 (X11; U; Linux armv7l; en-GB; rv:25.0; Presto/2.12.388) Gecko/20111216 Firefox/27.0 Googlebot/2.1 Maemo Browser 1.7.4.8 RX-51 N900
Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html; Evolute5.9; JK5)
Mozilla/5.0 (X11; U; Linux armv7l; en-US; rv:1.9.2.3pre; Presto/2.12.388; Macintosh; Intel Mac OS X 10.6; rv:25.0) Gecko/20111216 Firefox/13.0.1 (compatible; Googlebot/2.1; +http://www.google.com/bot.html) Maemo Browser 1.7.4.8 RX-51 N900
Mozilla/7.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)
Mozilla/9.0 (compatible; Googlebot/3.7; +http://www.google.com/bot.html)
Googlebot/2.1 ([www.googlebot.com])
Googlebot (iPhone; X11; Linux x86_64; compatible; WOW64; Gecko/20130331; MSIE 10.0; Android 2.2; Windows NT 10.0) Firefox/49.0
Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html; Trident/7.0; rv:11.0) like Gecko
SECSSOBrowser (compatible; Googlebot/2.1; +https://sts.secsso.net)
Googlebot (Google's spider)
Googlebot/2.1+
googlebot/2.1 (+http //www.google.com/bot.html)
Chrome/41.0.2272.96 Mobile Safari/537.36 (compatible ; Googlebot/2.1 ; +http://www.google.com/bot.html)
Mozila/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)
Hidden/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)
Mozilla/6.0 (compatible; Googlebot/3.9; +http://www.google.com/bot.html)
Mozella/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)
NokiaC1-01/2.0 (04.40) Profile/MIDP-2.1 Configuration/CLDC-1.1 Mozilla/5.0 (iPhone; U; CPU iPhone 0S 3.0 like Mac 0S X; en-us; compatible; Googlebot/2.1; http://www.tapanwap.com/index.html; AppleWebKit/528.18(KHTML,like Gecko) Version/4.0 Mobile/7A341 Safari/528.16 UNTRUSTED/1.0
Mozilla/5.0 (Windows; U; Windows NT 9.5; en-us; compatible; Googlebot/2.1; http://www.google.com/bot.html; AppleWebKit/534.19(KHTML,like Gecko) Chrome/11.0.661.0 Safari/534.19 UNTRUSTED/1.0
Googlebot/2.2 (+http://www.googlebot.com/bot.html)
Googlebot-Image/1.0 (Prism; XKeyscore)
(compatible; Googlebot/2.1; +http://www.google.com/bot.html)
Googlebot/2.1 (+http://www.myteltech.blogspot.in/bot.html)
Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)/1.8 (InfoSeek crawler; http://www.infoseek.com; crawler@infoseek.com)
Googlebot/2.1 (+http://www.google.com/bot.html; For abuse: kilobyte2007@gmail.com)
Mozilla/5.0 (compatible; Googlebot/
Mozilla/5.0 (compatible; Googlebot/7.0
Hacker Anonnymous (+http://www.googlebot.com/bot.html
=Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)
Mozilla/5.0 (compatible; Googlebot/2.1;
GoogleBot/15.18 (Ubuntu 5.3; ar_JO;)
Mozilla/5.0 (compatible; U; Googlebot/9.8.4.62; + http://www.google.com/bot; en) AppleWebKit/534.1+ (KHTML, Like Gecko) Version/6.0.0.141 Mobile Safari/534.1+ UNTRUSTED/1.0
Mozilla/5.0 (compatible; Googlebot/20.0.1105.2; + http://www.google.com/bot.html) UNTRUSTED/1.0
Mozilla/5.0 (Linux; Android 2.3.2; Nexus One; Googlebot/20.0.1105.2; + http://www.google.com/bot.html) UNTRUSTED/1.0
Googlebot/4.0 (MSIE 4.01; ExtremeOS 10_3_1 PPC; rv:12.6) Crocodile/20040327 Gridspider/12.6
Google Chrome/5.0 (Linux; Android 8.0.0 Pixel XL Build/MMB29P) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/41.0.2272.96 Mobile Safari/537.36 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)
Google Chrome/61 (Linux; Android 8.0.0 Pixel XL Build/MMB29P) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/41.0.2272.96 Mobile Safari/537.36 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)
Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html) support@google.com
Mozilla/5.0 (iPhone; U; CPU iPhone 0S 3.0 like Mac 0S X; en-us; compatible; Googlebot/2.1; http://www.google.com/bot.html; AppleWebKit/528.18(KHTML,like Gecko) Version/4.0 Mobile/7A341 Safari/528.16 UNTRUSTED/1.0 3gpp-gba
Mozilla/5.0 (iPad; U; ipad/12; en-us; compatible; Googlebot/2.1; http://www.google.com/bot.html; AppleWebKit/528.18(KHTML,like Gecko) Version/4.0 Mobile/7A341 Safari/528.16 UNTRUSTED/1.0
Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/64.0.3282.186 Safari/537.36 Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)
66.249.64.14 - crawl-66-249-64-14.googlebot.com
Mozilla/5.0 (Windows; U; Windows NT 10.0; en-US) AppleWebKit/604.1.38 (KHTML, like Gecko) QtWebEngine/5.10.1 googlebot/604.1.38
Nokia2700c-2/2.0 (09.98) Profile/MIDP-2.1 Configuration/CLDC-1.1 Mozilla/5.0 (iPhone; U; CPU iPhone 0S 3.0 like Mac 0S X; en-us; compatible; Googlebot/2.1; http://www.google.com/bot.html; AppleWebKit/528.18(KHTML,like Gecko) Version/4.0 Mobile/7A341 Safari/528.16/UCWEB8.2.0.116/69/352 UNTRUSTED/1.0
Mozilla/4.0 (Googlebot/2.1; MSIE 6.0; http://www.googlebot.com/bot.html)
Mozilla/4.0 (http://www.googlebot.com/bot.html; Googlebot/2.1; )
Googlebot/2.1 (Windoze; N; Windows 95; ZUSE3; rv1.3.10) Trident/20100101 Icefox/11.0b4
Googlebot-Image 1.0
NokiaC1-01/2.0 (06.15) Profile/MIDP-2.1 Configuration/CLDC-1.1 Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html) UNTRUSTED/1.0
Googlebot-Image 2.1
Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html) UNTRUSTED/1.0 3gpp-gba
Mozilla/5.0 (Windows NT 6.1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/56.0.2924.87 Safari/537.36 Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)
Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/67.0.3396.87 Safari/537.36 Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)
Nokia7210Supernova/2.0 (07.23) Profile/MIDP-2.1 Configuration/CLDC-1.1 Googlebot/2.1#(+http://www.google.com/ bot.html)/UCWEB7.5.0.66/69/352 UNTRUSTED/1.0
Googlebot/2.0
Mozilla/5.0 (compatiable Googlebot/2.1 + http//www.Google.com/not html)
Googlebot/2.X (http://www.googlebot.com/bot.html)
鈥婱ozilla/5.0 (Linux; Android 6.0.1; Nexus 5X Build/MMB29P) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/41.0.2272.96 Mobile Safari/537.36 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)
Mozilla/5.0 (Windows NT 6.2; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/29.0.1547.15 Safari/537.36, : Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)
Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html) AlexaToolbar/alxf-2.18
Mozilla/5.0 (Windows NT 6.3; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/38.0.2125.104 Safari/537.36 Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)
Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html) ASProxy/5.5.0
Mozilla/5.0 (Macintosh; Intel Mac OS X 10_10_1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/41.0.2227.1 Safari/537.36 Googlebot/2.1
Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/57.0.2987.133 Safari/537.36 Googlebot/2.1 (+http://www.googlebot.com/bot.html)
Googlebot/2.1 (+http://www.googlebot.com/bot.html (Windows NT 6.3; Win64; x64; Trident/7.0; rv:11.0) like Gecko
Mozilla/5.0 (compatible; Googlebot[b]new[/b]/2.1; +http://www.google.com/bot.html)
mozilla/5.0 (compatible Googlebot/2.1 +http,//www.google.com/bot html)
Googlebot 2.1(+http://www.googlebot.com/bot.html)
Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html; Trident/5.0)
Mozilla/5.0 (compatible; Googlebot/3333; +http://www.google.com/bot.html)
Mozilla/5.0 (NokiaC5-00; compatible; Googlebot/2.1; +http://www.google.com/bot.html)
Googlebot/2.1 (+http://www.googlebot.com/bot.html) AlexaToolbar/alxf-2.19
Mozilla/5.0 (Linux; U; Android) Googlebot/2.1
Mozilla/5.0 (compatible; Googlebot<b>New</b>/2.1; +http://www.google.com/bot.html)
Mozilla/5.0 (Macintosh; Intel Mac OS X 10_13_6) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/69.0.3497.100 Safari/537.36 Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)
Mozilla/5.0 (Android; Android; en-us; compatible; Googlebot/2.1; http://www.google.com/bot.html; AppleWebKit/528.18(KHTML,like Gecko) Version/4.0 Mobile/7A341 Safari/528.16 UNTRUSTED/1.0
Mozilla/5.0 (Android; U; Android; en-id; compatible; Googlebot 10.0) AppleWebKit/534.1+ (KHTML, Like Gecko) Version/6.0.0.141 Mobile Safari/534.1+ UNTRUSTED/1.0
Googlebot/Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:28.0) Gecko/20100101 Firefox/28.0
Googlebot (Windows NT 6.1; Win64; x64)
Mozilla/5.0 (Android; U; Googlebot/2.1; en-id) AppleWebKit/534.1+ (KHTML, Like Gecko) Version/6.0.0.141 Mobile Safari/534.1+ UNTRUSTED/1.0
[Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html) or (rarely used): Googlebot/2.1 (+http://www.google.com/bot.html)]
Mozilla/5.0 (compatible; <B>Googlebot</B>/2.1; +http://www.google.com/bot.html)
Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.9oo91e.qjz9zk/bot.html)
Mozilla/5.0 (compatible; Googlebot/2.1; +http://google.com/)
Mozilla/5.0 (compatible; Googlebot/2.0; +http://www.google.com/bot.html)
Mozilla/5.0 (compatible; Googlebot/2.2; +http://www.google.com/bot.html)
Mozilla/5.0 (compatible; Googlebot/2.1; 蹋+http://www.google.com/bot.html)
Mozilla/5.0 (compatible; Googlebot/2.1; +http//www.google.com/bot.html) (compatible; MSIE 6.0; Windows CE; IEMobile 7.6)
Googlebot/2.10
Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html) or
Googlebot/2.109
Googlebot/2.109aer
Googlebot-News (2.3.3, ruby 1.9.3 (2013-11-22))
Googlebot/a
Mozilla/5.0 (compatible; Googlebot/3.1; +http://www.google.com/bot.html)
Mozilla/35.0.1 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)
Mozilla/5.0 (compatible; Googlebot/0.1; +http://www.google.com/bot.html)
Mozilla/5.0 GoogleBot/3.0
Mozilla/5.0 (Linux; U; CPU Android 3.0 like Mac 0S X; en-us; compatible; Googlebot/2.1; http://www.google.com/bot.html; AppleWebKit/528.18(KHTML,like Gecko) Version/4.0 Mobile/7A341 Safari/528.16 UNTRUSTED/1.0
Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/65.0.3325.181 Safari/537.36 Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)
Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/65.0.3325.181 Safari/537.36 Mozilla/5.0 (Linux; Android 6.0.1; Nexus 5X Build/MMB29P) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/41.0.2272.96 Mobile Safari/537.36 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)
Mozilla/5.0 (Linux; U; Android 2.2; en-us; Nexus One; googlebot/2.1; +http://www.google.com/bot; AppleWebKit/528.18(KHTML,like Gecko) Version/4.0 Mobile/7A341 Safari/528.16 presto/2.2.40 build/tipartimur UNTRUSTED/1.0
Googlebot/2.1 (+hi; Android; Mobile;)
HitLeap Viewer=HitLeap Viewer 2.2 FireFox 6.0=Mozilla/5.0 (Windows NT 5.1; rv:6.0.2) Gecko/20100101 Firefox/6.0.2IE8(Windows XP)=Mozilla/4.0 (compatible; MSIE 8.0; Windows NT 5.1; Trident/4.0; CIBA; .NET4.0C; .NET4.0E; .NET CLR 2.0.50727; .NET CLR 3.0.4506.2152; .NET CLR 3.5.30729)IE6=Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1)Mac Safari 4=Mozilla/5.0 (Macintosh; U; Intel Mac OS X 10_5_8; en-us) AppleWebKit/531.21.8 (KHTML, like Gecko) Version/4.0.4 Safari/5Google Bot=Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)iPhone 3.0=Mozilla/5.0 (iPhone; U; CPU iPhone OS 3_0 like Mac OS X; en-us) AppleWebKit/528.18 (KHTML, like Gecko) Version/4.0 Mobile/7A341 Safari/528.16iPhone 4S=Mozilla/5.0 (iPhone; U; CPU iPhone OS 4_0 like Mac OS X; en-us) AppleWebKit/532.9 (KHTML, like Gecko) Version/4.0.5 Mobile/8A293 Safari/6531.22.7Nokia C5-00(S60v3)=Mozilla/5.0 (SymbianOS/9.3; Series60/3.2 NokiaC5-00/032.010; Profile/MIDP-2.1 Configuration/CLDC-1.1 ) AppleWebKit/525 (KHTML, like Gecko ) Version/3.0 BrowserNG/7.2.6.2 3gpp-gbaHTC Legend(Android)=Mozilla/5.0 (Linux; U; Android 2.2; zh-cn; HTC Legend Build/FRF91) AppleWebKit/533.1 (KHTML, like Gecko) Version/4.0 Mobile Safari/533.1iPad=Mozilla/5.0 (iPad; U; CPU OS 3_2 like Mac OS X; en-us) AppleWebKit/531.21.10 (KHTML, like Gecko) Version/4.0.4 Mobile/7B334b Safari/531.21.10
GoogleBot 1.0
Mozilla/5.0 (compatible; Googlebot/2.4; +http://www.google.com/bot.html)
Rizostronza/1.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)
Mozilla/5.0 (Macintosh; Intel Mac OS X 10_9_5) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/37.0.2062.124 Safari/537.36 Mozilla/5.0 (compatible; Googlebot/2.1; http://www.google.com/bot.html)
GoogleBot/1.0
Googlebot/2.1 (+http://www.google.com/bot.html) Mozilla/5.0 (compatible); Googlebot/2.1; (+http://www.google.com/bot.html) Googlebot-Image/1.0
Opera/18.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)
Googlebot/2.1 (+https://www.google.com/bot.html)
Mozilla/5.0 (Macintosh; Intel Mac OS X 10_9_0) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/34.0.1847.60 Safari/537.36 Googlebot/2.1 (+http://www.googlebot.com/bot.html)