好多Spider 不遵守 robots.txt 规则

By ego008 at 2013-04-03 11:15:49 • 957次点击

User-agent: Baiduspider
Disallow: /
User-agent: sogou spider
Disallow: /
User-agent: Googlebot
Disallow: /
User-agent: Slurp
Disallow: /
User-agent: ia_archiver
Disallow: /
User-agent: MSNBot
Disallow: /
User-agent: Robozilla
Disallow: /
User-agent: googlebot-image
Disallow: /
User-agent: *
Disallow: /

苦逼码农 at 2013-04-03 13:07:35
1

@苦逼码农 有些蜘蛛压根就没看robots.txt

ego008 at 2013-04-04 19:51:36
2
登录 后发表评论