I’ve moved a few servers at work to our new datacenter, using the “normal” method: adjust TTLs from 3600 to 120, wait an hour, sync servers, change DNS to new IP, all traffic should be going to new server within 2 minutes, bump TTLs back up to 3600.
Google and Yahoo seem to be compliant, but there are invariably the same 3 bots that appear on the old server up to a day after the switch:
- msnbot/2.0b (+http://search.msn.com/msnbot.htm)
- Mozilla/5.0 (compatible; YandexBot/3.0; +http://yandex.com/bots)
Maybe there should be an option in robots.txt.
User-agent: * Honor-DNS-TTLs: no-really-you-should-be-doing-this