Yahoo has incorporated a new feature in it’s slurp bot to find how your website is handling missing pages. Yahoo occasionally may be requesting random gibberish URLs on your server – usually with slurpconfirm404 in them – to see how missing pages are being handled on your server. Understanding how Search Engine robots are crawling and responding to your site is vital in communicating the accessibility and relevancy of your content.
What is a 404 error?
A 404 error is a low level code that is supposed to be sent by a webserver when the page that has been requested is not found on the server. It communicates to the web browser (or search engine robot) that the content couldn’t be found. Even a missing graphic should return a 404 error. Don’t worry, your site correctly returns a 404 error unless you (or your “web guy”) has intentionally manipulated it to do otherwise.
What should be returned when an item is missing?
It makes sense to stick with predictable behaviors when dealing with both users and search engines alike. Some sites may have gotten away from sending 404 errors because that is ALL they were sending. It isn’t very user-friendly to send a page that simply says “404 error”, but with a little tweaking a sitemap and branded page can be sent to your user. This simple step will help you keep more visitors, and communicate to search engines when they have requested missing content.
See more about Yahoo and slurpconfirm404.