I can’t believe what I just found on the Yahoo!Search Blog about removing pages from a website. The author says “The best way to remove dead URLs from the Yahoo! Search index is to return an HTTP Error 404 when our crawler requests the page.”
Are they serious, really serious?
The HTTP spec clearly says return code 404 is “Not Found” temporarily and 410 is “Gone” permanently. They even say in th explanation for code 404 “The 410 (Gone) status code SHOULD be used if the server knows, through some internally configurable mechanism, that an old resource is permanently unavailable and has no forwarding address.”
Yahoo slurp is free to treat a 404 page as if removed although I don’t think it serves the searching public well. However, I can’t understand why the Yahoo!Search blog teaches webmasters to send a 404 if a 410 return code is appropriate.
Just needed to rant about this, because this blog has for sure a wide readership.