« Can library patrons carry unconcealed weapons in Kansas? | Main | Weak 4 digit PIN numbers »


A little more information is here. Summary: I think the glitch was temporary, and the Wayback Machine indexes NSF just fine. The Index of FOIA Frequently Requested Documents, however, does not appear to be available.

I first read your headline as "Is the *NSA* hiding something?"

My response to which was "duh!"

Thanks Bill - Good legwork. - Mary Minow

Got a response in short order:

NSF blocks all indexing of the site between 7AM and 7PM ET, our peak traffic hours, for the convenience of our users. However, there is no block on the site from 7PM to 7AM ET. This is standard policy for most high traffic sites. The owner of http://www.archive.org/index.php need only comply with our policy in order to index our pages.

So, neither malice nor stupidity by the look of it. I'd never heard of such a policy, but it makes sense.

Perhaps we should not attribute to malice that which is adequately explained by stupidity. My guess is that, rather than some sinister plot to remove information, someone somewhere thinks that they are fighting the good fight against spam by way of robots.txt.

That said, we are talking about an administration in which it's hard to say whether malice or stupidity is in greater oversupply, so I wrote to the webmaster to ask about this policy. I'll let you know if I get an answer.

The comments to this entry are closed.