I was just looking on the National Science Foundation's web site to try to find the Index of FOIA Frequently Requested Documents. The Index is mentioned in the NSF's Public Information Handbook. When I couldn't find the Index, I realized the Handbook was written in 1999, and perhaps an older version of the NSF website had a copy of the Index. So I went to the Internet Archive's trusty Wayback Machine, and put in the NSF's web address. Yesterday when I looked at the results page, there were no results, and the statement that the site had been blocked by robots.txt was the only information returned. Today, the Wayback Machine's results page shows each instance when the site was archive, from 1997 to 2005, but when you click on a link, the resulting page is empty and has this message:"We're sorry, access to http://www.nsf.gov/ has been blocked by the site owner via robots.txt."
Why? The archived web pages contain valuable information for researchers, scientists, journalists, and the general public. The web pages are agency documents, and, having been published on the Internet, are part of the FOIA public domain. See Let the People Know the Facts: Can Government Information Removed From the Internet Be Reclaimed?, 98 L. Libr. J. 1, 23 (2006). Since the Federal Records Act (44 U.S.C. 3301 et seq.) prohibits destruction of agency information, the older versions of the web site should be archived somewhere. So who is hiding information, and why? Every single version of the NSF site I tried to view was blocked.
If you know of another Federal agency that has refused to allow access t its archived web sites, I'd like to hear about it. If you know why agencies are blocking access, or have a theory, I'd like to hear about that, too. [Posted by Susan Nevelow Mart]
7/7/06 - Ed. Note: Be sure to read Bill's comments below based on his investigation - Mary Minow