Adding a robots.txt causes all pay content to be removed according to this. Yikes.

http://blogs.law.harvard.edu/philg/2015/01/28/web-publishers-can-delete-stuff-from-archive-org/