Digital Spelunking at America Online

The ephemeral nature of the web has appalling implications for the preservation of information. Even many once common Mac OS 9 shareware apps are now impossible to find, to say nothing of older and more obscure programs. How many web and ftp sites from 1994 still exist?

In the days when BBSes were more popular than the internet, America Online's file libraries rivaled the entire internet in their breadth of files. It was in an effort to determine whether these ancient libraries still existed that I gingerly installed AOL on a spare computer and signed up for an account.

My hope was that even though the original keywords for these libraries no longer worked, I might still be able to find a link or url that would take me to the old file library search function, much in the way that it's still possible to access the Apple II forum on AOL by going to keyword: aol://4344:1264.a2main.10029531.514525857.

After much effort, I failed to find the search page, but I was able to use the Apple II libraries to understand the url syntax. The url for the Apple II New Files library is aol://4400:8287, and here is the URL for the UnForkIt file, contained in that library: aol://4401:8287:636250. The first value identifies the resource type. In this case, either 4400 for a library, or 4401 for a file. The second number, 8287, is the library ID. 636250 is the file ID. The file IDs are not consecutive within libraries.

By changing the library id, it is possible to access file libraries that that no longer have direct links. I began trying random IDs and was soon bookmarking the Connectix Macintosh Library and MacHacks. Many libraries would load immediately. On other occasions I'd get an error message stating no library existed at that ID. Most disturbing were the libraries that AOL's software believed should exist, but that were "not responding".

As AOL has moved away from its proprietary Rainman language in favor of html, and sophisticated users have left for the Internet, these old areas of AOL have been abandoned and forgotten. Each file records the date it was most recently downloaded. In many of these libraries, the most recent download was from 1999. In a few, the records have been confused with those for other files — the description will be for a shareware utility, but the file will be a jpeg. Other file downloads fail half way through. The system, it appears, has been left to decay.

Nonetheless, the bulk of the files I found were in salvageable condition. The challenge now came of finding all the surviving software libraries and downloading their contents. The AOL application isn't easily scriptable, but QuicKeys can automate it by simulating key combinations and clicks, so I wrote a QuicKeys script to open the "Keyword" window, type in a possible library address, and bookmark the address if it worked. Over a period of several days, this script tried about 70,000 possible library locations. Another QuicKeys script is now visiting each library and downloading the files.

To date, 7,236 files have been downloading, filling 4.1GB of disk space. Much more work remains to be done, and volunteers are welcome. Help is needed in the following areas:

• Downloading files from AOL. This requires a dedicated Mac OS X computer running QuicKeys (or at least the 30-day demo).
  
• Writing scripts to check for redundancies and damaged files.
  
• Modifying upload scripts so that the files can be added to the file libraries.
  

If you'd like to get involved, please email Tom Owad