Github Backup

Currently no one has backed up github.com (aside from Github). This webpage is about progress toward that. If you have 150-200TB of disk space and really good internet, please contact me about getting a copy of github.

List of Respositories

I host some metadata about github's repositories. This includes a lot of basic data about the repository, but NOT the issues, any wiki, downloads, or the git repository. As of Nov 2015, github has 28.3 million repositories.

You can grab greatly abbreviated metadata (recommended) as JSON (2.4G compressed, 6.9G uncompressed). This includes the repository name and URL, whether it is a fork (and what of), and a short description.
You can get a txt file of just the repo names: txt (332M compressed, 676M uncompressed).
Finally, full repository metadata is available in JSON format. The format is explained on the github API.

The files are available in batches of 10,000 at
```
http://za3k.com/github/repos-<X>0000-<X>9999.json
http://za3k.com/github/repos-<X>0000-<X>9999.json.gz
```
To download all files, run
```
    for x in {0..5000}; do \
      echo "https://za3k.com/github/repos-${x}0000-${x}9999.json.gz"; \
    done | wget -N -i -
    
```
The combined size of these files is 9.7G compressed, 115G uncompressed. Files are grouped by github's internal id; since some repositories are deleted or privated, each file contains less than 10,000 repositories.
This data was downloaded using a custom tool I wrote. My tool gets the data from the github API v3, with as little modification as possible.

List of Gists

Metadata for gists is currently unavailable from github, but I'm working with them to make it public.

Github Timeline

The Events Timeline is emphemeral, and being successfully recorded by githubarchive.org. A second person running the same program in case of downtime would be a plus.

(New!) http://ghtorrent.org/ is downloading the same timeline, and also fetching fuller historical data.

Estimates on archiving repositories

I selected 1000 random repositories from the above list, removing 427 forks. I then checked out all repositories. The total size was 4.3G, with or without compression. It was around 3 GB for a shallow checkout. If we assume forks take no space, this means an average github repository takes up 4.3M. Omitting the largest repositories may improve this estimate, but I didn't run further tests.

If there are 28,000,000 repositories on github at an average size of 4.3M each, that multiplies out to around 120TB data total for the git repositories.

Additional information

Archive Team has some information about github, and you probably should talk to them if you want to help back up github. We basically just don't have anyone with space--the technical issues are not too large.