=== valhalla3 === Author: zachary "za3k" vance (za3k@za3k.com) Written 2020-04-29 (document version 1.0.9) valhalla3 is a p2p protocol designed to manage downloading and making available a very large dataset among a set of volunteer computers. The intended use case is to back up the 14000TB Internet Archive (I'll talk to them after I have something working), as 3+ always-online copies of 1,400,000 10GB chunks. My estimates says we're targeting ~100K users and ~1M files. The protocol is implemented as a general-purpose way to ensure a set of torrents are populated, together with a mechanism to download from HTTP and turn the downloaded files into torrents. The goal is to have 3+ distributed full copies of the internet archive, online at all times. === simple === Here's my simplified plan: Goal: store an immutable copy of Internet Archive Nongoal: Replace Internet Archive Nongoal: (maybe) Browse old websites on it Nongoal: Update content (you can add content, though) 2. The adoption model is that someone downloads a Windows program, clicks install, and it starts on boot. They feel good for a week, never interact with it, and it runs until they get a new computer. Each computer is a 'peer'. We use something like 20% of their free space. The only relevant differences between peers I know are: how much space they have, and whether they can get incoming connections (behind NAT/firewall). 3. Each peer has a bittorrent client, designed to seed 100-10K torrents per peer. To download a file they don't have, peers either grab it on HTTP from Internet Archive (if new) or from other clients seeding it on torrents (if old). To share a file they do have, they seed the file on torrents (24/7). 4. I add a p2p protocol, to manage a distributed key-value store. Each peer controls one key, and can shove whatever they want in there. The entire key-value store is replicated to every peer, and they "gossip" to spread around updates. There's some crypto to make all this secure. 5. Clients use the key-value store to both to see what other peers are online + active, and to decide what needs downloaded autonomously. 6. There's a trusted 'admin', who makes the list of files that should be mirrored. This gets put in the key-value store under some special 'admin' keys. 7. There are also some websites, which the peers can grab key-values from. These are 99% to push out a self-update and unjam everything if we break stuff during alpha. 8. I get a working technical implementation, shove the installer at Internet Archive and related people. Someone else magically handles "get 100K users for a long period". Happy to throw in some more detail if you want, but I expect there are plenty of problems with this approach, and I'd like to hear about those more than the specific problems with the p2p protocol. Danger 1, Internet Archive and its mirrors go down. If IA goes broke, probably this just stays up. If IA is down because of lawyers, governments etc, my system is in danger of going down too in the near future. A good samaritan gets together some spreadsheets and gets more people online to seed the same stuff. In the meantime, a different good samaritan buys $300K of computer equipment (at today's prices) and torrents all the content to a single location in 1 month, before the system goes down. (This can be done arbitrarily fast with more money, google can do it in under a day) Anyone can torrent the content, no coordination with admins or IA is needed. Danger 2, Internet Archive has to start removing stuff. Basically see Danger 1, but less so. Danger 3, the admins for this disappear. The program can no longer self-update on Windows, and the file list will never update again. Also, after enough time new nodes may have a hard time joining. Nothing else bad happens. === design goals === - Actually get a copy (!) - Make it available to everyone (people not running the system can get the dataset) - Let anyone easily run a client, so we can get 100K users. - Run on Windows (lots of people) and Linux (rare people with huge amounts of space). OS X isn't a priority. - Run forever after initial install (autostart on boot), with no user action. - "Run forever" also means some form of auto-update, since we can't assume the initial valhalla3 client is adequate. - Zero configuration if possible, or at least minimal configuration. - Works behind NAT with no IPv6 or port forwarding - Robust - The system should work if the volunteers keep participating but maintainers disappear. - 50% of computers get knocked out in 1 hour, no big deal - Scales to 100K users - Scales to 14000TB of files (1M torrents, each with a single 10G file) - Under typical use (0.1TB-10TB stored) doesn't significantly tax the system it's running on. <10% CPU and memory use, no freezes, and nothing crazy on bandwidth. - User-friendly GUI (not covered here) - Detects bandwidth caps some ISPs do, so people don't get horrible overage charges. (Just turning off is acceptable) - Cleans up files on uninstall - Can mirror a large dataset without active cooperation from the current HTTP host (for example, pre-computed hashes should be optional) - If we have 100K honest users, 1K dishonest users can't sabotage it === current state and feedback === valhalla3's protocol is designed, but it isn't prototyped yet. I'm currently asking for feedback on the protocol before doing another prototype, since I've already had to throw out two. email is preferred (za3k@za3k.com). I'm interested in feedback about big things: I'm solving the wrong problem, I have the wrong goals, there are better ways to store Internet Archive, I have the wrong set of components. Feedback on technical protocol details is less interesting: in order my priorities are reliability, scaling, security, and efficiency. Vague feelings ("this is too complicated") are fine. Areas where you didn't understand something in this writeup are fine too, and feel free to ask questions. No security feedback yet please. I already know you're clever. I need to find really fundamental design problems, impress me with wisdom :). === organization of valhalla3 === ## metadata-store and terminology A p2p network is used to share changing information about which files are being downloaded and seeded, and what clients are online. This information is stored in the metadata-store. We call computers in the network peers or clients. Conceptually, the metadata-store is a key-value store, with the special property that each key-value pair (called a 'val') has a timestamp indicating when it was last written. Each key is owned by one entity (usually a peer), which possesses a special secret cryptographic secret which is needed to update the value for that key in the metadata-store. Each peer knows only its own cryptographic secret, so it's able to write to a single val, and unable to write to any other vals in the metadata-store. The metadata-store keys is formatted as "client-", so anyone looking at the metadata-store can verify each value was signed by the correct secret. Just like real signed documents, a signature only means the author is who you think--not that the information in the document is all correct. Each peer maintains its own metadata-store, which is gradually updated with news it learns by gossiping with other peers. val writes take less than one day to propogate through the network, but a peer will never have the "one true copy" of the metadata-store. Each peer's metadata-store will be have some keys be newer and some older in different ways. The way the metadata-store is implemented is as two data structures, the immutable-store and the stamp-store, which get updated together when a 'val' is written. The immutable-store contains actual metadata of <10KB, which can be looked up by hash. Metadata is never changed in the immutable-store, but it is deleted when no longer needed. Here is an example 'client' metadatum giving information about one peer, represented as though it was a JSON object for readability: { "ipv4": "1.1.1.1", "ipv6": null, "ipv4_port": 25555, "file_manifest_version": "2015-01-01T1:00:00Z", "files_claimed": [1,2,4,5,11,...,1945], "files_downloaded": [2,4,5,11,...,1945], "files_bittorrent_seeding": [2,4,5,11,...,1945], "file_hashes": {2: "1ff71991c121616486323c545292e3fe82139a2c, ...}, "file_torrent_infohashes": {2: "69c20664e48549b760dbc89a72b33e853fbe8993", ...}, "file_sizes_bytes": {2: 10000000000, ...} "tls_certificate": "", } # note: pretend this has length 100 and hash "adc83b19e793491b1c6ea0fd8b46cd9f32e592fc" The stamp-store is a collection of stamps. Each stamp points from one key (ex. client-994b8cc93f94894fed9ec350c9c5309face107c3) to one metadatum in the immutable-store. The stamp-store never has two stamps with the same key. stamps are small, with exactly the same fields and length for each one. Here is a sample stamp: KEY client-994b8cc93f94894fed9ec350c9c5309face107c3 METADATA_HASH adc83b19e793491b1c6ea0fd8b46cd9f32e592fc # This is used to look up the metadata in the immutable-store METADATA_LENGTH 100 TIMESTAMP 2020-01-01T1:00:00Z CRYPTOGRAPHIC_SIGNATURE 7fa11744b8add1b3141d9099ad333a10aa542a63 So to review, a 'val' in the metadata-store is made up of a metadatum in the immutable-store, plus a stamp in the stamp-store. The peer looks up a key in the stamp-store, which gets it a METADATA_HASH which we look up in the immutable-store, which gives it the actual metadata. Along this trip, the stamp verifies that the owner wrote the 'val' and when it was written. To receive an update to a val, a peer makes sure the timestamp in the update is newer than its own version. Then, it checks the cryptographic signature is correct. Assuming it is, the stamp update is really from the key's owner, and it is more recent than the information the peer had. The peer updates the stamp and adds the metadatum. Updates with no existing version just mean a key is newly-seen for the peer, so the timestamp is ignored and the val is always added. To write a val (update a key with a new value), the owner first puts the value into the immutable-store. Then it generates the stamp without the signature, using the current time as the timestamp. Finally, it uses the cryptographic key associated with the key to sign the stamp, generating a signature. It places the stamp in the stamp-store, replacing any current value for that key. An owner sometimes writeas the current value to a key as well--this updates the timestamp to show that this information is still current and correct. If a metadataum ever has no stamps pointing at it, it is deleted. In addition to the common 'client' stamp keys which peers own, there are 4 special 'admin' vals (see 'admin vals' for details). 'admin' metadata is allowed to be slightly larger, <10MB. There is no other data in the metadata-store. ## peers and the metadata-store Each peer generates a public-private cryptographic signing keypair on first startup. The same keypair is used by the peer forever. The peer owns and is responsible for maintaining exactly one val, with a key of the form "client-". Because vals are signed, peers receiving updates know that the data is really from the signing peer. Timestamps prevent replay attacks--only higher timestamps are accepted when receiving updates. Each peer maintains its own metadata-store, which is gradually updated with news it learns by gossiping with other peers. val writes take less than one day to propogate through the network, but a peer will never have the "one true copy" of the metadata-store. Each peer's metadata-store will be have some keys be newer and some older in different ways. Each peer maintains a full copy of the entire metadata-store, which is persisted to disk across executions. It is <1GB with the target number of users and files. There is no "authoritative" version, but the GOSSIP protocol below means any updated val should reach a peer within 1 day. Peers "expire" and delete vals based on timestamp after 7 days. Peers write their val once a day even if nothing has changed (just change the timestamp and signature), so the network can tell they are still online. ## GOSSIP updates the metadata-store Under normal conditions (for the very first time the peer goes online, see 'bootstrap process'), a list of online peers can be calculated from the metadata-store. Any 'client' val with a timestamp in the last day, is assumed to be from an online peer. Roughly once an hour, each peer does a pairwise update with another peer, chosen randomly from online peers with a public IP. This command is called GOSSIP. At the start of the exchange, each peer has its own set of stamps and metadata, all valid. During the exchange, the peers receive stamps and metadata from one another, updating those with newer timestamps. After the exchange, both peers have the same, more up-to-date metadata-store as one another. The exchange is designed to be moderately fast (<1s, bandwidth permitting). After the network as a whole does enough exchanges, everyone's writes reach everyone else, but the copies are never "synchnonized" and identical across the network. With 100K peers, the perfectly fastest way all of them can exchange data takes 17 (parallel) rounds of pairwise exchanges. Using random periodic GOSSIPs instead, a simulation shows it consistently takes around 20 GOSSIPs per peer, so it's fine to do things at random. At the rate of starting 1 GOSSIPs/hour (that's actually participating in 2/hour), writes should easily be available to all peers in half a day. If a 'client' val is 2 days old, since clients update their val daily, that peer can be safely assumed offline. NAT is a potential issue for many p2p protocols, and valhalla3 does not to implement any kind of NAT piercing. Instead, IPv4-only peers behind NAT just always choose to exchange with peers with a public IPv4 address. Peers with public IPv4 addresses will get many times more traffic as a result, but propagation times get even faster. GOSSIP Because this is the most frequent operation, it's likely to be optimized. Treat this as very much a proof-of-concept. Wire/protocol details may be changed before release to reduce overhead. Security feedback is not welcome at this stage and will be ignored--please focus on more fundamental problems. 1. (Over TLS) Peer A contacts Peer B, saying "I want to GOSSIP (version 1). The current time is 4:01pm.". 2. Peer B responds "Yes, I support GOSSIP (version 1). I agree the current time is near 4:01pm." [Because stamps expire based on time, the peers agree on a 'working' current time to make sure they will agree on the set of valid stamps. This is a real edge case thing--finicky to get right but uninteresting. So, I'll pretend the peers agree on the time exactly for the rest of this summary. A hand-wavy solution to illustrate there are no real problems follows for sticklers. We allow clients to differ by up to 6 hours. To make sure contested stamps/metadata are available, peers internally wait 1 extra day before deleting expired stamps/metadata from disk. The exchange is done with a modified copy of the metadata-store including exactly those stamps within 7 days on the agreed-on time, whether expired or not. Changes are merged into the real metadata-store after. There are no major problems that happen from accepting stamps signed ahead of your system clock.] 3. Peer A and Peer B exchange their stamp-stores by sending a full copy to one another. [In reality, this is done using some variant on the rsync protocol, which sends only changes, reducing bandwidth] 4. Peer A and Peer B reconcile the two stamp-stores without sending messages. Both peers arrive at the same result, called the "canonical" stamp-store for the exchange. - They throw out any stamps with invalid keys. A peer MAY blacklist the other exchanging peer as a bad actor. - They throw out any stamps with invalid signatures. A peer MAY blacklist the other exchanging peer as a bad actor. - They throw out any stamps with expired timestamps, or timestamps in the future of the "working" time. [This should never occur and is an error condition]. They SHOULD NOT blacklist the signing peer if the timestamp is in the future (this could be used to DoS the system). - If they have two values for a key, they select the newer one and throw out the older one. - If they have two values for a key with the same timestamp, the lower hash is taken. The peer MAY blacklist the signing peer as bad actor. 6. Peer A sends to Peer B (and vice versa) - The hash of the canonical store - All immutable values peer B needs, concatenated together. If the peers did not agree on the canonical store hash, this is an error condition. 7. Peer A and Peer B verify immutable values. Each value should be in the mutable store, with a matching hash length and hash. Peer A and Peer B add any metadata received to their immutable-store. Peer A and Peer B remove from the immutable-store any metadata no longer referenced in the stamp-store. 8. Both peers now have the same stamp-store and immutable-store, which they commit to disk. ## admin vals There are a few 'admin' vals which provide some benefits. valhalla3 is designed to operate OK even if 'admin' vals are never updated. 'admin' vals are signed with unique keys, generated in advance by hand. 'admin' vals will be updated by the project maintainers (admins) infrequently and by hand. The current plan is to treat them normally except not to expire them. The 'admin' vals are: 'admin-manifest-': The manifest is the list of files to download. Includes HTTP source(s), any known hash, and any known torrent infohash. This is where you would put a list of bittorrent trackers for another project, although the plan for Internet Archive is not to use one. [Note that while the example client data includes the hash, infohash etc for all files, it can be left out if in the manifest] 'admin-clients-': A list of known peers, which is used to bootstrap (see 'bootstrap'). These should be in the valhalla3 network and always online. 'admin-httpclients-': A list of web mirrors, which can be used to bootstrap or reduce load (see 'HTTP pseudo-peers' below). These should be always online. 'admin-selfupdate-': Can be used to signal to peers to install a self-update of the client software. Will include the binary so p2p self-updates work. I haven't decided, but the binary blob _may_ be downloaded from http peers only if updated, making it a bit of an exception to the usual uniform process. The binary itself includes a signature with a hardcoded key, to add an additional security check. # HTTP pseudo-peers The 'admin-httpclients-' val will contain a list of public websites which serve valhalla3 metadata. All peers will regularly LISTEN from the website, checking signatures and timestamps as usual. This is similar to GOSSIP, but it's one-way, so I think of these mirrors as 'pseudo-peers'. Details of exactly when/how frequently clients fetch metadata TBD, scaling to 100K clients is tricky. There are a couple reasons this is a good idea. - If we totally break the client somehow during alpha, we can still get them to autoupdate. - If there is enough churn or a build is old enough, all hardcoded bootstrap peers may be unavailable. This lets the client update the boostrap peer list. - It's very easy to keep a static website working. - If we use CDNs, this is near-instant and can tons of traffic. ## bootstrap process Once the peer is on the network, it knows the list of other peers through GOSSIP. But how does it get some IP addresses to start talking to anyone? On first boot, the peer creates the metadata-store from a hardcoded database. Only the 4 'admin' stamps are in the hardcoded database. The peer now has a hardcoded list of peers and HTTP pseudopeers. This "known nodes" bootstrap is how most p2p systems work. valhalla3 tries to provide a full known set of peers, rather than one or two fast and reliable ones as some clients (bittorrent DHT) do. The bootstrap process is as follows: 1. The peer creates the metadata-store from a hardcoded database. 2. The peer LISTENs from all HTTP pseudopeers. 3. If there is an client software self-update in 'admin-selfupdate-', the peer install the update and restarts the client instead of finishing the bootstrap process. 4. If any HTTP pseudopeers were added in 'admin-httpclients-', the client LISTENs to them as well and goes to step 3. 5. The peer GOSSIPs with several (10+) "bootstrap" peers from 'admin-clients-', to get new contacts. 6. The peer keeps GOSSIPing with its contacts until it stops discovering large numbers of new contacts. If the client software starts up after being offline for a while, it will go through a similar bootstrap process to get its stale metadata-store up to date. ## data Wait, but weren't we downloading something? Yeah! So although the really complicated bit is maintaining this metadata-store, the peer's _important_ job is to download and seed files, not to gossip with its peers. The peer knows the whole list of files the sytem wants to download ('admin-manifest-" val, and all peers start trusting the hash in the manifest. In the case of Internet Archive, we will likely be able to provide hashes as part of the initial manifest. - If 3+ peers have the file, but the file is not in the manifest, the manifest tells the peer what to do: nothing (pick another file), http download, or use the most frequently reported infohash Any time a peer downloads a file, it verifies the SHA hash and file size match any present in the manifest. It rejects and deletes the downloaded file if either is incorrect. ## list of commands okay this section is a bit vague right now, I need a real prototype and more networking knowledge yes GOSSIP - explained above. update metadata-store from a peers, while sending them updates. LISTEN - fetch data from an HTTP pseudopeer to update the metadata-store. this is one HTTP download, nothing fancy SEND_LOG - the alpha will probably have some form of logging or statistics-gathering, to debug probably yes IS_MY_IP_PUBLIC / YES_YOUR_IP_IS_PUBLIC - used during boot to detect whether a peer is behind a firewall, untraversable NAT etc. also returns apparent IP as seen by Peer B. need to learn more networking to figure out if this is needed. needs DDoS mitigation too. probably no STUN/TURN - don't think we need this, also no idea how to coordinate it without one central server. Lets two peers behind NAT talk to each other using a peer with a public IP. GIVE_ME_PAR2 - request for PAR2 data for a file, if a client discovers corruption. probably won't make the final cut due to complexity and wanting NAT traversal == Opinionated options and non-options Feel free to let me know if you disagree with these, I'll listen but probably won't respond. Options (set during setup program, all have reasonable defaults) - Does your ISP have a bandwidth cap? (auto-detect too) - How much space do you want to use? (default to 20%) - What folder should I download stuff to? (reasonable default) Auto-detected - Are you on IPv4? IPv6? Can either of those addresses be contacted from outside your network? - What is your uplink/downlink bandwidth? (we'll either autodetect or use uTP to avoid congestion) Config-file only options - Turn off self-updates on Windows - Turn off HTTP pseudopeers Non-options - Turn off torrents - Use HTTP proxies - Turn ON self-updates on Linux (we'll rely on package managers) We won't push people to turn on port forwarding or to turn on IPv6. == Known issues Feedback on how to fix these welcome. Feedback on issues NOT listed here even more welcome :). - Storing and reteiving compressed 10GB chunks is not a good way to actually distribute access to the Internet Archive. This is a backup only, which is not as useful, or exciting to volunteers. - Storing 3 full copies of data is not the most efficient use of volunteer space--we could be doing something like reed-solomon coding - Scalability: Big clients would need to run 10,000-1,000,000 torrents. You can't seed 1,000,000 torrents on one machine, it's not really possible. But you can't get 14PB on one machine either. I'm going to keep working on improving torrent clients--some of this is fixable, and I think we can get to the point of allowing this. Announces are really a scalability issue. Things I don't think are that big a deal - Reliability: All bootstrap peers can be down (can't fix) - Reliability: All bootstrap http-pseudopeers can be down (non-issue, system will still work) - Reliability: Admin keys are a centralized feature. They could be lost and clients would lose self-update. (open to suggestions) - Reliability: Admin keys are a centralized feature. They could be lost and the network could fail to ever add new files. (won't fix) - Reliability: If all bootstrap peers are down, and either http-pseudopeers are down or don't have new peers (ex. maintainers vanish), bootstrap is impossible for new nodes. (seems fixable, open to suggestions. rendevous over some public channel?) - Reliability: 10GB chunks are pretty big. Peers generating and sharing some kind of parity data (error correcting codes like PAR2) to correct local disk errors or HTTP download errors, might be a good idea. - Scalability: IPv4 peers with public IPs may get overwhelmed. (open to suggestions, but if GOSSIP is fast, I think it may just be OK) I could use some data on how many people have static public IPv4 vs DHCP public IPv4 vs IPv4 NAT; and of those on DHCP how frequently IPs change. - Scalability: The average client will be running 100s or 1,000s of torrents. I've done quite a bit of research and testing and I think this should be doable, but it's a risk. - Scalability: Collective torrent announce load. Is there a risk we could take down any tracker we add, or cause issue on the DHT? It seems like probably no, but in the worst case we'd have to switch off mainline DHT or use a private tracker, which makes the torrents hard to access to the public at large. - Race Condition: Because peers only get writes after about a day, if there are few high-priority files, many peers will all download that file. - Admin writes propogate slowly. It's possible getting an updated 'admin' val should trigger "infection" GOSSIP calls to propogate admin writes quickly. - User-friendliness: If you have a way to auto-detect that the user is on an ISP that has particular behavior let me know. I want to detect and enable different client behavior for ISPs with a bandwidth cap, or that do nasty tricks like throttling to people on bittorrent. A hardcoded database of ISPs/IPs is good enough. - I've had problems using DHT-only bittorrent behind NAT, I'll have to make sure everything is ok. Not interested in security feedback yet. Here are the known issues. - Security: Self-update could be used maliciously (lack of self-update is an equally risky, but you can turn it off. open to suggestions but this is probably wontfix) Security: Admin keys are a centralized feature. They could be abused. (this is 99% self-update, calling this a duplicate) - Security: This entire thing is pretty vulnerable to sybil attacks, making banning bad peers ineffective. Even if it wasn't, distibuted ban/reputation systems are hard to keep beneficial. I'm trying to design it so bad peers don't get banned and don't matter. - Security: Malicious peers can falsely report having a file, causing honest peers to pick the wrong file to prioritize. This can prevent peers from having the full data set or cause load on one endpoint. Files are not evicted. One fix is have some fraction of files selected _not_ prioritize rare files. Proof-of-content merkle tree challenges could detect bad actors, but you'd also need a way to ban them. === Reasonable but rejected designs Random download. Each peer randomly selects files from the hardcoded manifest, downloads, and seeds them with no coordination. - PRO: Very simple - PRO: Robust, nothing to take down - CON: bittorrent is hard to scale to 1000s of files (same as valhalla3) - CON: More peers needed than with coordination (about 40% more) - CON: Hard to tell if you're done or making progress. You can't see how many peers there are, and it's actually quite hard to practice to even check which of 1million torrents are seeded - CON: Disallows direct peer-to-peer transfer being part of the design - CON: Disallows a central HTTP download directory other form of peer directory, which helps restore - CON: Manifest can't be updated. If we add a webserver to update the manifest, it's a central point of failure. Honestly I think this one isn't half-bad, the main disadvantage is that you can't add any kind of transfer other than torrents. I rejected it before I was using torrents, so tracking and restore was worse. I do think valhalla3 is better for something this size, but for a smaller project this would work. Master-slave. Each slave gets the list of things to download from a master, who tells it "download these files". Slaves are tracked by the master. - PRO: Simple, and easy client - CON: Centralized restore. If the maintainers disappear or don't have the bandwidth to do a 14PB restore in time, you're screwed. - CON: Centralized operation. Easy to take down. You can have clients default to picking their own files to download, but it doesn't help restore. - CON: Scaling. 100K clients is pushing it with one master. The easiest solution is multiple masters, which adds complexity. (valhalla2) Federated coordinators. Each peer gets the manifest from a list of website mirrors. It regularly talks to several trackers, telling the tracker what is has and is getting, and asks what other peers are getting. It looks at what other peers have and picks some files to download. There may be an emergency "direct contact" mode where we tell the client to update or debug the alpha. - PRO: Pretty scalable. - PRO: Decentralized restore possible (but needs deep technical knowledge) due to directory of peers - CON: Complicated, there are 3-4 different kinds of software running - CON: Either trackers are a point of failure (if trusted), or can take down the system (if anyone can run one) - CON: Website mirrors are a point of failure for updates. === the old IA.BAK (valhalla1) There is an existing project called IA.BAK from Archive Team to back up the Internet Archive. This project is inspired by trying to come up with a version of that that works. If you don't know what IA.BAK is already, skip this section. Let me preface this by saying I've talked to the IA.BAK author (joey hess) and I think he's pretty onboard with a shiny new solution targeting Windows. It's acknowledged experimental alpha software, and my feeling is that the experiment showed it wouldn't work. Problems with the existing IA.BAK, any of which may be incorrect: - Doesn't have a copy of IA. (Targeted 1PB rather than 14PB, got 100TB) - (Effectively) targeted only at the technical part of Archive Team, who collectively don't have 14PB of storage. By this I mean it's usable only by extreme linux nerds (me)--you can't even do a package install. But even if it was trivial for all linux users to install, that's missing 99% of the population. - Restore can only be done by the maintainers. - The backend, git-annex, is frequently updated and complex (you could argue bittorrent is too) Of these, I think the biggest problem is the target audience. I'm basically imitating projects like BOINC or SETI@Home--the correct solution is to get a LARGE number of nontechnical people interested in helping out. Then we have them run a Windows installer once which asks them zero questions, and IA.BAK then do useful stuff with their computer at no cost to them for 10 years until they buy a new one. If they click on the tray icon during that first interesting week, we show them shiny statistics like "We have 1% of the internet backed up. You personally have backed up 99,999,999,999 websites wow such a good person!". And they actually are. === feedback As I get feedback I'll post it here. --- fenn: your names suck. change immut-val and mut-val -> claim and stamp. also make the metadata-store section clearer. the start of GOSSIP is a bad explanation of bootstrap, start with why. me: metadata store = metdata-store immut-val -> metadata immut-store = immut-store mut-val -> stamp mut-store -> stamp-store fenn: you know about DHT put/get (BEP 44)? it's similar me: yes i do, not coincidence. GOSSIP is scalable here and DHT is not. fenn: you should have an option for times of day to run. screensaver mode that shows you progress when you're not touching the computer. wait maybe that's bad. add a global leaderboard, it works for SETI@home you should have a merkle-tree timestamp system that collaboratively timestamps the log of changes, so you can find the oldest claims for a file and trust those peers richard: there's a trusted root so you're not storing child porn, right? would be good to have plausible deniability so copyright holders can't sue volunteers how do you get content? me: Yeah I think this is biggest flaw in the design... you should really be able to browse old versions of websites on it (it should be a mirror, not just a backup). me: This needs DoS protection from lots of garbage data. We should add a number-of-clients limit, and somehow prefer pingable/old clients like Kademelia does. Unforuntately, Kademelia is designed around being able to contact all your contacts regularly, which isn't true here, so I'll need to think about it.