Trove to IPFS

HRDAG has been given a trove of about 5 million little files, about 5.5TB, and I don’t want to lose them. The files aren’t public, but they’re not entirely confidential, so it’s ok if they’re available in a semi-public way. I want to share these files with my partner organizations, so the files need to be accessible. We would rather not use cloud services like Amazon’s S3, Dropbox, or Google Drives because, well, we’re a human rights group, and we don’t trust big companies to prioritize protecting human rights information over frivolous attacks by extremist governments.

We decided we’d like to experiment with IPFS, a decentralized storage network and protocol for sharing data. As the Filecoin Foundation for the Decentralized Web explains,

IPFS is a peer-to-peer storage network that promotes the resiliency, immutability, and auditability of data. Through IPFS, content is tracked not by where it is located, but rather by WHAT data is being stored. This means that multiple organizations (or individuals, or communities) can store the same data, and if one of them is taken down, the data remains online. IPFS is an open protocol, which means that there are many implementations in many languages and platforms.

We have been gifted with a pinning service for this project until we can get our own IPFS nodes running. The service is called Storacha, which provides a couple of advantages for us. We get lots of copies (3-7 copies are pinned throughout the Filecoin network). And Storacha uses some cool hacks to make the sharing across groups a little easier.

What are we doing here?

This README explains a little bit of the code I wrote to shepherd the files from janky external USB drives to IPFS. This is a little tricky only because I wanted to keep track of the metadata at every step. And I am exploring the IPFS toolkits to get a sense of what they do well, and what they don’t do all that well. Link to the GitHub repository to continue reading …

---