Explore interplanetary file systems today!
LinuxTowers’ very own space cadet, Mats Tage Axelsson, boldly explores interplanetary file systems, the web tech that’s so hot right now…
The interplanetary file system (IPFS) isn’t for colonising Mars, but rather to create a freer, robust system for the web. Indeed, its developers refer to IPFS as the permanent web. Let’s find out more about this blockchain technology…
The web’s IP protocol was designed to work without a central server. This was partly due to the risks of nuclear war. By design, anyone can connect a computer to the net and add a server. There is a caveat though: most communication goes over a small number of services owned by large corporations. While these are not evil by default, they represent a central point in the system and, as such, can be exploited.
Take Egypt during the Arab spring. The government shut down international connections, so the populous had major problems spreading ideas and organising protests. The lockout was possible because they were using specific servers, like Twitter, making it easy for the government to block services from a specific host. If the protesting groups had used IPFS or any decentralised system, they could have kept on using Twitter.
IPFS works on any kind of active network, even if there’s no contact with the rest of the internet. Take a scenario where a group of people are working together need to share data. In this case, you can host your data within the conference and have it update afterwards.
Using current technologies, organisers would have to manually upload all data after the conference. With IPFS, the data and indexing will go to the blockchain and update when it’s possible. All participants can choose to keep the data and let it update on its own later. The point is that organisers can choose to publish to the web even without a connection.
The aim of the project is to replace HTTP as a means of transferring content over the web. Another reason is that HTTP requires a web page to be stored on one computer. If that computer goes down, the page is gone. Of course, backups exist and most people and organisations use them. But when a server goes down for any reason, content just disappears.
No more 404 messages!
There are many solutions to this problem, but with a distributed network the copies are everywhere. Often, users can automatically download from a source that’s close to them. This saves on global bandwidth. Currently, a request that happens on another continent will always transfer all the content through many routers across the plane – even static content. With a distributed solution, most of the transfers will be local.
IPFS enables you to transfer a web page from more than one place. Where you take it from will depend on which is the closest and fastest for the receiver. A user may also store a cached copy on their own machine and offer it to others. IPFS will improve access for people with unreliable links. If you publish resources – files, web pages, media and more – to IPFS, then the viewer won’t be as dependant on a reliable link, which is the case with regular web pages.
The project has many aspects, IPLD (InterPlanetary Linked Data) is a major part of it. It defines the protocol that transports the hashes. Services like git are already using hashes to keep track of changes. It also verifies that any change aren’t tampered with. Most work on IPFS is to make it compatible with these and other web technology systems.
Since the idea is to decentralise the whole web, IPFS uses technologies such as Bitcoin, Git and Ethereum. In fact, the IPLD protocol supports Ethereum for smart contracts in IPFS context. To go through how it works you need to know the basics of the protocol stack:
IPLD This is the protocol that binds it all together. It describes how to identify and address your data. This is where the protocol adds support for other systems. Implemented standards are Bitcoin, Git and Ethereum.
Libp2p The network layer is a separate project that others can use. It started life in IPFS, but then the project’s developers found a need for it elsewhere. One
example is OpenBazaar, which is a market for selling goods and services.
IPNS This is the name server resolver. Remembering the hashes is impossible for most users. The name server has the same function as in IP – it uses a name to connect to a hash. This system also makes it possible to point to one set of content even after it’s changed. Any content on IPFS you change receives a new hash, which is the equivalent of a new address. IPNS provides a solution for this problem.
Multihash This creates a system for describing the hashes. The final goal is to have the hash describe itself. In principle, there’s a marker at the beginning of the hash that identifies the standard. The common name for this is self-describing hashes.
Making money from IPFS
You can use Filecoin to create transactions for storage and file transmission services. IPFS enables you to choose how much and what content you store. The amount you store and serve to others counts towards your reward. In general, you’ll pay for others to store your files, and you’ll get paid if you store files for others.
To install IPFS, all you need to do is download the package, unzip it and run the shell script: $ wget https://dist.ipfs.io/go-ipfs/v0.4.14/go-ipfs_ v0.4.14_linux-amd64.tar.gz $ tar -xvf go-ipfs_v0.4.14_linux-amd64.tar.gz $ cd go-ipfs $ sudo ./install.sh
The script copies the binaries to the appropriate libraries and sets the path. Now let’s see what you can do with it. The power of IPFS won’t be obvious at first – you’ll need to experiment with what’s available.
The first thing to do is to create your own key-pair and peer identity. The peer identity is you: everything you publish is placed logically under this address. $ ipfs init initializing ipfs node at [your hash].go-ipfs generating 2048-bit RSA keypair...done peer identity: [your hash] to get started, enter: ipfs cat /ipfs/QmYwAPJzv5CZsnA625s3Xf2n emtYgPpHdWEz79oj|WnPbdG/readme Included in the printout is your own node. You can also check your identity after you started: $ ipfs id
The printout also includes your private key, so keep it safe. Your private key is also stored in .ipfs/config.
The file that the script points you to is on the blockchain, so try it out: $ ipfs cat /ipfs/QmYwAPJzv5CZsnA625s3Xf2nemt YgPpHdWEz79ojWnPbdG/readme
You may wonder how much disk space the cache keeps. To manage the space you need to use the repo subcommands. Check how much space you’re using: $ ipfs repo stat --human NumObjects: 1949 RepoSize (MiB): 51 StorageMax (MiB): 9536 RepoPath: /home/matstage/.ipfs Version: fs-repo@6 Clear out the garbage with the garbage collector, at this stage, the change will be small. $ ipfs repo gc Check how much space you saved. $ ipfs repo stat --human NumObjects: 26 RepoSize (MiB): 33 StorageMax (MiB): 9536 RepoPath: /home/matstage/.ipfs Version: fs-repo@6
You can change the parameters in the config file. If you only want to see the values, use the ipfs command. $ ipfs config show
There are more settings in the file than you can see in the output.
To edit the config file, go to ~/.ipfs/config. Another option is to set one option at a time using ipfs config . Use a key-value pair to set values. You can see the key in the configuration file; the key value is built with a full stop for subkeys. To set how much space you’re willing to keep locally set the StorageMax value. $ ipfs config Datastore.StorageMax 25G Check that the value has been set. $ ipfs config show | grep StoreageMax “StorageMax”: “25G” Follow up by checking the value in your local repo. $ ipfs repo stats NumObjects: 929 RepoSize: 79427591 StorageMax: 25000000000 RepoPath: /home/matstage/.ipfs Version: fs-repo@6
You need to find content to play with. The community is still small so not much information has been published. A wikipedia in Turkish was one of the first big pieces. Thankfully, the English version has just been published, so check out this link: $ ipfs/QmXoypizjW3WknFiJnKLwHCnL72vedxj QkDDP1mXWo6uco/wiki/ This hash is the top level of the wikipedia, so if you need to keep the whole collection to read offline, pin it: $ ipfs pin add -r /ipfs/QmXoypizjW3WknFiJnKLw HCnL72vedxjQkDDP1mXWo6uco/wiki/ --progress
The pin command enables others to use your copy of the file. So, in this example, your download of the wikipedia can be read by anyone who runs an IPFS node. This then begs the question: how do we know that your copy is the correct one? This is where the cryptographic hash comes in – the hash depends on the content. It’s similar to an MD5 hash. Change anything in the document and the hash won’t match. Because this means that any changes have to have a new hash, the developers have built version control into the protocol.
Building the blockchain
The examples above are web pages that are put on the IPFS blockchain, and you can do this, too. Start by collecting files you want to share. These files will need to be of differing sizes so we can see how they’re stored on the blockchain. You can also use your existing webpage. Even a single picture can be pointed to. To do this on the command line all you need to do is run: $ipfs add MyPicture.png You may have a small webpage or application that you want to publish. $ ipfs add -r MyWebpage
The command parameter -r means recursive, so you point at a directory and a web browser can find your
index.html file and use it as a webpage. The resulting printout will report if you’re successful and give you many hashes. At the end of the listing, the last hash points to the current version of your new site. Use this hash to point viewers to your new permanent webpage. Any user can now choose to look at the page remotely or add it to its own disc. If they look at the page, the content is saved in their cache. $ ipfs cat [the new hash] | firefox
When you use this command the result goes to standard out, so you’ll need to pipe it to your browser to view it.
IPFS will aggressively keep files a user has read, but the data will be removed regularly. Use the ‘GCPeriod’ parameter to set the interval. Files that you want to keep and be pinned as usual: $ ipfs pin add [the new hash]
You have a webpage on the IPFS, but to view it you must share a long, clumsy hash. How do we solve this problem? First, you need to publish to the blockchain:
$ ipfs name publish [your hash] Published to [Your peer id]: /ipfs/[your hash]
The ‘Your peer id’ will now point to all versions of your web page. Up until now, anyone who wanted to see your content had to have a hash for each piece of content. The address will be http://localhost:8080/ipns/[Your
peer id]/. Note the ‘n’ in IPNS – this is the Inter Planetary Naming Service that handles the lookup to your current web page. Now that you have it published, you can update it and other users will see always the latest version. Updating your content is a matter of changing the files, then republishing to your own hash.
To publish to the web you need to have two things: your own domain and an IPFS gateway. You’ll set the DNS text records in your provider’s web panel. A good idea is to use the gateway.ipfs.io address. Point your domains A record to that IP address and add dnslink=/
ipns/[your peer id] to your DNS TXT record. To have an overview of what is going on while browsing for other stuff, the developers have created plug-ins for Firefox and Chrome. There are two on Github, IPFS Companion and IPFS Station. Note that the Companion gives a direct link to sharing your files on IPFS. You can access this function in the web-UI also.
The web-UI is a neat tool for your web browser – you can use it to check your own files and see your peers. The peers on the network are listed with their IP information and location in the world.
Adding as a service
To add the IPFS daemon as a service, use a systemd service file. The file is included here. All you put is the description under the unit section. Under the service section, you include the user that the service should be running as. The user has to have initiated the repository with the ipfs init command before the service can run: [Unit] Description=IPFS daemon [Service] ExecStart=/usr/local/bin/ipfs daemon Restart=always User=[username] Group=[username] [Install] WantedBy=mutli-user.target
Place the file in /lib/systemd/system/ipfs.service. Now systemd can monitor the service. Note that this is beta software, so make sure you are going to use it before making it permanent.
Start the service and check that the service is running: $ sudo systemctl start ipfs.service $ systemctl status ipfs.service When you’re satisfied that your system will handle it smoothly, enable the service. $ systemctl enable ipfs service
A good idea is to host your IPFS service on a VPS, the binary is small and you can choose how much data you keep around with the built-in settings.
Filecoin services are not up and running in a meaningful way yet. Early miners are invited to register, so give it a shot. When it’s ready for live deployment you can earn Filecoins by installing the software on your computer and offering storage to other users.
Anyone serving files will be rewarded in several ways. The price is calculated depending on amount and speed, so a home user may be able to make money even with small discs. The speed and reliability of your storage are also taken into account when the price is set.
Bazaar services
There are many services using IPFS, but most are in development. Worth mentioning is Shift ( www.shiftnrg.
org) that has services running and an appimage for Linux. It has big plans for its distributed app called
Phantom, a file management user interface. Other platforms that use IPFS include the Openbazaar. You can pick up an Ubuntu binary for it from its github page: www.openbazaar.org/download. Openbazaar is a sales platform similar to eBay, but without compulsory fees. You can pay for a moderator that handles disputes for a fee. There are already thousands of sellers on the platform. With a new way to handle micropayments, it’s likely that many other systems will emerge as soon as all the components are fully developed and in a stable development stage.
Considering the early stages of development for IPFS, it’s remarkably stable. However, the tools that are available from the website make it clumsy to handle the hashes. This will put off most people and must be addressed. IPNS will grow to handle this problem so mass adoption will eventually become a dead cert. Blockchain technology usually has a scalability problem in that everyone has to have the entire chain. Such a problem has been tackled from the start so this won’t prevent uptake.
Another factor that points to a successful future for IPFS is all the supported systems and projects that use IPFS, IPNS and IPLD. Once the incentive to store items using Filecoin is implemented and operational, there’ll always be someone willing to “host” files. This should lead to stability coupled with the advantages inherent in distributed networks. Watch this space!