Archives and Inter-Planetary Linked Data: The Perfect Fit

Image by Pete Linforth from Pixabay

Decentralised, Peer-To-Peer, Blockchain-Based Academic Archiving & Publishing Ecosystem

We’re well underway with our development of KnowledgeArc Network, the decentralised, peer-to-peer, blockchain-based academic archiving and publishing ecosystem.

Latest Decentralised Technologies

Employing the latest decentralised technologies such as Inter-Planetary File System (IPFS), OrbitDB and Ethereum, we’re solving the problem of needing truly permanent repository solutions—permanent, immutable and censorship-resistant.

The Inter-Planetary Linked Data (IPLD) model provides us with a new way to describe information on KnowledgeArc Network along with the metadata that describes the information that’s being stored.

IPLD is built on top of IPFS, a decentralised peer-to-peer file storage system.

Location-Based vs. Content-Based Addressing

The brilliance of IPFS comes from the way it references what it stores: instead of the traditional method of using a location-based address to a document for file, IPFS identifies the file using a unique identifier, or hash. This is known as content addressing. Doesn’t make sense?

Here’s a brief example:

Let’s say I have some very important research I’ve undertaken and I want to distribute it to as many people on the web as I can reach.

Using some kind of web-publishing software, I upload this important research in a PDF I’ve called “my-academic-research-that-will-change-the-world.pdf” from my computer to a web site somewhere:

I put it in a folder called “research-papers”.

I then distribute this file to everyone, with the web address

It’s critical that this research is easily accessible, unchangeable, and built to last for generations.

There are a number of problems with this solution:

  • What happens if the web site address changes?
  • What happens if the web site gets moved?
  • What happens if the web site administrator decides to change the research papers path to something else?
  • What happens if this file gets deployed to other web sites?
  • How do I ensure people know that the file my-academic-research-that-will-change-the-world.pdf contains MY research and hasn’t been changed, manipulated or completely replaced?

Basically, the problem we have is that this important research cannot be guaranteed to:

  • be accessible using the same address
  • be unique
  • be tamper-resistant

IPFS Solves These Problems

IPFS solves these problems by generating an identifier or “hash” based on the contents of the file. A hash which is mathematically unique enough to never result in a clash between two files.

By generating a hash based on the contents of my-academic-research-that-will-change-the-world.pdf, I can now locate this file no matter where it’s stored.

Additionally, IPFS includes a network of computers which can store this file so that it can be found from the network rather than from one computer (for example,

Linked Data – Decentralised

IPLD is another part of this decentralised picture. IPLD can link pieces of data together.

Again, taking our research ‘my-academic-research-that-will-change-the-world.pdf’, I can now add some metadata which I can link to the document.

Metadata might include:

  • the title of the paper
  • authors
  • keywords
  • an abstract from the paper
  • other important descriptive information

Additionally, I can extend the linked data to include whole profiles about the author, link keywords to other data so that I can find similar works. I can even store links to revisions, so a historical change log of the paper could be easily retrieved from the most current version.

And The Benefit Of All Of This?

  1. Everything is uniquely identified based on its content.
  2. It can never be corrupted because the corruption will be easy to identify.
  3. It will be censorship-resistant because any change will generate a whole new set of identifiers.
  4. And it’s easy to locate because the identifier never changes.
CIDs are content identifiers. Metadata is captured as individual documents and can be compared for changes. Items store a reference to assets and metadata. A chain of items provides a version history. Each keyword is also a data block and multiple keywords can be linked to metadata.

Follow The Developments

Follow our progress by checking out our code at GitLab

+Follow Knowledge Arc Network on LinkedIn

Join the Knowledge Arc Network discussion on Telegram

Self-signed certificates with local root CA

This tutorial briefly covers the creating and trusting your own certificate authority (CA) for issuing self-signed SSL certificates, and is designed to work with OribitDB’s new REST API HTTP/2 push services.

This tutorial is aimed at Unix-based systems, in particular Ubuntu and other Debian-based Linux distributions so you will need to modify the commands for your own platform. All code examples are intended to be copied and pasted directly to the command line and will generate certificates in your current working directory.

To get started, we are going to create a root certificate which we will use to sign additional SSL certificates with.

First, create your root CA private key:

openssl genrsa -des3 -out rootSSL.key 2048
Generating RSA private key, 2048 bit long modulus
e is 65537 (0x010001)
Enter pass phrase for rootSSL.key:

You will be prompted for a password. Be sure to specify one that is long enough as you may encounter errors if your password is too short.

Next, use your CA private key to create a root certificate:

openssl req -x509 -new -nodes -key rootSSL.key -sha256 -days 1024 -out rootSSL.pem

Once launched, you will need to re-enter the password you assigned to your private key:

Enter pass phrase for rootSSL.key:

If successful, provide information about your certificate:

 You are about to be asked to enter information that will be incorporated
 into your certificate request.
 What you are about to enter is what is called a Distinguished Name or a DN.
 There are quite a few fields but you can leave some blank
 For some fields there will be a default value,
 If you enter '.', the field will be left blank.
 Country Name (2 letter code) [AU]:
 State or Province Name (full name) [Some-State]:WA
 Locality Name (eg, city) []:
 Organization Name (eg, company) [Internet Widgits Pty Ltd]:
 Organizational Unit Name (eg, section) []:
 Common Name (e.g. server FQDN or YOUR name) []:localhost
 Email Address []:

You are now ready to install the new CA certificate into your CA trust store. The following commands will copy the root certificate into Ubuntu’s CA store so you may need to modify the paths if you are on a different distribution or OS platform:

sudo mkdir /usr/local/share/ca-certificates/extra
sudo cp rootSSL.pem \/usr/local/share/ca-certificates/extra/rootSSL.crt
sudo update-ca-certificates

Now it is time to generate a certificate for your development environment. Create a private key for your new certificate:

openssl req \
 -new -sha256 -nodes \
 -out localhost.csr \
 -newkey rsa:2048 -keyout localhost.key \
 -subj "/C=AU/ST=WA/L=City/O=Organization/OU=OrganizationUnit/CN=localhost/"

Next, create the certificate, signing it with your Root CA:

openssl x509 \
 -req \
 -in localhost.csr \
 -CA rootSSL.pem -CAkey rootSSL.key -CAcreateserial \
 -out localhost.crt \
 -days 500 \
 -sha256 \
 -extfile <(echo " \
    [ v3_ca ]\n \
    authorityKeyIdentifier=keyid,issuer\n \
    basicConstraints=CA:FALSE\n \
    keyUsage = digitalSignature, nonRepudiation, keyEncipherment, dataEncipherment\n \
    subjectAltName=DNS:localhost \

Your SSL certificate is now ready for use. To use it with OrbitDB’s REST API, launch the cli.js script with the flags –https-key and –https-cert, using the new localhost.key and localhost.crt files we just created:

node src/cli.js api --ipfs-host localhost --orbitdb-dir ./orbitdb --https-cert ./localhost.crt --https-key ./localhost.key

The certificates should validate against your Root CA when used with tools such as curl:

curl -vs --http2 -X POST https://localhost:3000/db/my-feed --data 'create=true' --data 'type=feed'
 successfully set certificate verify locations:
 CAfile: /etc/ssl/certs/ca-certificates.crt
 CApath: /etc/ssl/certs
 SSL connection using TLSv1.2 / ECDHE-RSA-AES128-GCM-SHA256
 ALPN, server accepted to use h2
 Server certificate:
 subject: C=AU; ST=WA; L=Ellenbrook; O=Organization; OU=OrganizationUnit; CN=localhost;
 start date: May 25 14:56:35 2019 GMT
 expire date: Oct  6 14:56:35 2020 GMT
 common name: localhost (matched)
 issuer: C=AU; ST=WA; L=Ellenbrook; O=Internet Widgits Pty Ltd; CN=Local Certificate
 SSL certificate verify ok. 

In the above, you can see the CA being loaded from the correct location (/etc/ssl/certs) and details about the certificate (Server certificate:).

You can now successfully run the new OrbitDB HTTP API with self-signed certificates on your development environment.


How to get HTTPS working in localhost development environment,,

OrbitDB HTTP API – A Practical Guide

OrbitDB is the distributed, p2p database system which will revolutionize the way we store, replicate, and disseminate information and will become the cornerstone of any dApp which requires data storage.

Too much centralization has put control of the internet into the hands of a few. Web3 aims to decentralize the internet, providing a more democratized, distributed ecosystem.

Most of the hype is around cryptocurrencies and the freedoms they will bring, cutting out the middleman and putting control back in the hands of the people. However, there are a number of less “high-profile” but equally game changing projects which will reshape the internet as we know it.

This how-to briefly covers the basics of how to create an OrbitDB database and store data as well as introduce the most powerful feature of OrbitDB; replicating the data across multiple locations.

OrbitDB, the decentralized database

OrbitDB is a decentralized database system which uses IPFS for distributing stored data via P2P technologies. Storing data in OrbitDB ensures high availability and low latency due to the nature of distributed architectures such as IPFS.

Originally, OrbitDB was available as a Node.js library, so usage was limited to Node-based applications. However, with the recent release of the OrbitDB REST API, any language which supports REST calls can leverage this distributed database.

Setting up

Running an OrbitDB REST server is relatively straight-forward but some knowledge or working on the command line will be required. These steps assume you are running Linux or some other Unix-based operating system. For Windows users, you will need to translate the commands to your environment.


Firstly, this guide assumes you can use a command line and install software. You don’t need to know node.js or how peer-to-peer systems work but you will need to be able to execute commands in a terminal. In this guide, all commands will be run from the terminal and will be represented like so:

type commands at the command line

You will also need two machines running since we will be replicating a decentralized database. This can either be two physical computers, a couple of virtual machines or docker containers.

Lastly, because the OrbitDB server uses Node.js you will also need npm (bundled with Node.js) to install the dependencies. This tutorial will not cover the installation and configuration of these requirements.

Running IPFS

OrbitDB uses IPFS to distribute and replicate data stores. The OrbitDB HTTP server runs in one of two modes; local or api.

When run in Local mode, OrbitDB will run its own IPFS node. When run in api mode, OrbitDB will connect to an already-running IPFS node.

For this tutorial we will connect to a running IPFS daemon and will assume you already have this installed. You will also want to run IPFS daemon with pubsub enabled.

Start your first IPFS daemon by running:

ipfs daemon --enable-pubsub-experiment

Building the REST server

Now get a copy of the code. You can grab it via Github at


Alternatively, you can clone the git repo:

git clone

Install your dependencies:

npm install

Setting up the SSL certificates

The latest version of the OrbitDB HTTP API incorporates HTTP/2. Therefore, to run the server, you will need to generate SSL certificates.

There are a couple of options available for obtaining certificates; you can issue a certificate using a certificate authority such as Let’s Encrypt, or, you can become your own CA. For development environments, the second option may be better and a thorough overview on how to do this is covered by the tutorial Self-signed certificates with local root CA.

The rest of this guide will assume you have a trusted SSL certificate set up and that curl will use your trust store to validate the certificate. If not, you will need to tell curl to ignore the certificate verification by passing the -k flag:

curl -k -X GET ...

Up and Running

Starting the HTTP API server

Start up the OrbitDB server and connect to your running ipfs:

node src/cli.js api --ipfs-host localhost --orbitdb-dir ./orbitdb --https-key localhost.key --https-cert localhost.crt

The –https-key and –https-cert options above assume you are using the certificate and key generated from the tutorial Self-signed certificates with local root CA. If not, replace with your own certificate and key.

Consuming our first request

The REST server is now running. You can test this by running something simple (we are going to use cURL to run the rest of these command so make sure you have it installed):

curl -X GET http://localhost:3000/identity

This will return a JSON string representing your OrbitDB node’s identity information. This includes your public key (which we will use later).

Create a database

Creating a data store is very easy with the REST API and you can launch a store based on any of the supported types. For example, you can create a feed data store by running:

curl -X POST http://localhost:3000/db/my-feed --data 'create=true' --data 'type=feed'

You can also use JSON to specify the initial data store features:

curl -X POST http://localhost:3000/db/my-feed -H "Content-Type: application/json" --data '{"create":"true","type":"feed"}'

Add some data

Let’s add some data to our feed:

curl -X POST http://localhost:3000/db/my-feed/add --data-urlencode "A beginner's guide to OrbitDB REST API"

And viewing the data we have just added:

curl -X GET  http://localhost:3000/db/my-feed/all

["A beginner's guide to OrbitDB REST API"]

Be aware that there are two different endpoints for sending data to the store, and which endpoint you use will depend on the store’s type. For example you will need to call /put when adding data to a docstore.


Replicating is where the real power of distribution lies with OrbitDB. Replication is as simple as running an OrbitDB REST node on another machine.

Assuming you have a second computer which is accessible over your intranet or via Docker or a virtual machine, you can replicate the my-feed feed data store.

Getting ready to replicate

Before you replicate your feed data store, you will need to make a note of its address. You can do this by querying the data store’s details:

curl http://localhost:3000/db/my-feed


Copy the id. We’re going to use it in the next step.

Running another copy of the data store

On your second machine, make sure you have IPFS running and the OrbitDB REST server installed and running.

Replicating the my-feed data simply requires you query the first machine’s my-feed data store using the full address. Using the address of the my-feed data store I queried earlier, request the data:

curl http://localhost:3000/db/zdpuAzCDGmFKdZuwQzCZEgNGV9JT1kqt1NxCZtgMb4ZB4xijw%2Fmy-feed/all

["A beginner's guide to OrbitDB REST API"]

You may need to run the curl call a couple of time; OrbitDB will take a small amount of time to replicate the data over.

There are two important things to note about the address; 1) we drop the /orbitdb/ prefix and 2) we need to url encode the /. The html encoded value of / is %2F.

And that’s it. You have successfully created a new OrbitDB data store on one machine and replicated across another.

Let’s test it out. Back on your first machine, add another entry to the feed data store:

curl -X POST http://localhost:3000/db/my-feed/add --data-urlencode "Learning about IPFS"

On your second machine, retrieve the feed list again:

curl http://localhost:3000/db/zdpuAzCDGmFKdZuwQzCZEgNGV9JT1kqt1NxCZtgMb4ZB4xijw%2Fmy-feed/all

["A beginner's guide to OrbitDB REST API","Learning about IPFS"]

Adding data in a decentralized environment

What happens if you want to add more entries to the my-feed data store from your second machine:

curl -X POST http://localhost:3000/db/my-feed/add --data-urlencode "Adding an item from the second OrbitDB REST peer."
{"statusCode":500,"error":"Internal Server Error","message":"Error: Could not append entry, key \"03cc598325319e6c07594b50880747604d17e2be36ba8774cd2ccce44e125da236\" is not allowed to write to the log"}

If you check the output from your REST server you will see a permissions error. By default, any replicating node will not be able to write back to the data store. Instead, we have tell the originating OrbitDB instance that the second instance can also write to the my-feed data store. To do this, we must manually add the public key of the second OrbitDB instance to the first instance.

It is important to note that the data store must be created with an access controller pre-specified. Start by deleting the data store on the first machine:

curl -X DELETE http://localhost:3000/db/my-feed

We must now set up the my-feed database again and add some data:

curl -X POST http://localhost:3000/db/ -H "Content-Type: application/json" --data '{"create":"true","type":"feed","accessController":{"type": "orbitdb","write": ["048161d9685991dc87f3e049aa04b1da461fdc5f8a280ed6234fa41c0f9bc98a1ce91f07494584a45b97160ac818e100a6b27777e0b1b09e6ba4795dcc797a6d8b"]}}'

Note the accessController property; this specify the controller type and the key which can write to the database. In this case it is the first machine’s public key, which can be retrieved by running:

curl http://localhost:3000/identity

On the second machine, retrieve the public key:

curl http://localhost:3000/identity

Grab the publicKey value. We will now enable write access to the my-feed database:

curl -X PUT http://localhost:3000/db/ --data 'publicKey=04072d1bdd0e5e43d9e10619d997f6293f4759959e19effb958785b7f08413fb81501496a043385c245dedc952ee01c06bc9c654afe79b11dd5f130796baf8d2da'

publicKey will be the publicKey of the second machine. We must execute this request from the first machine because only the first machine currently has write permissions to the data store.

With the second machine’s publickey added, we can go ahead and add a new my-feed from the second machine:

curl -X POST http://localhost:3000/db/my-feed/add --data-urlencode "Adding an item from the second OrbitDB REST peer."


This brief introduction to the new OrbitDB HTTP API will hopefully provide some insights into how OrbitDB functions and will hopefully highlight some of the benefits a distributed database system brings to the decentralized web.

We have only scratched the surface of what is possible with OrbitDB. You could go ahead and add other machines to my-feed’s write access controller or create different data stores for storing data in different formats. Also the HTTP API is only in its infancy and there are a number of new features being actively developed.

This new chapter in OrbitDB’s brief history is going to bring a lot of new development and providing access to other languages will expand its usability.