FoundationDB
We suspect Mihalis Tsoukalos is a bit of an Apple fan-boy at heart, as he explores FoundationDB from the fruit-fronted corporation.
We suspect Mihalis Tsoukalos is an Apple fan-boy as he explores FoundationDB from the fruit-fronted corporation.
H ow does a distributed key value store database that can handle huge amounts of structured data sound to you? FoundationDB is an Apple product( we’ re not criticising–Ed) that became an open sourced Apache-licenced project in April 2018. Here we’re going to teach you how to install, configure, administer and use FoundationDB on your Linux machines, as well as how to communicate with
FoundationDB using Python and Go.
Dropping ACID
Broadly speaking, databases are everywhere, and realising that you need a database for one of your projects is an easy task. The difficulty lies in deciding which database to use. In order to make the right decision, you’ll need to know the features of each database and what differentiates one database from the other. But first we’ll start by explaining what ACID is.
Data consistency in a database is crucial. ACID (atomicity, consistency, isolation and durability) is a set of properties that guarantee database transactions are performed reliably. Atomicity is the term given when you do something to change a database, the change should work or fail as a whole. A database’s consistency indicates that it remains consistent all the time. Isolation means that if other things are taking place at the same time on the same data, they shouldn’t be able to see half-finished data. Durability refers to the guarantee that once the user has been notified of the success of a transaction, the transaction will persist, and won’t be undone if the hardware or the software crashes afterwards. So, ACID support can be important for some kinds of applications such as applications where losing data isn’t accepted, whereas being able to store structured data is also important for other kinds of applications – those that work with text and web data. Well FoundationDB has both features, which differentiates it from other database systems!
Apple provides drivers for working with FoundationDB using Python, Ruby, Java, C and Go. Later on in this tutorial you’ll see the Go and Python drivers in action.
Dig your FoundationDB
Although you can build FoundationDB by compiling it from source, you can also install FoundationDB on Linux machines by following the instructions at
www.foundationdb.org/download. For the purposes of this tutorial, an Ubuntu 18.04 Linux system will be used for the FoundationDB installation. So, for an Ubuntu Linux machine you should execute the following: $ wget https://www.foundationdb.org/downloads/5.2.5/ u bun tu/ installers/ found at iondb- server _5.2.5-1_ amd 64. deb $ wget https://www.foundationdb.org/downloads/5.2.5/ u bun tu/ installers/ found at iondb- clients _5.2.5-1_ amd 64. deb $ sudo dpkg -i foundationdb-clients_5.2.5-1_amd64.deb foundationdb-server_5.2.5-1_amd64.deb The first package is for the server part of
FoundationDB whereas the second package is for the client part of FoundationDB – you ‘ll most likely need both of them. The last command is for installing the two packages; if you’re using a different Linux variant such as RedHat or CentOS both the downloaded packages and the installation commands will be different.
The installation script of the server package automatically starts the FoundationDB server process on the Ubuntu machine. You can easily find the version of FoundationDB you’re using by executing either
fdbserver --version , which is the executable of the
server process, or fdbcli --version , which is the executable of the client for connecting to FoundationDB. This tutorial uses FoundationDB version 5.2.5.
The screenshot ( belowleft) shows a part of the installation process as well as the version of
FoundationDB that will be used in this tutorial.
Administering FoundationDB
After a successful installation you can start the FoundationDB service process if it’s not already running by executing service foundationdb start . You can restart a running instance of the database by executing service foundationdb restart . Finally, you can completely stop FoundationDB by executing service foundationdb stop . All these commands should be executed with root privileges. On the used installation, the database files of the FoundationDB instance are stored in the 4500 directory inside the /var/lib/foundationdb/data directory. You can easily change the place where FoundationDB stores its data by modifying the configuration file.
The screenshot ( right) shows the output from the start, restart, status and stop commands, as well as the contents of the data directory of FoundationDB.
Over the page, the screenshot shows the output of the status command, which gives you information about your running FoundationDB instance. Among other things, you can see the path of the cluster file used, the redundancy mode used, the workload of your FoundationDB instance, the memory that is available to FoundationDB and the storage engine used. The Cluster File stores a connection string that’s composed of a cluster identifier and a list of IP addresses that specify the coordination servers. The Coordination Servers are used for maximising the fault tolerance of a FoundationDB cluster, in case one or more cluster machines have connectivity issues.
On a cluster configuration that has only one machine – such as the installation used in this tutorial – the cluster file will look like the following: $ cat /etc/foundationdb/fdb.cluster GNdd2hDP:YnaZHBEh@127.0.0.1:4500 The main configuration file of FoundationDB is called foundationdb.conf and is usually located at /etc/ foundationdb whereas the log files of FoundationDB are usually stored in /var/log/foundationdb/. The log files of FoundationDB use the XML format and the log entries look similar to the following: <Event Severity=”10” Time=”1535567080.132189” Type=”CodeCoverage” Machine=”127.0.0.1:6206” ID=”0000000000000000” File=”fdbclient/ ReadYourWrites.actor.cpp” Line=”1162” Condition=”true” logGroup=”default”/>
A directory in FoundationDB is a way of creating a path similar to a UNIX directory, to better administer your applications by using a different directory for storing the data of each application. A subspace in
FoundationDB is used for defining namespaces to store different kinds of data. Both directories and subspaces are used for improved data organisation.
Command lines
You are going to need to know how to perform basic tasks with the FoundationDB client: storing, retrieving and deleting data. You can create a new key value entry
from the FoundationDB command line utility, which is named fdbcli, as follows: fdb> writemode on fdb> set “Hello” “World!” Committed (22721077671)
The first statement is used for allowing the writing of data to FoundationDB and it should be executed only once. You can disable that functionality by executing
writemode off . The next statement stores a key named Hello with a value named World! into the database. If you try to insert a key value pair that already exists in the database, nothing will happen.
After that you can retrieve that entry with the get command, as follows: fdb> get “Hello” `Hello’ is `World!’ The getrange command can also produce a range
of values. The “” value used here means get everything: fdb> getrange “” Range limited to 25 keys `Hello’ is `World!’ `Linux’ is `Format’ Finally, you can delete an entry with the clear command, as shown here: fdb> clear “Hello” Committed (22864240953) fdb> get “Hello”
`Hello’: not found
Although you can use the commit command to commit the current transaction, fdbcli operates in autocommit mode so there’s no need for that.
Because you’ll most likely use FoundationDB from the programming language of your choice and not the fdbcli utility, you’ll find the next two sections that illustrate how to access a FoundationDB database using Python 3 and Go pretty useful.
Python in Foundation
Coders are, of course, going to want to use Python 3 to interact with a FoundationDB server. In order to be able to talk to FoundationDB from Python 3 you’ll need to have a Python 3 module named fdb installed. You can find information about downloading the Python 3 fdb package at https://apple.github.io/foundationdb/ downloads.html – you’ll have to install it on your own. The name of the Python 3 script will be fDB.py and the logic of the script can be found in the following statements of code: fdb.api_version(520) db = fdb.open() @fdb.transactional def add_issue(tr, c):
tr[LXFSub.pack((c,))] = fdb.tuple.pack((100,)) @fdb.transactional def available_issues(tr):
return [LXFSub.unpack(k)[0] for k, v in tr[LXFSub. range(())]]
Calling the fdb.api_version() function before actually using the functionality of the API is mandatory for the API to become available. Additionally, the @fdb.
transactional statement is provided by FoundationDB in order to make the lives of developers easier, because it automatically creates a transaction and retries until success. So, the use of @fdb.transactional makes each function a transactional function and requires the use of an argument named tr that enables each function to perform reads and writes. Moreover, the
add_issue() function is used for adding data to the LXFSub subspace of the database. Finally, the
available_issues() function reads all the data of the LXFSub subspace and returns that to the specified calling function.
The code differences between a fDB.py Python 3 script in combination with an another.py script are shown here: $ diff another.py fDB.py 16c16 < years = [‘2015’, ‘2016’, ‘2017’, ‘2018’] --> years = [‘2011’, ‘2012’, ‘2013’, ‘2014’] 26c26 < del tr[LXFSub.range(())] # Clear the directory --> # del tr[LXFSub.range(())] # Clear the directory This means that another.py uses different values in years and the another.py deletes the sub space before inserting any data to it. So, if you call another.py first and fDB.py second, fDB.py will also display the data inserted by another.py.
FoundationDB and Go
As you might have guessed, in this section we’re looking at how to communicate with a FoundationDB database from Go using the github.com/apple/foundationdb/ bindings/go/src/fdb Go package by developing two Go programs. The first one is named hFDB.go and shows how you can connect to FoundationDb, write an entry and retrieve that entry. The Go code of hFDB.go is the following: package main import ( “fmt” “github.com/apple/foundationdb/bindings/ go/src/fdb” ) func main() { fdb.MustAPIVersion(520) db := fdb.MustOpenDefault() key := “Hello” _, _ = db.Transact(func(tr fdb.Transaction) (ret interface{}, e error) { tr.Set(fdb.Key(key), []byte(“World!”)) return }) ret, _ := db.Transact(func(tr fdb.Transaction) (ret interface{}, e error) { ret = tr.Get(fdb.Key(key)).MustGet() return }) v := ret.([]byte) fmt.Printf(“%s, %s\n”, key, string(v)) } Four important things happen in hFDB.go. First, the db.MustAPIVersion() call specifies the API version that will be used, which enables programs to know what to do even if the API is modified in the future. Then, you
have to initialise the connection using
MustOpenDefault() . After that, the db.Transact() function is used for interacting with the FoundationDB database. Finally, the implementation of the function that’s given as a parameter to the db.Transact() function specifies the functionality that you want. It’s more or less the same idea as in Python, but with different statements.
The Set() function is used for adding a new key to the database. The first function parameter is the key and the second function parameter is the value associated with that key.
Please note that the error-checking code in hFDB.go has been omitted in order to make the program shorter – you should never go that far in real-world applications!
In order to download the fdb Go package along with its subpackages on your local machine, you’ll need to execute the go get github.com/apple/foundationdb/ bindings/go/src/fdb command.
Managing multiple entries
The name of the second program is foundDB.go and performs two main things. First, it initialises
FoundationDB and creates a subspace. Then it inserts multiple entries in the database and after that retrieves all that data.
The Go code for working with directories and subspaces is the following: d at aDir,err:=di rectory. Create Or Open(db ,[] string{“myData”}, nil) varLXFSub=d at aDir. Sub (“linux format ”) The first statement defines a new directory named
myData whereas the second statement defines a new subspace that’s saved in the LXFSub variable and is named linuxformat . Apart from the Go code that deals with directories and subspaces, the following Go code populates the database using the data stored in the issues slice: _, err=db. Transact( func(trfdb. Transaction) (interface{}, error) { tr. Clear Range( d at aDir) for i := range issues { tr.Set(LXF Sub. Pack( t up le. T up le{ issues[i ]}),[] byte( strconv. Format Int (100,10))) } return nil, nil }) Finally, the following Go code retrieves all the data using an iterator provided by FoundationDB: _, err=db. Transact( func(trfdb. Transaction) (interface{}, error) { ri:=tr.GetR an ge(LXFSub,fdb. Range Options {}). Iterator() for ri.Advance() { kv := ri.MustGet() t,err:=LXFSub.U np ack(kv. Key) if err != nil { fmt.Println(err) return nil, err } f mt. Println(t [0].( string )) }
return nil, nil })
The ri.Advance() keeps bringing new data from the database, whereas the code inside the for loop decodes the data and prints it on screen. The value of kv.Key is
linuxformat, which is the name of the subspace that’s kept in the LXFSub variable. The screenshot ( below) shows the output of the
foundDB.go Go program. The last part of foundDB.go tries to read the data from a subspace named
doesNotExist that does not exist, which is the reason why you don’t see any data after the “Printing anotherSS” message.
The documentation page of the fdb Go package can be found at https://godoc.org/github.com/apple/ foundationdb/bindings/go/src/fdb.
The presented Python and Go code should be enough for you to start using FoundationDB programmatically and start writing interesting applications! After all, you have just talked to a distributed and consistent NoSQL database. Today, a database… tomorrow, the world!