Node-RED: MicroPython
Tim Armstrong helps you take over the AirWaves with a wireless “Thing” on the Internet, using LoRa, MicroPython and TheThingsNetwork.
With our wireless “Thing” beaming data over LoRa, Tim Armstrong switches to using Node-RED to collect and process the data into pretty graphs we can all use!
For this tutorial we’re going to take live data that we’re collecting from TheThingsNetwork and present it in a simple website. To build our website we’ll be using Node-RED, which is an excellent flow-based graphical programming language. You drag nodes that represent inputs, functions and outputs into a flow that can be deployed as a service on anything from a Raspberry Pi to a cloud platform.
Previously in LXF235 we built a simple sensor that measured the voltage across a light-dependant resistor (LDR) and sends that data over LoRa to TheThingsNetwork (TTN). This circuit design can be applied to most analog sensors (and even some microphones) by simply replacing the LDR with the analog sensor you want to use.
Node-RED is offered as a “boilerplate application” on IBM Bluemix. We can just sign up for a free developer account (called a lite-account) and launch an instance. This enables us to get started on what we want to do without needing to think about server deployment, installation, management and software updates. All that stuff is left to IBM’s Network Operations Center. RED nodes on a Bluemix To get a lite-account on Bluemix head to https://
console.Bluemix.net/registration. Fill out the form and press Create Account. You’ll then need to open a link in your email to confirm your sign up. You can just skip through the introductory overview because it doesn’t really tell you much. By default, you’re automatically given a lite account, so we can go straight to the catalogue and select a Node-RED Starter from the Boilerplates section of the catalogue.
Give your app a unique name, select a region to deploy in, and click Create. Once it’s started we can get to the Node-RED flow editor by clicking the Visit App URL link next to the title of our App and follow the setup wizard to secure your Node-RED instance. Finally, click the Go to your Node-RED flow editor link.
Now we’re in the flow editor we can see there’s a palette of nodes on the left, a debug panel on the right and the workspace in the middle. Now we need to add a couple of node sets to our palette. Since we’re creating a dashboard site where we can see and interact with the data from our IoT device, we’re going to add the nodered-dashboard set, and because our IOT device is pushing data to TTN we’re going to add the node-redcontrib-ttn set. To do this we need to modify the deployment and redeploy our Node-RED instance via the Cloud Foundry CLI.
To install the CloudFoundry CLI we can download it from the link in the Getting started page of the Node-RED section of our Bluemix Console. Once you’ve got it downloaded open up a terminal in the same folder and run the following:
$ sudo dpkg -i cf-cli-installer_6.34.1_86_64.deb
While it’s busy installing, download our deployment’s starter-code from the Bluemix Console and then extract it. Next, open up the package.json file in your preferred text editor and the following lines to the top of the dependencies section, before saving the changes:
“node-red-contrib-ttn”:”2.0.x”, “node-red-dashboard”:”2.8.x”,
The next step is to open a terminal in the starter-code directory, set the Cloud Foundry API URL to the Bluemix API for the region you chose to deploy your Node-Red instance in: $ cf api https://api.<REGION>.bluemix.net And login using: $ cf login -u <YOUR_EMAIL_ADDRESS> Now you’re all logged in we can push our modified deployment to bluemix using $ cf push <YOUR_APP_NAME>
Once the redeployment is finished you’ll need to refresh the node-red flow editor page and log in using the credentials that we created in the Node-RED set-up wizard earlier.
Incoming transmission
A common misconception is that TTN stores data. This seems to be because it has that handy Data tab in the console. But neither TTN nor Node-RED store data, so the first thing we need to do is set up our data storage so that we can store the incoming data.
Because Node-RED keeps its configuration and flows in a BlueMix-hosted CouchDB called Cloudant, we can make use of the same cloudant instance and store our data there.
To do this we need to drag a cloudant out node from the storage section of our palette and double-click it. This brings up its configuration panel. From the configuration panel we can select our cloudant service, then we can give our new database a name so let’s call it lopy. Make sure our operation is set to insert and that the Only store msg.payload object checkbox is ticked. Now we can click the Done button in the top right of the config panel.
Next, we need the data source, which in this case is a ttn uplink node, which we can find in the Inputs section of our palette. Again, we need to enter its configuration, so go ahead and double-click it to open the panel. Then in the App section you’ll notice that there’s a pencil icon to the right. Clicking that will bring up the TTN access key sub-menu. To fill this out we need a couple of fields from our TTN Console ( https://console.thethingsnetwork.org) so let’s open that in another tab.
In the TTN Console we need to go to Applications and select our application, and from the Access Keys section near the bottom of the page click Manage keys and then Generate new access key. Give the new key a name and make sure the only checkbox that’s checked is the Messages one. Then click the Generate Access Key button in the bottom right and navigate back to the Overview tab. Under the Access Keys section you’ll now see our new key. Copy it to the clipboard by using clicking the clipboard icon to the right of the obscured key.
Back on our Node-RED tab we can type our Application ID and paste the Access Key in their respective fields. Finally, click Add and then Done to save the settings.
Now we have our input and our storage ready, we just need to add a timestamp to the incoming data and store it. Because this isn’t a predefined node that we just drop in, we need to use the function node to do it ourselves. The function node enables you to modify the content of the msg object using javascript. Convention dictates that a function node only does one task, which should be clearly indicated by its name. To that end let’s drag a function node into the flow, double-click it, call it Add Timestamp and add the following line to the top:
msg.payload.time = Date.now();
Click Done, and now it’s time to connect the nodes in our first flow. Click and drag from the dot on the right of the TTN uplink node to the left dot on the function node, then do the same between the Add Timestamp node and the cloudant node. Finally, click the Deploy button in the top right of the console. Now each measurement from our LoPy that gets received by a LoRa Gateway and sent to The Things Network will be timestamped and then stored as a document in our database.
Seeing is believing
Storing the data is useful, but this means that we have to retrieve it from the database before we can see it – and we’ll get to that – but while we’re still dealing with the live data, let’s set up our first dashboard element. For this we need another function node and a gauge node: you can find this node under the dashboard section of the palette. Our function in this case can be called get volts and is as simple as:
msg.payload = msg.payload.millivolts/1000; return msg;
Now let’s drag another line from the right dot of our TTN uplink node, this time connecting it to the our get_ volts function and then drag a line from its right dot to the gauge node. Now we can hit the Deploy button again and then in the dashboard tab of our debug panel we can click the little pop-up icon in the top right, which
should open a new tab showing our gauge, and after a little while it should show the latest received value from our LoPy.
Sitting on the job
Now we’ve got our first dashboard element going, it’s time to look at the second. Back to the Node-RED editor we need to start our next flow. To do this we need to click the little [+] tab to the top-right of our workspace. Retrieving the history is a little more involved, so we’ll build it in sections, specifically as a Query, a Response and a View.
To start our flow we need an input, in this case we’re going to use an inject node, so drag one into the flow and double-click it. Then set its payload to “boolean”, the repeat to interval, and set it to run every 120 seconds.
Because of how Cloudant works, we need to create a search index so that we can query against it later. In this case that’s the “time” field we injected in the first flow. To do this we need to go back to our bluemix console ( https://console.bluemix.net/dashboard/apps/) and select the Cloudant instance that was created automatically for us, and then click the Launch button. We should now be presented with our two databases – double-click the one called lopy.
To create our search index, click the (+) next to the Design Documents item from the menu on the left. Then from the sub-menu select New Search Index: let’s call it by_timestamp and our index time. Then in the function section replace index(“name”, doc.name); with index(“time”, doc.time); and press the Create Document and Build Index button at the bottom of the page. To test our index copy time: [0 TO Date.now()] in to the query field and hit Enter. This should return all of our measurement data.
Now we can head back to Node-RED and finish our query section. To do this we need another function node and a cloudant in node, so let’s drag those into the flow and double-click the function node. In this function we need to replace the payload with a query similar to the one we used to test our database, but because we’re this time only interested in the last 24 hours, to do this insert msg.payload=”time:[“+(Date.now()-86400000)+” TO “+Date.now()+”]”; above the return. Moving on to the cloudant node, we need to set it up to use the search index we just created. Set the database name to lopy, then set the search by to Search index. In the left of the next two fields type by_timestamp and in the right one type time, then press Done.
Finally, connect the inject node to the left dot of the query function, and the function’s output to the cloudant node. If you want to test this now, drag in a debug node and connect it to the cloudant’s output and click Deploy. Then when you trigger the square on the left of the Inject node you’ll see the payload in the debug panel, which will be an array of up to 200 results.
Size matters
As you might have noticed, 24 hours divided into 100 second samples would be 864 results. This isn’t the maximum of 200 that the cloudant node is giving us, so how do we get the other 664 results? This brings us on to the Response section. If you set the Output setting of the debug node to complete msg object, hit Deploy and then trigger the inject node, then you’ll notice that there’s a cloudant section to our object. Expanding this you’ll see it contains a property called bookmark. The bookmark points to the next page in the response to our query. In order to use it we’ll need a function node and a switch node and a join node.
The join node holds each message it receives until a given condition is met, and then sends the bundle forward as one. In our case that condition will be if our msg object contains a property called complete, and we want it to combine our msg objects as an array. To configure this open the join’s settings, then set its mode to manual, and its “combine each” to msg.payload, and the “to create” to “an Array”.
The switch node passes on any message it receives if it meets a simple criteria. We want it to check if msg. payload “is not null”. To set this node up, change the drop-down that’s currently set to “==” to be “is not null”.
Next, we need to set up the function where we need to check if the payload we receive from the cloudant node has any results. If it has results then we want to pass the data on to the join and fetch the next page. If it’s empty then we’ve received all the pages and we can trigger the join by setting the msg.complete property on the message to the join and send a null payload to the switch. To do this we need to use two outputs: one for our join node and one for our switch node. You can control the number of outputs just below the code section. When we have more than one output we need to supply a list in the return with the same number of msg objects as the number of outputs we have. if (msg.payload.length > 0){ next_page = {payload:{}} next_page.payload.query=”time:[“+(Date.now()- 86400000)+” TO “+Date.now()+”]”; next_page.payload.bookmark=msg.cloudant. bookmark;
return [msg, next_page]; } else {
msg.complete = true; return [msg, {payload:null}]; }
Now connect the output of our cloudant node to the input of the function, then the top-right output of our function to the input of our join node, the bottom output of the function to the input of the switch, and the output of our switch to the input of our cloudant node.
Buena vista!
Finally, we need to pre-process our data so that it can be displayed in our 24 hours chart. Because of the way the join node works we have a payload that contains an array of the payloads of each of the pages we retrieved from the database. So the first stage of our preprocessing will be to depaginate the data by iterating over the pages and linking each page’s payload into one, which we will then sort by timestamp. To do this drag in a new function node and copy in the following: new_msg = {} payload = [] for (i=0; i<msg.payload.length; i++){ if (msg.payload[i].length > 0){ payload = payload.concat(msg.payload[i]) } } payload.sort(function(a,b){return b.time-a.time;}) new_msg.payload = payload return new_msg;
The chart from the dashboard palette requires the data to be supplied as an array containing one object that has two properties: one called series that contains an array of label(s), and one called data that contains an array of arrays that contain objects with an x property and a y property. Confused? Don’t worry, it’s not the most obvious data format unless you’ve worked with graphing libraries before. To do this create a new function object and copy the following code into it: t = Date.now(); payload = {} payload.series = [“Voltage”]; payload.data = [[]]; for (i=0; i<msg.payload.length; i++){
row = {x:msg.payload[i].time, y:msg.payload[i]. millivolts/1000}; payload.data[0].push(row); } return {payload:[payload]};
Last, but by no means least, drag in a chart node from the dashboard palette and set its x-axis to 24hrs. Connect the output of the join node to the input of the de-pagination function, connect the output of the de-pagination function to the input of the data formatting function, and connect the output of the data formatting function to the input of the chart. Then press the Deploy button, trigger the inject node and go to the dashboard. You should now see your finished dashboard with both your live data meter and your historic data cart.
So now we have a sensor to LoRa to website. You could add more sensors, set up alerts or use the TTN downlink to trigger an action on the sensor device. Congratulations on your Internet Thing!