While Arduinos are mainly used for controlling hardware, they can do a lot more – find out how to use an Arduino to create your own web server which can double as an IOT device
Arduinos are designed for hardware, but the right model can still power through a lot of computing
Establishing an online presence is a task easily accomplished with a personal website. While websites are a great platform to showcase your skills, there’s no reason we can’t show off our Arduino prowess and avoid paying money to host content on a server at the same time. This month we are going to begin a series of Arduino tutorials that take the microcontroller board out of its comfort zone and we start by transforming it into a simple HTTP web server.
Alarm bells should be ringing in your head right now: aren’t Arduinos meant for hardware projects? Well, yes entirely. Arduinos also have a maximum program size and a relatively small amount of memory, making this difficult. We could also do this cheaply with an actual, albeit small, computer such as the Raspberry Pi. We also won’t be able to do much (if any) server-side processing – so no database queries. These are three strong arguments against using an Arduino for this. However, if you have an Arduino Yún then this isn’t the case at all, and if you have a Mega the program size isn’t an issue and extra features can be added as needed.
The benefit of using an Arduino and programming your own server is that you can also serve other commands of your choosing. This allows your web server to double up as an IOT device, controlling items around your home. You could be boiling a kettle, opening the garage door, or even controlling motors on some kind of robot you built yourself, all feeding back to the user on their browser. An Arduino server can become the gateway into your world.
Set up the hardware
You’ll need an Arduino board, a way of connecting it to your internet router and SD card reading capability. For this tutorial, we used the Arduino Leonardo and the Ethernet & SD card shield. Immediately, you’ll need to insert the shield into the Arduino’s header pins and connect your shield to the router with an Ethernet cable. You’ll also need to power your Arduino – but for now, plug it into your computer with a USB cable.
You should log into your router, identify your device’s local IP address and check that you have configured your router correctly. If you don’t have access to the administrator login, you might be able to determine its network address using nmap or arp -a. At the top of the example ‘WEBSERVER’ sketch (under Ethernet Library sketches), you’ll need to change your IP and MAC addresses to match.
Upload this sketch, as is, to your Arduino. If all is well, you should be able to connect to your device from any browser and it should display a sensor reading. As you haven’t connected anything to the input pin, it’s probably going to be zero or just garbage, but that’s okay, we’re going to get rid of it anyway.
Read HTTP requests
Open the serial monitor within the Arduino IDE and connect to your device on your browser. A stream of text should appear. This is the HTTP request that the browser is sending to the Arduino. At the beginning is the request method in all caps, followed by the resource requested. Briefly, the browser is sending a request to ask the server to GET a resource, such as a webpage
(in this case just ‘/’). A web server should then send a response message after determining the nature of the request and assessing resources. For example, if a page is not found, it should reply ‘HTTP/1.1 404 Page Not Found’. This is a header and would be a response to a HEAD request. This can be followed by a message, such as the main body of a webpage.
This allows your web server to double up as an Internet of Things device, controlling items around your home
What we are going to do here is emulate the server behaviour by responding appropriately to certain HTTP request types. To begin, you’ll need to write a function to read the HTTP request letter-by-letter, saving to a character array, and pausing when you receive a whitespace character. You should then convert that character array to a string and compare the received request to the methods your Arduino is going to serve.
For now, we’ll deal with GET and HEAD. You’ll also want to store the resource name.
Serve HTTP requests
The GET request is just like the HEAD request, but with an additional body that follows the header information. To save on program space, the GET request should call a function that returns the header information and then finishes off with the message body. Begin by making a function which can reply with the HEAD information, as in the first four client.println() calls in the example code. Depending on the availability of resources on the SD card and the request sent, we will want to change this code by passing it in as an argument, but for now we’ll just use the ‘200 – OK’ code.
Now add another function which can process GET requests. This function should check the SD card for a file with the same path as the requested resource and return an OK code if present, or an error code if not, followed by the content of the resource (or an error page). This check can be done with SD.exists(), but to initialise the SD card you’ll need to add a line at the top of the sketch:
To return the content, you also need to declare and open the file, read the contents (see the SD card example sketches), then print the file contents back to the client.
Create the website files
When saving your files, be sure to not use more than three characters in the extension and eight characters
in the filename. The SD library won’t allow more, and it’s very easy to think your code isn’t working rather than assume it’s something as simple as this – your author wasted a day on this problem: use .htm. When done, insert the SD card into the shield and reset your Arduino.
Test and fix your server
In the main loop you should now be able to compare the string comprising of the requested method and the methods served by the Arduino and then call the appropriate function. Upload this new sketch to the board and try out your website. You should be able to use your local IP address followed by the filename, for example 192.168.1.42/home.htm. The server isn’t yet fully operational, but we’re getting there. As you navigate, several problems should become apparent.
First of all, regardless of what happens, the ‘200 – OK’ code is returned. Second, the server is always responding with ‘type/html’, even if the Arduino is actually sending over JPEGs or plain text. Also, if any images are large, you’ll already be experiencing performance issues and, if too large, you can be denied service altogether.
To fix these problems, you’ll need to create a function which determines if the file is on the SD card and, if so, look at the extension and return a string for the response code and the resource type. This should be used in the HEAD method when called. You should also return a 404 error if there is no file, and a 405 error if the method isn’t served by the Arduino. Images should be compressed and data sent in batches using a buffer.
Prune your program
If you’ve fixed the above problems then pat yourself on the back: you’ve managed to create a miniserver which can host a basic website that, with the right router configuration, can be accessed from the outside world. Any cool features, like CSS animations, filters and scripts, will need to be done client side, and any database lookups just can’t be done. The good news is that we at least comply with some of the HTTP protocol and, for most viewers, this will just seem like a normal website. Nevertheless, our functionality is still quite restricted. Let’s suppose we want someone to be able to leave comments, or use this to run a personal blog. We’re going to need to add in another HTTP method: POST.
However, as those using an Arduino Uno or Leonardo have probably noticed, we’re already running up against the maximum program size, which is limited to 28KB.
The SPI, SD and Ethernet libraries take up a lot of the program space, but we’ve contributed our fair share too. It’s time to get rid of any unnecessary print statements (although keep printing the incoming request to screen),
Creating a forwardfacing minimalist website is possible and can be done in a few hours
and clean out the code. We only need to make a little bit of room for an extra function and an extra if condition in the main loop.
Serving POST requests
POST requests tell the server that a new resource needs to be created. On an HTML page these requests can be generated using the form and input tags. These allow you to specify the method (e.g. POST) as well as the action (designating the page to be loaded after submission). When a user submits, the HTTP request sends the method, the resource to load and, at the very end after two line breaks, the message submitted. This message is in the format ‘field=message data’.
To serve POST requests, you will want to identify the requested method and call a new function. This function should read the name of the resource specified by the action in the form and save it to a spare buffer. It should then open a file on the SD card, read from client and save everything after the double-line break. The function
should finish by calling the GET function (which also calls HEAD) and retrieving the stored resource name. You may need to modify the existing code and will want the filename to increment. If you still have program space, you may also want to split the field name from the data in the message body by checking for an = sign and use the field name as the filename for the new resource created.
Load submitted posts
To finish the blog or comments section, you’re going to want to load all of the information submitted to the server and display it on the webpage. One might instinctively think of storing the information in a database and accessing entries as needed. The only problem is, we can’t install software or try PHP and MYSQL. Instead, if you’ve got an Arduino with larger program storage, you might want to make a custom method which acts like a database query. If you’ve got an ‘ordinary’ Arduino, everything’s going to have to be done on the client’s side.
Share with the world
The web server is finished. You can send and receive webpages, text, images, GIFs even. It should even run reasonably quickly. With a static IP or DynamicDNS, you could buy a domain and let people connect to your website without handing out a series of numbers.
However, there is still the issue of sending and receiving obscenely large images. This will always be slow – and is even the case with some popular websites. For now, we can ensure that we only host small images, but if we really need to we could always link to the resource on a faster cloud storage service instead.
But there are a hundred-and-one ways someone could deny all service to the Arduino server. We’ve also added no security for data stored on the device. However, at least there’s no way of someone deleting information or using the Arduino to connect to other local devices.
Add hardware methods
That was Arduino truly out of its comfort zone! Now the web server is complete, let’s bring the device back to what it’s good at: interfacing with hardware. We can use the web server to act as the infrastructure for controlling the Arduino remotely. To do this you’ll need to create a new request comparison condition within the loop function, as with the other HTTP requests. To control the device, you can just send this arbitrary keyword and any further instructions over Telnet (or equivalent) to the server address to trigger an action for the Arduino to perform. You could even create a very simple smartphone app that sends a preset command at the touch of a button.
And what to do with it? At this point the task could be anything at all, from turning on a light to watering some flowers. Depending on the size of the web server program, it might be useful to use this Arduino as a base station which then relays the command via radio to other Arduinos scattered around the house. The bigger the project, the more likely you are going to need to upgrade to the larger Arduino. Alternatively, you could piggyback off of the POST method and put the world in charge of your hardware using a simple HTML form as a controller.