pyroute2 – python manages networks
Linux tools use tricky protocols to communicate with the kernel. You can do it, too, but with the comfort of pure Python.
Being a veteran Linux admin, you likely remember times when you ran ifconfig to set up network addresses and route to manage IP routes. These days, a single all-in-one tool, ip, handles everything, from IP address assignment and route management to creating network namespaces and setting up tunnels between the hosts.
Given this versatility, it comes as no surprise that ip is typically the workhorse behind many networkautomation scripts. Quite often, these scripts are written in Python, and call with subprocess.check_ output() or similar. Then the output is parsed, the exit code is analysed and so forth. There is nothing wrong with this approach, but wouldn’t it be nice to have all of
ip in a Python-native format?
Sniffing communications
That’s what the library pyroute2 (https://pyroute2. org) is all about. ip uses Netlink to communicate with the kernel; most notably, the RTNL protocol as described in the rtnetlink(7) man page. pyroute2 implements it, and some more, in pure Python.
So you can do whatever ip does, in pretty much the same syntax, with no dependencies other than the Python standard library. The latter makes pyroute2 trivial to install even if your distribution doesn’t provide prebuilt packages. Installing from PYPI also gets you the latest and greatest pyroute2 (0.5.4 at time of writing), while packaged versions typically lag behind. The library supports both Python 2 and Python 3, so be sure to pick the right flavour.
pyroute2 supports several Netlink families and protocols (see ‘What is Netlink?’, right), and exposes these as top-level classes which wrap the respective sockets and associated machinery. In this Administeria, we’ll be mostly concerned with rtnetlink, which is made available through the Iproute class, and a few others.
Iproute implements context manager protocol, so you don’t have to close() the underlying socket manually, as we can see:
from pyroute2 import Iproute with Iproute() as ip: pass
Netlink communication is bi-directional. The userspace sends configuration bits such as addresses to be assigned. The kernel responds, but also emits occasional asynchronous events such as ‘address assigned’ or ‘route added’. If you want to monitor these events, you’ll need to bind() the underlying Netlink socket. So the simplest ip monitor command equivalent in pyroute2 could look like this:
from pyroute2 import Iproute with Iproute() as ip: ip.bind() while True: for ev in ip.get(): print(ev)
This should yield a result similar to one shown in the screen grab above. get() reads a single message which is decoded and printed to the console. There are quite a few fields, most notably ‘event’, which explains what happened ( RTM_NEWADDR means ‘address added’), and ‘attrs’, which are event-specific attributes conveying the details such as interface name or route metric. The ‘ifindex’ field shows which network interface the message belongs to; we’ll revisit this shortly.
While monitoring Netlink events is fun, it’s unlikely to be the main scenario for your automation scripts. Recall that we’ve come up with pyroute2 as a way to stop spawning ip. How can you use it for that?
Again, you begin with creating an Iproute instance. However, for cases other than monitoring, binding is no longer a requirement. Iproute has a one-to-one mapping to rtnetlink, as the pyroute2 documentation explains, and since ip is also close to that, both feel pretty much the same.
Imagine you want to create a veth (Virtual Ethernet) pair, bring it up, assign the local end an IP address and add a directly connected route to a remote network through this link. This is how you do it with pyroute2: from pyroute2 import Iproute with Iproute() as ip: ip.link(‘add’, ifname=’veth0p0’, peer=’veth0p1’, kind=’veth’) ifindex = next(iter(ip.link_lookup(ifname=’veth0p0’))) ip.link(‘set’, index=ifindex, state=’up’) ip.addr(‘add’, index=ifindex, address=’192.168.1.2’, prefixlen=24) ip.route(‘add’, dst=’172.16.0.0/12’, oif=ifindex) peer_ifindex = next(iter(ip.link_ lookup(ifname=’veth0p1’))) ip.link(‘set’, index=peer_ifindex, state=’up’)
Pretty straightforward, isn’t it? Iproute exposes different rtnetlink objects such as addresses or routes as methods; the first argument to this method is the command to execute. This follows the same pattern as
ip, which you typically call as ip OBJECT COMMAND
(see ip(8)). One notable difference is that ip accepts interface names where Iproute (and rtnetlink, in fact) operate on interface indexes. So we need to obtain the interface index by name first.
This task is accomplished with the help of the Iproute.link_lookup() method. However, it can list links not only by name, as we do here, but also any other first-level attribute, such as hardware address ( address ), operational state ( operstate ) or MTU. Listing by operstate , for instance, yields more than one value on a typical Linux machine: ip.link_lookup(operstate=’up’) # [52, 53, 58]
Thus the method returns a list, not a single value. That’s why we use next(iter()) to get the first (and, in this case, the only) element of it. For more complex searches, try ip.get_links() which retrieves the complete list of interfaces, along with their attributes, and dig as deep as you want. If the kernel signals an error for any of the above operations, pyroute2 throws a Netlinkerror. Comment out ip.link(‘set’) to see this in action. Now try to ping 172.16.0.1. Naturally, you’ll see no responses, as this address is not assigned anywhere in your system. So how do you check if it works as expected? One option would be to assign a suitable address to the peer end. This would yield you some replies, but in fact, they won’t go through the veth pair. To prove this, just bring the local end (veth0p0) down. You’ll be still getting replies as the address you ping is assigned to one of the local interfaces.
So, a better test would be not to assign anything and just generate some traffic, which will be directed to the veth pair by virtue of the route we’ve installed. While it’ll be left unanswered for now, you can still see the ARP requests which ping triggers in tcpdump. Bear with me: you’ll see the thing working end-to-end by the end of this Administeria.
A (tiny) bit of Docker
So you’ll want to hide the peer end of the veth pair to run the experiment as appropriate. Linux has a feature called ‘namespaces’, and Docker and friends have used them to provide container isolation from day one. A container is actually a set of namespaces: a process namespace to keep container PIDS separate from the host; a mount namespace so containerised apps can bring in their own libraries; a network namespace to keep network interfaces, routing tables and IP addresses contained, and so on.
We don’t need all these goodies for our task: a single networking namespace will do the job. A standard approach is to create a veth pair, then put its peer end into a separate namespace. This is easily accomplished with ip, so you may expect pyroute2 to handle this as well. And you know what – it does! Moreover, you can use a popen() -like API to spawn processes within namespaces. This is how you change the code above to create a netns creatively named ‘sandbox’ (yup, not ‘test’) and assign it to veth0p1 : from pyroute2 import Iproute, NETNS, netns netns.create(‘sandbox’) with Iproute() as ip:
# No changes to veth0p0 peer_ifindex = next(iter(ip.link_ lookup(ifname=’veth0p1’))) ip.link(‘set’, index=peer_ifindex, net_ns_fd=’sandbox’) with Netns(‘sandbox’) as ns: ns.link(‘set’, index=peer_ifindex, state=’up’) ns.addr(‘add’, index=peer_ifindex, address=’172.16.0.2’) ns.route(‘add’, dst=’192.168.1.0/24’, oif=peer_ifindex)
The netns module handles management stuff, such as namespace creation, deletion ( netns.remove() ) or listing ( netns.listnetns() ). You can also assign a netns to the current process with netns.setnents() : see the docs for details. Also note that the netns module uses ctypes (which is typically available), so it may violate local Selinux policies, as it issues syscalls that Python typically doesn’t.
Back to our code. After we finish configuring veth0p0 (not shown above for brevity reasons), the code determines the peer’s end interface index and assigns it a networking namespace with net_ns_fd keyword argument to ip.link(‘set’) . Note that moving an interface between namespaces clears all settings, so it doesn’t make sense to configure IP addresses first, for instance. A recommended approach is to have namespace assignment as a single standalone operation. After it completes, you can no longer see the interface in the main namespace. So how do you configure the peer end?
The NETNS class is here to help. Externally, it provides the same API as Iproute; you can even pass it to IPDB (see the box, right). Internally, it spawns a child process within the networking namespace you specified and calls into the multiprocessing module to exchange data. That’s why it’s important to call Netns.close() when you are done. Here, we rely on the context manager to do it for us.
The rest of the code is quite straightforward. We configure veth0p1 quite similarly to veth0p0 , except that the routes are reversed. Now, try pinging the peer end’s IP address from the main namespace: ping 172.16.0.2 . You should see the replies as expected. Looking in tcpdump would prove the traffic really goes through the veth pair.
If it doesn’t work, look for any exceptions pyroute2 may have thrown; note that, for simplicity, there is no
error handling in the script (though of course you should add some!). It could also be the case that 192.168.1.0/24 and 172.16.0.0/12 are already used in your system; if so, update the addresses accordingly.
Hopefully, this gives you some impression of pyroute2’s capabilities. Technology-wise, it’s admirable how it manages to provide a rich low-level feature set with no external dependencies other than the standard library. In the practical sense, pyroute2 can serve as a drop-in replacement for any occasion when you call ip in your scripts. It’s not source code-level compatible, of course, but as ip also follows rtnetlink’s approaches to doing things, converting one to another seems like a
sed’s job. Perhaps it’s not the library you’d want in every script you author, but it’s definitely worth keeping in mind (and your toolset).