This issue, we look at one of the options available to make sharing your Raspberry Pi projects easier
How to more easily share your Python projects using various utilities
We’ve looked at several techniques in Python programming that you can add to your own project, but once you have a project put together, how do you share it with the broader audience of DIY and maker people out there? In this tutorial, we’ll look at one of the options available to package your Python program, called PyInstaller. This is designed to package together everything you would need in order to run your code on a different machine. This is probably the easiest way for a new user to pick up your project and start using it.
We’ll see how to install PyInstaller, how to configure it for your project, and also some possible problem areas that might come up in special cases. In addition we’ll look at some other options, such as py2exe, which offer a different set of capabilities that could fit your requirements better. As in most cases, there are different tools because people have different requirements that need to be fulfilled.
So what does PyInstaller actually do to package your project up? Normally, you give it a single Python script that is the starting point for your project. PyInstaller then goes through it and collects a list of all of the other modules and scripts used by this main script. All this code is then copied into a distribution directory, along with the active Python distribution being used on the system. At this stage, you should have everything you need to run the project on a similar system.
A similar system, in this case, means the same type of hardware and the same operating system. This means that if you want to create a package for a particular system, say macOS, you will need to build the PyInstaller package on that same system – you can’t make a ‘universal’ package that works on everything.
Once all the files are collected into one location, you can create an executable file which includes all the files needed. There are several options available to control how these files are unpacked on the end-user’s system. We will look at a few of them later in this article, to see what options you might find most useful.
Most distributions do not include PyInstaller as a module managed by the system package management system, which means you’ll need to install it using pip. Most Linux distributions have different versions of pip for Python
2.x and Python 3.x. For example, if you wanted to install PyInstaller for Python 3, you could use the following:
sudo pip3 install pyinstaller
This installs PyInstaller to the system Python library. If you want to have the latest development version, you can hand a URL to pip, as in the following example.
sudo pip3 install https://github.com/pyinstaller/ pyinstaller/tarball/develop
Once this is done, you should everything you need to start packaging up your project. You can verify that everything installed correctly by running the following:
With PyInstaller installed, you can use it right away with very little effort – for small projects, at least. Let’s say your project all fits inside a single Python script and uses just a few of the most common modules. In this case, you can simply change directory to the one where your script resides and run the following command:
This creates a subdirectory called myscript, and places all of the requirements for your script there. This includes the Python engine that was used to run your script. To run your program, an executable is also created, called myscript, that launches your code for the end user. Sharing is as easy as zipping up the entire folder and sending it to interested parties. They then simply have to unzip the file and run the executable.
One big advantage to this method, at least for initial development, is that everything that PyInstaller thinks you need is copied into a single location. You can then verify that it did the right thing very simply by going through this destination directory. If PyInstaller did make some mistakes, you can easily figure out what these mistakes are and correct them before they go out. Once you’re confident that everything is working correctly, you can use a different option to create a single big executable that contains everything needed, as in the command below:
pyinstaller --onefile myscript.py
This single executable file can then be sent to your users. This is a much more comfortable experience for them, so you will likely want to use this method as the final delivery option. When your user runs this executable, it self-unpacks into a temporary directory, and then runs the executable, just as in the folder method above. When it finishes, it should clean up after itself and delete all the temporary files. If anything fails here, there may be temporary files and directories that need to be cleaned up manually. That’s not a particularly major problem, but rather just a potential event that you should be aware of.
More complex setups
If you have everything contained within your Python code, you’re done at this point. But for many programs, you will have some extra data or files that need to be part of the package which is sent out. For example, say you had a readme file that contained important information for the user. In these cases, you can include them with a command like the one below.
pyinstaller --add-data "src/README.txt:." myscript.py
The format is SOURCE:DESTINATION for the --adddata option. In the above example, we are taking the readme file from the src subdirectory and putting it in the root distribution directory. If you have multiple files, you can include as many --add-data options as needed.
This is fine for data files and documentation, but what if you have some piece of optimised code in a shared library? In this case, you use the --add-binary option to add a list of files to be included in the final executable. It has the same format as the --add-data option, and you can also have multiple --add-binary options. With this option, you are telling PyInstaller to include those binary files in the list of files to package, but also to check these binary files to see what dependencies they have. This is where PyInstaller may have the most difficulty, so you will definitely want to test the executable being created before sending it out to the wider world.
Create spec files
For smaller projects, you can probably manage the executable creation with the options --add-data and --add-binary directly on the command line. This way of doing things will very quickly become untenable, however. Luckily, there is another way of organising everything that needs to be bundled together into the executable version of your program: the spec file. This file contains everything that you might want to configure about how your executable operates.
The first step is to create a spec file that you can edit for your project. The following command does just this.
If you have everything contained within your Python code, you’re done at this point. But for many programs, you’ll have some extra data or files that need to be part of the package
To make this starting spec file as specific as possible to your project, you can include any command line options that you would have included when calling PyInstaller directly. See the following for a very simple, one-folder Python script:
block_cipher = None a = Analysis(['myscript.py'],
binaries=None, datas=None, hiddenimports=, hookspath=None, runtime_hooks=None, excludes=None, cipher=block_cipher) pyz = PYZ(a.pure, a.zipped_data,
cipher=block_cipher) exe = EXE(pyz,... ) coll = COLLECT(...)
This script is run as code by PyInstaller, and ends up creating objects of the four classes Analysis, PYZ, EXE and COLLECT. The Analysis object checks for all the files that are meant to be included and generates the list. The COLLECT object actually puts copies of all the required files into the distribution directory. The
EXE object generates the executable file that will be used to run the program. If you are sending out a ZIP file containing the distribution folder, this is where PyInstaller stops. If, instead, you generate a single executable file, PyInstaller uses the PYZ object to handle this step.
As you can see in the above example, there are several entries to manage the creation of your standalone application. The entries datas and binaries are equivalent to the two command line options we looked at earlier, --add-data and --add-binary. So for example you could use the code below to add a readme file to your packaged application.
As you can see, the list of files is simply a Python list. This means you can easily add a large number of files without too much trouble.
There is a lot more you can do during the generation of your executable. For example, you can add command line options to Python to be used when it runs your program. The following code adds the Python command line options v (to write a message to stdout each time a module is initialised) and W ignore (an option to change warning behaviour).
options = [ ('v', None, 'OPTION'), ('W ignore', None, 'OPTION') ] a = Analysis( ...
In order to use these options, you need to add the following to the creation of the EXE object:
exe = EXE(pyz, options, ... )
For when your project actually involves several different Python programs that it makes sense to package together, PyInstaller includes a function named MERGE. You can create a spec file where you define a series of Analysis objects, one for each program. You then define a MERGE entry, listing each of the Analysis objects that should be processed.
Check for problems
Because the P y thon environment is large, complex and everchanging,youwillinevitablyrunintoissueswherePyInstaller fails to generate a working executable version of your program. Luckily, it provides some tools that you can use to debug these problems.
The first option is to look at the output file where all warning messages are written. PyInstaller creates a subdirectory called build where the program is collected and built. Within this subdirectory is another subdirectory named after your script. So in our example, the warning file would be named ./build/ myscript/warnmyscript.txt. All the warnings from PyInstaller are written here, such as messages like “module not found”. If you want to see these messages on the console, you can use the command line option --log-level.
The dependency graph
A second tool available to you is the dependency graph. An HTML version of this is built automatically by PyInstaller. You can find it in the file ./build/myscript/ xref-myscript.html. This file lists the contents of the import graph; you can navigate around it by clicking the ‘imported by’ labels. If you use the --log-level command line option and set it to DEBUG, PyInstaller also generates a GraphViz file called ./build/myscript/ graph-myscript.dot. This file can then be run through GraphViz to generate a graphical representation of the dependency graph. This file can get rather large, since there are so many imported Python modules, even in the simplest of programs.
There is a lot more you can do during the generation of your executable. For example, you can add command line options to Python to be used when it runs your program”
These two tools will help you to identify what PyInstaller is looking for, and what it is having difficulty finding. There are also a number of special cases that have already been identified, such as building django files or packaging multiprocessing programs. They are available under the recipes section of the PyInstaller documentation site.
PyInstaller is not the only option available to package Python programs. Two other popular options are freeze and py2exe. Freeze actually comes in a couple of implementations, cx_Freeze and bbFreeze, although the latter hasn’t been updated in quite a while, so it might be more trouble than it’s worth.
cx_Freeze produces results similar to the distribution folder output from PyInstaller. Running the command:
cxfreeze myscript.py --target-dir dist
produces an executable file, plus all required libraries, within the dist subdirectory. If you want to distribute this to end users, you need to zip up the target directory and have the user unzip it locally. One part that is different from PyInstaller is that cx_Freeze doesn’t handle external data files as cleanly. You would need to add some extra code to manage locating them, as discussed in the cx_ Freeze documentation.
py2exe is a very different option to cx_Freeze and PyInstaller. py2exe is actually an extension to distutil. This means that you need to create a setup.py file in order to generate an executable version of your program. Using our
example above, the following would be a basic setup file:
from distutils.core import setup import py2exe
As you can see, the setup.py file imports both distutils and py2exe. You can then run this file with the following:
python setup.py py2exe
This process generates a huge amount of output, and also creates two subdirectories called dist and build. build is the working location where all the work of creating the executable happens; dist contains the final form of the executable. In order to test it, enter the dist subdirectory. You can then test it with the command ./myscript. To distribute this executable, zip up the dist subdirectory, as with cx_Freeze.
With both of these programs, dealing with special cases is more difficult than with PyInstaller. If your project is basic, any of the executable creation tools should be sufficient. If your project is more complex, however, or uses a fair amount of external data and/or code, PyInstaller might be your better option.
Sharing Python code has always been an issue. The combination of having an interpreted language, along with a huge collection of third-party modules, is usually a powerful environment for program development. However, this very same strength becomes a weakness when you want to share your work with others. Hopefully, this article has presented some ideas that you can use in order to share your Raspberry Pi projects easily.
Above By default, PyInstaller builds a dependency graph for all the imported modules in your program
Left PyInstaller starts by creating a spec file. You can edit it to add everything you need