How to Write Your Own Python Packages

Python is a wonderful programming language and much more. One of its weakest points, however, is packaging. This is a well-known fact in the community. Installing, importing, using, and creating packages has improved over the years, but it's still not on par with newer languages like Go and Rust that could learn a lot from the struggles of Python and other more mature languages.

In this tutorial, you'll learn everything you need to know to build and share your own packages. For general background on Python packages, please read How to Use Python Packages.

Packaging a Project

Packaging a project is the process by which you take a hopefully coherent set of Python modules and possibly other files and put them in a structure that can be used easily. There are various things you have to consider, such as dependencies on other packages, internal structure (sub-packages), versioning, target audience, and the form of the package (source and/or binary).

Example

Let's start with a quick example. The conman package is a package for managing configuration. It supports several file formats as well as distributed configuration using etcd.

A package's contents are typically stored in a single directory (although it is common to split sub-packages in multiple directories) and sometimes, as in this case, in its own git repository.

The root directory contains various configuration files (setup.py is mandatory and the most important one), and the package code itself is usually in a subdirectory whose name is the name of the package and ideally a tests directory. Here is what it looks like for conman:

> tree
.
├── LICENSE
├── MANIFEST.in
├── README.md
├── conman
│   ├── __init__.py
│   ├── __pycache__
│   ├── conman_base.py
│   ├── conman_etcd.py
│   └── conman_file.py
├── requirements.txt
├── setup.cfg
├── setup.py
├── test-requirements.txt
├── tests
│   ├── __pycache__
│   ├── conman_etcd_test.py
│   ├── conman_file_test.py
│   └── etcd_test_util.py
└── tox.ini

Let's take a quick peek at the setup.py file. It imports two functions from the setuptools package: setup() and find_packages(). Then it calls the setup() function and uses find_packages() for one of the parameters.

from setuptools import setup, find_packages
setup(name='conman',
      version='0.3',
      url='https://github.com/the-gigi/conman',
      license='MIT',
      author='Gigi Sayfan',
      author_email='the.gigi@gmail.com',
      description='Manage configuration files',
      packages=find_packages(exclude=['tests']),
      long_description=open('README.md').read(),
      zip_safe=False,
      setup_requires=['nose>=1.0'],
      test_suite='nose.collector')

This is pretty normal. While the setup.py file is a regular Python file and you can do whatever you want in it, its primary job is to call the setup() function with the appropriate parameters because it will be invoked by various tools in a standard way when installing your package. I'll go over the details in the next section.

The Configuration Files

In addition to setup.py, there are a few other optional configuration files that can show up here and serve various purposes.

setup.py

The setup() function takes a large number of named arguments to control many aspects of package installation as well as running various commands. Many arguments specify metadata used for searching and filtering when uploading your package to a repository.

name: the name of your package (and how it will be listed on PyPI)
version: this is critical for maintaining proper dependency management
url: the URL of your package, typically GitHub or maybe the readthedocs URL
packages: list of sub-packages that need to be included; find_packages() helps here
setup_requires: here you specify dependencies
test_suite: which tool to run at test time

The long_description is set here to the contents of the README.md file, which is a best practice to have a single source of truth.

setup.cfg

The setup.py file also serves a command-line interface to run various commands. For example, to run the unit tests, you can type: python setup.py test

running test 

running egg_info
writing conman.egg-info/PKG-INFO
writing top-level names to conman.egg-info/top_level.txt
writing dependency_links to conman.egg-info/dependency_links.txt
reading manifest file 'conman.egg-info/SOURCES.txt'
reading manifest template 'MANIFEST.in'
writing manifest file 'conman.egg-info/SOURCES.txt'
running build_ext
test_add_bad_key (conman_etcd_test.ConManEtcdTest) ... ok
test_add_good_key (conman_etcd_test.ConManEtcdTest) ... ok
test_dictionary_access (conman_etcd_test.ConManEtcdTest) ... ok
test_initialization (conman_etcd_test.ConManEtcdTest) ... ok
test_refresh (conman_etcd_test.ConManEtcdTest) ... ok
test_add_config_file_from_env_var (conman_file_test.ConmanFileTest) ... ok
test_add_config_file_simple_guess_file_type (conman_file_test.ConmanFileTest) ... ok
test_add_config_file_simple_unknown_wrong_file_type (conman_file_test.ConmanFileTest) ... ok
test_add_config_file_simple_with_file_type (conman_file_test.ConmanFileTest) ... ok
test_add_config_file_simple_wrong_file_type (conman_file_test.ConmanFileTest) ... ok
test_add_config_file_with_base_dir (conman_file_test.ConmanFileTest) ... ok
test_dictionary_access (conman_file_test.ConmanFileTest) ... ok
test_guess_file_type (conman_file_test.ConmanFileTest) ... ok
test_init_no_files (conman_file_test.ConmanFileTest) ... ok
test_init_some_bad_files (conman_file_test.ConmanFileTest) ... ok
test_init_some_good_files (conman_file_test.ConmanFileTest) ... ok
----------------------------------------------------------------------
Ran 16 tests in 0.160s
OK

The setup.cfg is an ini format file that may contain option defaults for commands you pass to setup.py. Here, setup.cfg contains some options for nosetests (our test runner):

1	[nosetests]
2	verbose=1
3	nocapture=1

MANIFEST.in

This file contains files that are not part of the internal package directory, but you still want to include. Those are typically the readme file, the license file, etc. An important file is requirements.txt. This file is used by pip to install other required packages.

Here is conman's MANIFEST.in file:

1	include LICENSE
2	include README.md
3	include requirements.txt

Dependencies

You can specify dependencies both in the install_requires section of setup.py and in a requirements.txt file. Pip will install automatically dependencies from install_requires, but not from the requirements.txt file. To install those requirements, you'll have to specify it explicitly when running pip: pip install -r requirements.txt.

The install_requires option is designed to specify minimal and more abstract requirements at the major version level. The requirements.txt file is for more concrete requirements, often with pinned-down minor versions.

Here is the requirements file of conman. You can see that all the versions are pinned, which means it can be negatively impacted if one of these packages upgrades and introduces a change that breaks conman.

PyYAML==3.11
python-etcd==0.4.3
urllib3==1.7
pyOpenSSL==0.15.1
psutil==4.0.0
six==1.7.3

Pinning gives you predictability and peace of mind. This is especially important if many people install your package at different times. Without pinning, each person will get a different mix of dependency versions based on when they installed it. The downside of pinning is that if you don't keep up with your dependencies development, you may get stuck on an old, poorly performing and even vulnerable version of some dependency.

I originally wrote conman in 2014 and didn't pay much attention to it. Now, for this tutorial I upgraded everything and there were some major improvements across the board for almost every dependency.

Distributions

You can create a source distribution or a binary distribution. I'll cover both.

Source Distribution

You create a source distribution with the command: python setup.py sdist. Here is the output for conman:

> python setup.py sdist
running sdist
running egg_info
writing conman.egg-info/PKG-INFO
writing top-level names to conman.egg-info/top_level.txt
writing dependency_links to conman.egg-info/dependency_links.txt
reading manifest file 'conman.egg-info/SOURCES.txt'
reading manifest template 'MANIFEST.in'
writing manifest file 'conman.egg-info/SOURCES.txt'
warning: sdist: standard file not found: should have one of README, README.rst, README.txt
running check
creating conman-0.3
creating conman-0.3/conman
creating conman-0.3/conman.egg-info
making hard links in conman-0.3...
hard linking LICENSE -> conman-0.3
hard linking MANIFEST.in -> conman-0.3
hard linking README.md -> conman-0.3
hard linking requirements.txt -> conman-0.3
hard linking setup.cfg -> conman-0.3
hard linking setup.py -> conman-0.3
hard linking conman/__init__.py -> conman-0.3/conman
hard linking conman/conman_base.py -> conman-0.3/conman
hard linking conman/conman_etcd.py -> conman-0.3/conman
hard linking conman/conman_file.py -> conman-0.3/conman
hard linking conman.egg-info/PKG-INFO -> conman-0.3/conman.egg-info
hard linking conman.egg-info/SOURCES.txt -> conman-0.3/conman.egg-info
hard linking conman.egg-info/dependency_links.txt -> conman-0.3/conman.egg-info
hard linking conman.egg-info/not-zip-safe -> conman-0.3/conman.egg-info
hard linking conman.egg-info/top_level.txt -> conman-0.3/conman.egg-info
copying setup.cfg -> conman-0.3
Writing conman-0.3/setup.cfg
creating dist
Creating tar archive
removing 'conman-0.3' (and everything under it)

As you can see, I got one warning about missing a README file with one of the standard prefixes because I like Markdown so I have a README.md instead. Other than that, all the package source files were included and the additional files. Then, a bunch of metadata was created in the conman.egg-info directory. Finally, a compressed tar archive called conman-0.3.tar.gz is created and put into a dist sub-directory.

Installing this package will require a build step (even though it's pure Python). You can install it using pip normally, just by passing the path to the package. For example:

pip install dist/conman-0.3.tar.gz
Processing ./dist/conman-0.3.tar.gz
Installing collected packages: conman
  Running setup.py install for conman ... done 

Successfully installed conman-0.3

Conman has been installed into site-packages and can be imported like any other package:

1	import conman
2	conman.__file__
3	'/Users/gigi/.virtualenvs/conman/lib/python2.7/site-packages/conman/__init__.pyc'

Wheels

Wheels are a relatively new way to package Python code and optionally C extensions. They replace the egg format. There are several types of wheels: pure Python wheels, platform wheels, and universal wheels. The pure Python wheels are packages like conman that don't have any C extension code.

The platform wheels do have C extension code. The universal wheels are pure Python wheels that are compatible with both Python 2 and Python 3 with the same code base (they don't require even 2 to 3).

If you have a pure Python package and you want your package to support both Python 2 and Python 3 (becoming more and more important) then you can build a single universal build instead of one wheel for Python 2 and one wheel for Python 3.

Python 3 is the current and actively supported Python version with ongoing updates, improvements, and community support; it is recommended to use Python 3 for all new projects and migrations.

If your package has C extension code, you must build a platform wheel for each platform. The huge benefit of wheels especially for packages with C extensions is that there is no need to have compiler and supporting libraries available on the target machine. The wheel already contains a built package. So you know it will not fail to build, and it is much faster to install because it is literally just a copy. People who use scientific libraries like Numpy and Pandas can really appreciate this, as installing such packages used to take a long time and might have failed if some library was missing or the compiler wasn't configured properly.

The command to build pure or platform wheels is: python setup.py bdist_wheel.

Setuptools—the engine that provides the setup() function—will detect automatically if a pure or platform wheel is needed.

running bdist_wheel
running build
running build_py
creating build
creating build/lib
creating build/lib/conman
copying conman/__init__.py -> build/lib/conman
copying conman/conman_base.py -> build/lib/conman
copying conman/conman_etcd.py -> build/lib/conman
copying conman/conman_file.py -> build/lib/conman
installing to build/bdist.macosx-10.9-x86_64/wheel
running install 

running install_lib
creating build/bdist.macosx-10.9-x86_64
creating build/bdist.macosx-10.9-x86_64/wheel
creating build/bdist.macosx-10.9-x86_64/wheel/conman
copying build/lib/conman/__init__.py -> build/bdist.macosx-10.9-x86_64/wheel/conman
copying build/lib/conman/conman_base.py -> build/bdist.macosx-10.9-x86_64/wheel/conman
copying build/lib/conman/conman_etcd.py -> build/bdist.macosx-10.9-x86_64/wheel/conman
copying build/lib/conman/conman_file.py -> build/bdist.macosx-10.9-x86_64/wheel/conman
running install_egg_info
running egg_info
creating conman.egg-info
writing conman.egg-info/PKG-INFO
writing top-level names to conman.egg-info/top_level.txt
writing dependency_links to conman.egg-info/dependency_links.txt
writing manifest file 'conman.egg-info/SOURCES.txt'
reading manifest file 'conman.egg-info/SOURCES.txt'
reading manifest template 'MANIFEST.in'
writing manifest file 'conman.egg-info/SOURCES.txt'
Copying conman.egg-info to build/bdist.macosx-10.9-x86_64/wheel/conman-0.3-py2.7.egg-info
running install_scripts
creating build/bdist.macosx-10.9-x86_64/wheel/conman-0.3.dist-info/WHEEL

Checking the dist directory, you can see that a pure Python wheel was created:

1	ls -la dist
2	dist/
3	total 32
4	-rw-r--r-- 1 gigi staff 5.5K Feb 29 07:57 conman-0.3-py2-none-any.whl
5	-rw-r--r-- 1 gigi staff 4.4K Feb 28 23:33 conman-0.3.tar.gz

The name conman-0.3-py2-none-any.whl has several components:

package name
package version
Python version
platform version, and finally the "whl" extension.

To build universal packages, you just add --universal, as in python setup.py bdist_wheel --universal.

The resulting wheel is called conman-0.3-py2.py3-none-any.whl.

Note that it is your responsibility to ensure your code actually works under both Python 2 and Python 3 if you create a universal package.

Using Toml

Toml (Tom's Obvious Minimal Language) is an easy-to-use configuration file format for configuring software applications. Even though setup.py is still widely used and supported, PEP518 recommends using a pyproject.toml file instead of setup.py for packaging and distribution.

Let's use pyproject.toml to create a simple Python package called apex that checks the battery status and displays a message when the battery is full.

Create a directory structure called apex. Inside apex, add a pyproject.toml and a README.md file.

1	.apex
2	├── pyproject.toml
3	└── README.md

Inside the root apex directory, create a subdirectory called apex with an __init__.py file to make it a package and a battery.py file. Your project directory should now look like this:

.
├── apex
│   ├── battery.py
│   └── __init__.py
├── pyproject.toml
└── README.md

Install the psutil package with pip.

1	pip install psutil

Psutil (processes and system utilities) is a cross-platform library for process and system monitoring in Python. Psutil can retrieve information about processes running on the system, such as status, CPU usage, and memory usage. It can also manipulate system processes.

In the battery.py file, add the code that checks if the battery is full.

import psutil

def check_battery():
    battery = psutil.sensors_battery()
    plugged = battery.power_plugged
    percent = battery.percent

    if percent == 100 and plugged:
        print "Battery is full. Unplug your Charger"
    else:
        print "Battery is not yet full."

if __name__ == "__main__":
    message = check_battery()
    print(message)

The pyproject.toml file contains the metadata for the package, including:

build-system
name
version
project-dependencies.

Open pyproject.toml and specify setuptools as the build system.

1	[build-system]
2	requires = ["setuptools","wheel"]
3	build-backend ="setuptools.build_meta"

Specify the name of the packages, the version number, and the list of project dependencies. Now the pyproject.toml file looks like this:

[build-system]
requires = ["setuptools","wheel"]
build-backend ="setuptools.build_meta"

[project]
name="apex"
version ="1.0.0"

[project-dependencies]
psutil ="5.9.5"

Finally, in the README.md file, add the following contents.

# apex

**apex** is a Python package that allows you to check the battery status of your laptop and display a message when the battery is full.

## Features

- Retrieve battery status information
- Display a message when the battery is full
- Cross-platform support (Windows, macOS, Linux, etc.)

## Installation

Install **apex** using `pip`:

```shell
pip install apex

Build the package with the build command.

1	python -m build

You should see the output below, which I have truncated.

creating build/bdist.linux-x86_64/wheel/apex-1.0.0.dist-info/WHEEL
creating '/home/vaati/Desktop/apex/dist/.tmp-d8_lqaaj/apex-1.0.0-py3-none-any.whl' and adding 'build/bdist.linux-x86_64/wheel' to it
adding 'apex/__init__.py'
adding 'apex/battery.py'
adding 'apex-1.0.0.dist-info/METADATA'
adding 'apex-1.0.0.dist-info/WHEEL'
adding 'apex-1.0.0.dist-info/top_level.txt'
adding 'apex-1.0.0.dist-info/RECORD'
removing build/bdist.linux-x86_64/wheel
Successfully built apex-1.0.0.tar.gz and apex-1.0.0-py3-none-any.whl

The build creates a dist folder which contains the distribution files for your package:

apex-1.0.0.tar.gz: the source distribution of the apex package. It contains the source code required to install the package.
apex-1.0.0-py3-none-any.whl: the binary distribution of the apex package. You can use the binary distribution to install it directly from the source.

The build process also generates an apex.egg-info folder containing metadata about the package.

.
├── apex
│   ├── battery.py
│   └── __init__.py
├── apex.egg-info
│   ├── dependency_links.txt
│   ├── PKG-INFO
│   ├── SOURCES.txt
│   └── top_level.txt
├── dist
│   ├── apex-1.0.0-py3-none-any.whl
│   └── apex-1.0.0.tar.gz
├── pyproject.toml
└── README.md

Conclusion

Writing your own Python packages requires dealing with a lot of tools, specifying a lot of metadata, and thinking carefully about your dependencies and target audience. But the reward is great.

If you write useful code and package it properly, people will be able to install it easily and benefit from it.