How to Use Python Packages

Python packages allow you to break down large systems and organize their modules in a consistent way that you and other people can use and reuse efficiently. Python's motto of "Batteries Included" means that it comes preloaded with lots of useful packages in the standard library.

But there are also many amazing third-party packages you can take advantage of. In this tutorial, you'll learn all you need to know about what packages are exactly, how to import modules from packages, exploring the built-in packages in Python's standard library, and installing third-party packages.

What Are Packages?

Before we can talk about packages, let's talk about modules. Modules are the source files with *.py extension where you (and everyone else) put the functions and classes that comprise your program.

A package in Python is simply a folder with multiple Python files and should have an __init__.py file. The __init__.py file indicates that the directory is a package. The __init__.py file can be empty or contain some executable code.

Packages are the manifestation of Python's hierarchical namespaces concept. To quote from the Zen of Python:

"Namespaces are one honking great idea -- let's do more of those!"

To view the whole Zen of Python, type import this in a Python interactive session:

>>> import this
The Zen of Python, by Tim Peters

Beautiful is better than ugly.
Explicit is better than implicit.
Simple is better than complex.
Complex is better than complicated.
Flat is better than nested.
Sparse is better than dense.
Readability counts.
Special cases aren't special enough to break the rules.
Although practicality beats purity.
Errors should never pass silently.
Unless explicitly silenced.
In the face of ambiguity, refuse the temptation to guess.
There should be one-- and preferably only one --obvious way to do it.
Although that way may not be obvious at first unless you're Dutch.
Now is better than never.
Although never is often better than *right* now.
If the implementation is hard to explain, it's a bad idea.
If the implementation is easy to explain, it may be a good idea.
Namespaces are one honking great idea -- let's do more of those!
>>> 

Namespaces help with organizing code and preventing naming conflicts. This is critical when multiple people work together or when using packages developed by other people.

While packages represent a hierarchy of sub-packages and modules, which are files, the hierarchy doesn't have to file-system-based, where packages and sub-packages are directories and sub-directories. It is much more flexible than that.

Create a Python Package

Let's start with a simple example. Below we have a package called simple_package with two Python modules.

simple_package
.
├── __init__.py
├── tasks.py
└── views.py

0 directories, 3 files

__init__.py : indicates that it's a package
tasks.py and views.py are modules

Third-Party Packages

Let's take a look at a package called ansible. It is not a package from the standard library. You'll see later how to find and install third-party packages. Now, let's just check out the directory file structure.

The packages will typically be installed into the Python interpreter's site-packages directory, located somewhere (depending on version, OS, and distribution) under lib.

On the Mac, for example, Python 3.10 will be located in <interpreter root>/lib/python3.10/site-packages. Here is how the ansible package is organized:

tree ansible -L 1
ansible
├── cli
├── collections
├── compat
├── config
├── constants.py
├── context.py
├── errors
├── executor
├── galaxy
├── __init__.py
├── inventory
├── keyword_desc.yml
├── __main__.py
├── modules
├── module_utils
├── parsing
├── playbook
├── plugins
├── __pycache__
├── release.py
├── template
├── utils
├── vars
└── _vendor

18 directories, 6 files

There are 6 modules and 18 directories. Each directory is a sub-package of the main ansible package. Looking inside the ansible/utils directory, we can see it contains additional modules and even one more sub-package:

tree ansible/utils -L 1
ansible/utils
├── cmd_functions.py
├── collection_loader
├── color.py
├── context_objects.py
├── display.py
├── encrypt.py
├── fqcn.py
├── galaxy.py
├── hashing.py
├── helpers.py
├── __init__.py
├── jsonrpc.py
├── _junit_xml.py
├── listify.py
├── lock.py
├── multiprocessing.py
├── native_jinja.py
├── path.py
├── plugin_docs.py
├── py3compat.py
├── sentinel.py
├── shlex.py
├── singleton.py
├── ssh_functions.py
├── unicode.py
├── unsafe_proxy.py
├── vars.py
└── version.py

1 directory, 27 files


The Search Path

When you import a module, Python will go through a search algorithm based on the search path, which is a list of directories to start the search. The search path is a list of directories available through sys.path, and you can manipulate it dynamically (add, remove or move around items in the search path). The site-packages directory is always there.

To import the path.py module from ansible/utils, you'll need to use the following command:

1	import ansible.utils.path

To import both path and encrypt modules, you use the following commands:

1	import ansible.utils.path
2	import ansible.utils.encrypt

If you also want to use the standard os.path module, you'll use the following command:

1	import os.path

Now you can use either or both path modules with no conflicts due to the different namespaces they belong to.

Exploring the Standard Library

The standard library has a lot of packages. It's worth exploring it whenever you need to accomplish a task and you're not sure how. There is a very high likelihood that for any general-purpose task like math, shell integration, OS integration, string manipulation, networking and common file formats, there is a well-designed, well-performing and well-tested package in the standard library.

You can really trust standard library packages because it is a big deal to get into the standard library. Either the package was designed by Python's core developers or it was heavily reviewed and often heavily used in the field as a third-party library before making it into the standard library.

Here are all the packages in the standard library organized by topic.

PyPI

The standard library is awesome, but there'll often be some special functionality you need that is not standard. It doesn't mean you have to write it from scratch. Python has a vibrant and active community that develops and freely shares a lot of code. Enter PyPI: the Python Package Index. PyPI hosts all publicly available packages and provides a one-stop shop for browsing through them.

Browsing PyPI

PyPI organizes the packages in a browsable index. You can browse and search by topic, environment, framework, development, status, intended audience, license, natural language, programming language (yes, there are Python packages that support many programming languages), and operating system.

As of 2021, PyPI does not display download statistics for packages since it's ineffective due to the resources required to maintain the statistics.

Installing Packages

There are two ways to install packages from PyPI. You can download the package and then run python setup.py install. But the modern way is to use pip, setuptools, and wheel.

Pip and setuptools are included by default starting from Python 3.4 and Python 2.79, but you will need to upgrade to the latest version:

Linux/macOS: pip install -U pip setuptools
Windows: python -m pip install -U pip setuptools

Python 2 is, however, no longer supported, so you should already be using Python 3.0 or higher for improved performance.

Use pip to install wheel:

1	pip install wheel.

To install a package with pip, you issue this command.

1	pip install <package_name>

Where package_name is the name of the package. For example, to install Ansible, the command would look like this:

1	pip install ansible

If you need a specific version, you can also specify it as follows:

1	pip install ansible==7.0

Python packages are always installed into an environment. A common practice I will not cover here is to use virtual environments to manage multiple independent installations of Python with different interpreters and/or different sets of installed packages. You can read more about virtual environments here.

Best Practices

The Python packaging authority provides a lot of guidance on the best practices around packaging. This is important because it is an area of active development and recommendations evolve quickly.

Also, if you want to do something special like installing packages from alternative repositories instead of PyPI or using pip in a more sophisticated way, you'll find great discussions and practical advice.

Conclusion

When you're a Python beginner, you learn the core language and have fun playing with it. Pretty soon you discover the standard library, and as you gain more experience you benefit more and more from its richness.

The next stage in your evolution as a Pythonista is to incorporate the vast awesomeness the Python community has put on PyPI into your systems. Packages as the deployment unit of reusable Python code enable this ecosystem.