Revisiting Python Packaging With Pipenv

Overview

Python is one of the friendliest yet most powerful languages out there. It is easy for beginners to pick up, yet packs a strong punch and is used extensively in diverse domains such as scientific programming, web application programming, and DevOps. But one of the weakest points of Python has been its support for packaging complex applications and their dependencies.

Over the years, there have been many efforts to improve the situation. In August 2017, I wrote a tutorial on the state of the art in Python packaging: How to Write, Package and Distribute a Library in Python.

It's been only four months, and there is a new player in town. Pipenv is now the officially recommended tool for packaging by PyPA (Python Packaging Authority). In this tutorial you'll learn why Pipenv significantly improves the state of packaging and overall development workflow for Python developers and how to use it effectively.

Python Dev Workflow for Humans

The goal of Pipenv is to improve the development workflow of Python developers when it comes to managing dependencies and virtual environments. It is another fine library from the industrious Kenneth Reitz, who is known mostly for the requests package (HTTP for humans), but wrote a few other excellent packages.

Do We Need Yet Another Packaging Tool?

Yes, we do! Pipenv takes a page from modern package management practices and imports them into the Python world.

Installing Pipenv

You can install Pipenv with pip install pipenv. You'll get a nice output with emojis:

1	$ pip install pipenv
2	✨🍰✨

You'll have to do it just once. If you don't have pip installed, you can use this bootstrap command: $ curl https://github.com/pypa/pipenv/blob/master/get-pipenv.py | python

Pipfile and Pipfile.lock

Pipenv can create an empty virtual environment for you. Here is a quick demo:

~/git > mkdir testpipenv
~/git > cd testpipenv
~/git/testpipenv > pipenv --three

Output:

Creating a virtualenv for this project…
Using /usr/local/bin/python3 to create virtualenv…
⠋Running virtualenv with interpreter /usr/local/bin/python3
Using base prefix '/usr/local/Cellar/python3/3.6.3/Frameworks/Python.framework/Versions/3.6'
New python executable in /Users/gigi.sayfan/.local/share/virtualenvs/testpipenv-0GShD6dy/bin/python3.6
Also creating executable in /Users/gigi.sayfan/.local/share/virtualenvs/testpipenv-0GShD6dy/bin/python
Installing setuptools, pip, wheel...done.

Virtualenv location: /Users/gigi.sayfan/.local/share/virtualenvs/testpipenv-0GShD6dy
Creating a Pipfile for this project…

This will create an empty Pipfile with no dependencies. But since you'll probably want to install some packages for your project, you can just use pipenv to install a package and it will create the virtual environment automatically. For example:

~/git/testpipenv > pipenv install requests

Output:


Installing requests…
Collecting requests
  Using cached requests-2.18.4-py2.py3-none-any.whl
Collecting idna<2.7,>=2.5 (from requests)
  Using cached idna-2.6-py2.py3-none-any.whl
Collecting chardet<3.1.0,>=3.0.2 (from requests)
  Using cached chardet-3.0.4-py2.py3-none-any.whl
Collecting certifi>=2017.4.17 (from requests)
  Using cached certifi-2017.11.5-py2.py3-none-any.whl
Collecting urllib3<1.23,>=1.21.1 (from requests)
  Using cached urllib3-1.22-py2.py3-none-any.whl
Installing collected packages: idna, chardet, certifi, urllib3, requests
Successfully installed certifi-2017.11.5 chardet-3.0.4 idna-2.6 requests-2.18.4 urllib3-1.22

Adding requests to Pipfile's [packages]…
  PS: You have excellent taste! ✨ 🍰 ✨
Locking [dev-packages] dependencies…
Locking [packages] dependencies…
Updated Pipfile.lock (7b8df8)!

The level of detail is excellent, and it uses nice colors too. Here is the resulting Pipfile:

[[source]]

url = "https://pypi.python.org/simple"
verify_ssl = true
name = "pypi"


[dev-packages]



[packages]

requests = "*"


[requires]

python_version = "3.6"

The Pipfile keeps track of your project's top-level dependencies—here, just requests = "*". It uses TOML as its format, which is a popular choice these days for configuration files (Rust's Cargo, Python's PEP-518).

The Pipefile.lock file, on the other hand, is a JSON file that specifies some metadata and the exact versions (including hashes) of all the recursive dependencies (top-level dependencies and their dependencies). Here is the Pipfile.lock file:

{
    "_meta": {
        "hash": {
            "sha256": "33a0ec7c8e3bae6f62dd618f847de92ece20e2bd4efb496927e2524b9c7b8df8"
        },
        "host-environment-markers": {
            "implementation_name": "cpython",
            "implementation_version": "3.6.3",
            "os_name": "posix",
            "platform_machine": "x86_64",
            "platform_python_implementation": "CPython",
            "platform_release": "16.7.0",
            "platform_system": "Darwin",
            "platform_version": "Darwin Kernel Version 16.7.0: Wed Oct  4 00:17:00 PDT 2017; root:xnu-3789.71.6~1/RELEASE_X86_64",
            "python_full_version": "3.6.3",
            "python_version": "3.6",
            "sys_platform": "darwin"
        },
        "pipfile-spec": 6,
        "requires": {
            "python_version": "3.6"
        },
        "sources": [
            {
                "name": "pypi",
                "url": "https://pypi.python.org/simple",
                "verify_ssl": true
            }
        ]
    },
    "default": {
        "certifi": {
            "hashes": [
                "sha256:244be0d93b71e93fc0a0a479862051414d0e00e16435707e5bf5000f92e04694",
                "sha256:5ec74291ca1136b40f0379e1128ff80e866597e4e2c1e755739a913bbc3613c0"
            ],
            "version": "==2017.11.5"
        },
        "chardet": {
            "hashes": [
                "sha256:fc323ffcaeaed0e0a02bf4d117757b98aed530d9ed4531e3e15460124c106691",
                "sha256:84ab92ed1c4d4f16916e05906b6b75a6c0fb5db821cc65e70cbd64a3e2a5eaae"
            ],
            "version": "==3.0.4"
        },
        "idna": {
            "hashes": [
                "sha256:8c7309c718f94b3a625cb648ace320157ad16ff131ae0af362c9f21b80ef6ec4",
                "sha256:2c6a5de3089009e3da7c5dde64a141dbc8551d5b7f6cf4ed7c2568d0cc520a8f"
            ],
            "version": "==2.6"
        },
        "requests": {
            "hashes": [
                "sha256:6a1b267aa90cac58ac3a765d067950e7dbbf75b1da07e895d1f594193a40a38b",
                "sha256:9c443e7324ba5b85070c4a818ade28bfabedf16ea10206da1132edaa6dda237e"
            ],
            "version": "==2.18.4"
        },
        "urllib3": {
            "hashes": [
                "sha256:06330f386d6e4b195fbfc736b297f58c5a892e4440e54d294d7004e3a9bbea1b",
                "sha256:cc44da8e1145637334317feebd728bd869a35285b93cbb4cca2577da7e62db4f"
            ],
            "version": "==1.22"
        }
    },
    "develop": {}

If you want to see a graph of all your dependencies, type: pipenv graph

~/git/testpipenv > pipenv graph
requests==2.18.4
  - certifi [required: >=2017.4.17, installed: 2017.11.5]
  - chardet [required: >=3.0.2,<3.1.0, installed: 3.0.4]
  - idna [required: <2.7,>=2.5, installed: 2.6]
  - urllib3 [required: >=1.21.1,<1.23, installed: 1.22]

Using Installed Packages With Pipenv

Once you've installed a package with Pipenv, it is accessible in your virtual environment just like a standard package (the same as if you pip installed it). The only precaution is that you must use your virtual environment interpreter. Pipenv provides two helpful commands: run and shell.

You use pipenv run python <your program>.py to run your program, and you use pipenv shell to start a new shell with your virtual environment Python interpreter. Here is how to use the shell command to start an interactive Python session that uses the installed requests package to get a quote of the day from a REST API. The virtual environment is activated, and launching Python uses the right interpreter where requests is available.

~/git/testpipenv > pipenv shell
Spawning environment shell (/bin/bash). Use 'exit' to leave.
source /Users/gigi.sayfan/.local/share/virtualenvs/testpipenv-0GShD6dy/bin/activate
~/git/testpipenv > source /Users/gigi.sayfan/.local/share/virtualenvs/testpipenv-0GShD6dy/bin/activate
(testpipenv-0GShD6dy) ~/git/testpipenv > python
Python 3.6.3 (default, Nov 19 2017, 16:39:12)
[GCC 4.2.1 Compatible Apple LLVM 9.0.0 (clang-900.0.38)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import requests
>>> r = requests.get('https://quotes.rest/qod')
>>> r.ok
True
>>> r.json()
{'success': {'total': 1}, 'contents': {'quotes': [{'quote': 'Hang Out with People Who are Better than You.', 'author': 'Warren Buffett', 'length': None, 'tags': ['getting-better', 'inspire', 'people'], 'category': 'inspire', 'title': 'Inspiring Quote of the day', 'date': '2018-01-03', 'id': None}], 'copyright': '2017-19 theysaidso.com'}}
>>> quote = r.json()['contents']['quotes'][0]['quote']
>>> author = r.json()['contents']['quotes'][0]['author']
>>> print(f'{quote} ~~ {author}')
Hang Out with People Who are Better than You. ~~ Warren Buffett
>>>

Importing From requirements.txt

If you want to migrate an existing project with a requirements.txt, Pipenv has got you covered. Simply: pipenv install -r <path/to/requirements.txt>.

All your dependencies will be imported into the Pipfile. To actually install the dependencies and generate the Pipfile.lock, you need to pipenv install. Once you've verified that everything works as expected, you can delete your requirements.txt file.

If your requirements.txt exists in the same directory that you create the virtual environment then Pipenv will automatically generate the Pipfile. But beware that if your requirements.txt file contained pinned versions then they will be pinned in the Pipfile too. In the Pipenv world, pinning should happen in the Pipfile.lock file. Pipenv will give a friendly reminder. See below:

~/git/testpipenv > cat requirements.txt
requests==2.18.4
~/git/testpipenv > pipenv --three
Creating a virtualenv for this project…
Using /usr/local/bin/python3 to create virtualenv…
⠋Running virtualenv with interpreter /usr/local/bin/python3
Using base prefix '/usr/local/Cellar/python3/3.6.3/Frameworks/Python.framework/Versions/3.6'
New python executable in /Users/gigi.sayfan/.local/share/virtualenvs/testpipenv-0GShD6dy/bin/python3.6
Also creating executable in /Users/gigi.sayfan/.local/share/virtualenvs/testpipenv-0GShD6dy/bin/python
Installing setuptools, pip, wheel...done.

Virtualenv location: /Users/gigi.sayfan/.local/share/virtualenvs/testpipenv-0GShD6dy
Requirements.txt found, instead of Pipfile! Converting…
Warning: Your Pipfile now contains pinned versions, if your requirements.txt did.
We recommend updating your Pipfile to specify the "*" version, instead.

Here is the pinned version in the Pipfile that is recommended to change to "*":

1	[packages]
2
3	requests = "==2.18.4"

Let's install the dependencies now:

~/git/testpipenv > pipenv install
Pipfile.lock not found, creating…
Locking [dev-packages] dependencies…
Locking [packages] dependencies…
Updated Pipfile.lock (0b0daf)!
Installing dependencies from Pipfile.lock (0b0daf)…
  🐍   ▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉ 5/5 — 00:00:01
To activate this project's virtualenv, run the following:
 $ pipenv shell
~/git/testpipenv >

Editable Dependencies

You can tell Pipenv to install a path as editable. This is useful when you depend on packages you're developing and want to depend on your source package without actually installing them every time you make a change. In particular, it's useful for the current directory when you're actively working in it. To do that, use the -e and --dev flags:

1	> pipenv install '-e .' --dev

You need to have a proper setup.py file.

Managing Your Environment With Pipenv

You've already seen a lot of what Pipenv can do for you. Let's dig deeper into some additional commands and options.

Installing Packages

The pipenv install command supports several options:

--dev: Install both develop and default packages from Pipfile.lock.
--system: Use the system pip command rather than the one from your virtualenv.
--ignore-pipfile: Ignore the Pipfile and install from the Pipfile.lock.
--skip-lock: Ignore the Pipfile.lock and install from the Pipfile. In addition, do not write out a Pipfile.lock reflecting changes to the Pipfile.

Depending on your workflow and preferences, you may want to use one or more of these options at different times.

Uninstalling Packages

To uninstall a dependency, type: pipenv uninstall <package name>. For example:

~/git/testpipenv > pipenv uninstall requests
Un-installing requests…
Uninstalling requests-2.18.4:
  Successfully uninstalled requests-2.18.4

Removing requests from Pipfile…
Locking [dev-packages] dependencies…
Locking [packages] dependencies…
Updated Pipfile.lock (625834)!

Note that I didn't have to specify "requests==2.8.14" when uninstalling, even though it was pinned in the Pipfile.

Locking Dependencies

If you want to generate a snapshot of your current dependencies (e.g. before a release), use the lock command. This is the key to deterministic and repeatable builds: pipenv lock --pre.

Removing the Virtual Environment

Pipenv is awesome, but you may clean up some of your virtual environments from time to time. It's as simple as pipenv --rm.

Security

Pipfile.lock takes advantage of some great new security improvements in pip. By default, the Pipfile.lock will be generated with the sha256 hashes of each downloaded package. This will allow pip to guarantee you’re installing what you intend to when on a compromised network or downloading dependencies from an untrusted PyPI endpoint.

In addition, Pipenv provides the check command, which checks for compliance with PEP 508 -- Dependency specification for Python Software Packages as well as package safety:

~/git/testpipenv > pipenv check .
Checking PEP 508 requirements…
Passed!
Checking installed package safety…
All good!

Conclusion

Pipenv finally brings Python packaging to the forefront of modern software development. It takes inspiration from other successful dependency management systems like Rust's Cargo and Javascript's Yarn.

It marries virtual environments and package management and provides a superior experience with beautiful and colorful informational messages, and implicit best practices! I highly recommend that you start using Pipenv to manage your Python projects.

Additionally, don’t hesitate to see what we have available for sale and for study in the Envato Market, and don't hesitate to ask any questions and provide your valuable feedback using the feed below.