shiv 🔪 — shiv documentation (2023)

Shiv is a command line utility for building fully self-contained Python zipapps as outlined in PEP 441but with all their dependencies included!

Shiv’s primary goal is making distributing Python applications fast & easy.

How it works

Internally, shiv includes two major components: a builder and a small bootstrap runtime.

Building

In order to build self-contained, single-artifact executables, shiv leverages pip to stage your project’s dependenciesand then uses the features described in PEP 441 to create a “zipapp”.

The primary feature of PEP 441 that shiv uses is Python’s ability to implicitly execute a __main__.py file inside of a zip archive.Here’s an example of the feature in action:

$ echo "print('hello world')" >> __main__.py$ zip archive.zip __main__.pyadding: __main__.py (stored 0%)$ python3 ./archive.ziphello world

shiv expands on this functionality by packing your dependencies into the same zip and adding a specialized __main__.py that instructs the Python interpreter tounpack those dependencies to a known location. Then, at runtime, adds those dependencies to your interpreter’s search path, and that’s it!

Note

“Conventional” zipapps don’t include any dependencies, which is what sets shiv apart from the stdlib zipapp module.

shiv accepts only a few command line parameters of its own, described here, and under the covers, any unprocessed parameters aredelegated to pip install. This allows users to fully leverage all the functionality that pip provides.

For example, if you wanted to create an executable for flake8, you’d specify the requireddependencies (in this case, simply flake8), the callable (either via -e for a setuptools-style entrypoint or -c for a bare console_script name), and the output file:

$ shiv -c flake8 -o ~/bin/flake8 flake8

Let’s break this command down,

  • shiv is the command itself.

  • -c flake8 specifies the console_script for flake8 (defined here)

  • -o ~/bin/flake8 specifies the outfile

  • flake8 is a dependency (this portion of the command is delegated to pip install)

This creates an executable (~/bin/flake8) containing all the dependencies specified (flake8)that, when invoked, executes the provided console_script (flake8)!

If you were to omit the entry point/console script flag, invoking the resulting executable would drop you into an interpreter thatis bootstrapped with the dependencies you’ve specified. This can be useful for creating a single-artifact executablePython environment:

$ shiv httpx -o httpx.pyz --quiet$ ./httpx.pyzPython 3.7.7 (default, Mar 10 2020, 16:11:21)[Clang 11.0.0 (clang-1100.0.33.12)] on darwinType "help", "copyright", "credits" or "license" for more information.(InteractiveConsole)>>> import httpx>>> httpx.get("https://shiv.readthedocs.io")<Response [200 OK]>

This is particularly useful for running scripts without needing to create a virtual environment or contaminate your Pythonenvironment, since the pyz files can be used as a shebang!

$ cat << EOF > tryme.py> #!/usr/bin/env httpx.pyz>> import httpx> url = "https://shiv.readthedocs.io"> response = httpx.get(url)> print(f"Got {response.status_code} from {url}!")>> EOF$ chmod +x tryme.py$ ./tryme.pyGot 200 from https://shiv.readthedocs.io!

Bootstrapping

As mentioned above, when you run an executable created with shiv, a special bootstrap function is called.This function unpacks the dependencies into a uniquely named subdirectory of ~/.shiv and then runs your entry point(or interactive interpreter) with those dependencies added to your interpreter’s search path (sys.path).

To improve performance, once the dependencies have been extracted to disk, any further invocations will re-use the ‘cached’site-packages unless they are deleted or moved.

Note

Dependencies are extracted (rather than loaded into memory from the zipapp itself) for two reasons.

1.) Because of limitations of third-party and binary dependencies.

Just as an example, shared objects loaded via the dlopen syscall require a regular filesystem.In addition, many libraries also expect a filesystem in order to do things like building paths via __file__ (which doesn’t work when a module is imported from a zip), etc.To learn more, check out this resource about the setuptools “zip_safe” flag.

2.) Performance reasons

Decompressing files takes time, and if we loaded the dependencies from the zip file every time it would significantly slow down invocation speed.

Preamble

As an application packager, you may want to run some sanity checks or clean up tasks when users execute a pyz.For such a use case, shiv provides a --preamble flag.Any executable script passed to that flag will be packed into the zipapp and invoked during bootstrapping (after extracting dependencies but before invoking an entry point / console script).

If the preamble file is written in Python (e.g. ends in .py) then shiv will inject three variables into the runtime that may be useful to preamble authors:

  • archive: (a string) path to the current PYZ file

  • env: an instance of the Environment object.

  • site_packages: a pathlib.Path of the directory where the current PYZ’s site_packages were extracted to during bootstrap.

For an example, a preamble file that cleans up prior extracted ~/.shiv directories might look like:

#!/usr/bin/env python3import shutilfrom pathlib import Path# These variables are injected by shiv.bootstrapsite_packages: Pathenv: "shiv.bootstrap.environment.Environment"# Get a handle of the current PYZ's site_packages directorycurrent = site_packages.parent# The parent directory of the site_packages directory is our shiv cachecache_path = current.parentname, build_id = current.name.split('_')if __name__ == "__main__": for path in cache_path.iterdir(): if path.name.startswith(f"{name}_") and not path.name.endswith(build_id): shutil.rmtree(path)

Hello World

Here’s an example of how to set up a hello-world executable using shiv.

First, create a new project:

$ mkdir hello-world$ cd hello-world

Add some code.

hello.py

 def main(): print("Hello world") if __name__ == "__main__": main()

Second, create a Python package using your preferred workflow (for this example, I’ll simply create a minimal setup.py file).

setup.py

 from setuptools import setup setup( name="hello-world", version="0.0.1", description="Greet the world.", py_modules=["hello"], entry_points={ "console_scripts": ["hello=hello:main"], }, )

That’s it! We now have a proper Python package, so we can use shiv to create a single-artifact executable for it.

$ shiv -c hello -o hello .

Note

Notice the . at the end of the shiv invocation. That is referring to the local package that we just created.You can think of it as analogous to running pip install .

That’s it! Our example should now execute as expected.

$ ./helloHello world

Influencing Runtime

Whenever you are creating a zipapp with shiv, you can specify a few flags that influence the runtime.For example, the -c/--console-script and -e/--entry-point options already mentioned in this doc.To see the full list of command line options, see this page.

In addition to options that are settable during zipapp creation, there are a number of environment variablesyou can specify to influence a zipapp created with shiv at run time.

SHIV_ROOT

This should be populated with a full path, it overrides ~/.shiv as the default base dir for shiv’s extraction cache.

This is useful if you want to collect the contents of a zipapp to inspect them, or if you want to make a quick edit toa source file, but don’t want to taint the extraction cache.

SHIV_INTERPRETER

This is a boolean that bypasses and console_script or entry point baked into your pyz. Useful fordropping into an interactive session in the environment of a built cli utility.

SHIV_ENTRY_POINT / SHIV_MODULE

Note

Same functionality as -e/--entry-point at build time

This should be populated with a setuptools-style callable, e.g. “module.main:main”. This willexecute the pyz with whatever callable entry point you supply. Useful for sharing a single pyzacross many callable ‘scripts’.

SHIV_CONSOLE_SCRIPT

Note

Same functionality as ``-c/–console-script` at build time

Similar to the SHIV_ENTRY_POINT and SHIV_MODULE environment variables, SHIV_CONSOLE_SCRIPT overrides any valueprovided at build time.

SHIV_FORCE_EXTRACT

This forces re-extraction of dependencies even if they’ve already been extracted. If you makehotfixes/modifications to the ‘cached’ dependencies, this will overwrite them.

SHIV_EXTEND_PYTHONPATH

Note

Same functionality as -E/--extend-pythonpath at build time.

This is a boolean that adds the modules bundled into the zipapp into the PYTHONPATH environmentvariable. It is not needed for most applications, but if an application calls Python as asubprocess, expecting to be able to import the modules bundled in the zipapp, this will allow itto do so successfully.

Reproducibility

shiv supports the ability to create reproducible artifacts. By using the --reproducible command line option orby setting the SOURCE_DATE_EPOCH environment variable during zipapp creation. When this option is selected, if theinputs do not change, the output should be idempotent.

For more information, see https://reproducible-builds.org/.

Top Articles
Latest Posts
Article information

Author: Amb. Frankie Simonis

Last Updated: 01/12/2023

Views: 5830

Rating: 4.6 / 5 (76 voted)

Reviews: 83% of readers found this page helpful

Author information

Name: Amb. Frankie Simonis

Birthday: 1998-02-19

Address: 64841 Delmar Isle, North Wiley, OR 74073

Phone: +17844167847676

Job: Forward IT Agent

Hobby: LARPing, Kitesurfing, Sewing, Digital arts, Sand art, Gardening, Dance

Introduction: My name is Amb. Frankie Simonis, I am a hilarious, enchanting, energetic, cooperative, innocent, cute, joyous person who loves writing and wants to share my knowledge and understanding with you.