Using PEX files as release primitives deployments

01-10-2025

Overview:

https://docs.pex-tool.org/index.html
- tldr - https://docs.pex-tool.org/whatispex.html#whatispex
- PEP 441 - https://peps.python.org/pep-0441/
self-contained python virtual environments
- basically zip files with a shebang (#!/usr/bin/env python) and a __main__.py file
- shebang is the #! character sequence at the beginning of a script in a Unix-like operating system that specifies the interpreter to be used for executing the script (the script can therefore be run as an executable)
this builds on top of python zip applications defined in PEP 441
- python has ability to execute directories or ZIP-format archives as scripts since version 2.6
- __main__.py must be in the root directory of the archive, with extension .pyz will be associated with such files (and the python launcher will recognise this)
- the standard library defines a zipapp module for working with these archives
- https://github.com/python/cpython/blob/main/Lib/zipapp.py
can bundle the interpreter too - using scies https://docs.pex-tool.org/scie.html from science and python standalone builds https://github.com/astral-sh/python-build-standalone from astral
PEX files are simple release primitives (build, copy and paste, run anywhere; for example, upload to blob, download, run cli locally)

Case studies:

Usage (build and run):

install all dependencies and transitive dependencies (the wheel files)
- either resolved via PEX (using a requirements.txt file) or building the wheelhouse manually (using pip download)
reference a python interpreter (or embed a standalone python build)
- based on the environment and python version, specify the relevant specifiers for abi + platform + python-version

# set env variables
export SSL_CERT_FILE: "/etc/ssl/certs/ca-certificates.crt"
export UV_INDEX_URL="https://nexus.ny1.ninetyone.com/repository/pypy-all/simple"
export UV_NATIVE_TLS=true
export UV_INSECURE_HOST="nexus.ny1.ninetyone.com"
 
# create temp wheelhouse directory
mkdir -p temp/wheelhouse
 
# build package wheel to wheelhouse
uv build --wheel -o temp/wheelhouse/
 
# build package depdendency wheels to wheelhouse
uv export --format requirements-txt --no-hashes --no-emit-project --output-file temp/requirements.txt
uv run python -m pip download -r temp/requirements.txt \
  --only-binary :all: \
  --platform manylinux_2_17_x86_64 --platform manylinux_2_28_x86_64 --platform manylinux2014_x86_64 \
  --python-version 312 \
  --abi cp312 --abi abi3 \
  --dest temp/wheelhouse \
  --index-url="${PIP_INDEX_URL}" \
  --trusted-host="${PIP_TRUSTED_HOST}"
 
# build PEX file from wheelhouse
uv tool run --python 3.12 pex \
  temp/wheelhouse/*.whl \
  --use-pip-config \
  --no-transitive \
  --interpreter-constraint "CPython==3.12.*" \
  --python-shebang '#!/usr/bin/env python3.12' \
  --script uvicorn \
  --script streamlit \
  --script dagster \
  --output-file ai.pex \
  --sources-dir=.
  # --scie-python-version "3.12.7" \
  # --scie eager \
  # --scie-pbs-stripped \
 
# run PEX file
uv run ai.pex main:app --host 0.0.0.0 --port 6152

Inside a PEX file:

unzip the pex file:

.bootstrap
.deps
__pex__
__main__.py
PEX-INFO

{
  "bootstrap_hash": "069ebec6124b79045e73d503453010e23c852f0d",
  "build_properties": { "pex_version": "2.59.1" },
  "code_hash": "da39a3ee5e6b4b0d3255bfef95601890afd80709",
  "deps_are_wheel_files": false,
  "distributions":
    {
      "Brotli-1.1.0-cp312-cp312-manylinux_2_17_x86_64.manylinux2014_x86_64.whl": "5f2f1915d5065e7e6e7a4351a2ea2557aeb9c2335cf61d74a66c4be670e8bf8c",
      "PyJWT-2.10.1-py3-none-any.whl": "17bcf7fd145ef9dade0c112aa7aab551f1425112c3ca92694a3ff237595ce91f",
      "ai-0.9.1-py3-none-any.whl": "f466759b3f58cdf473b882307e3c8d704762a37fbafbb5a657bb6a7d9b2494e4",
    },
  "emit_warnings": true,
  "entry_point": "uvicorn.main:main",
  "excluded": [],
  "ignore_errors": false,
  "includes_tools": false,
  "inherit_path": "false",
  "inject_args": [],
  "inject_env": {},
  "inject_python_args": [],
  "interpreter_constraints": [],
  "max_install_jobs": 1,
  "overridden": [],
  "pex_hash": "d0ee458b14516defda574819605c914a3d8722c0",
  "pex_path": "",
  "pex_paths": [],
  "requirements": ["ai==0.9.1"],
  "strip_pex_env": true,
  "venv": false,
  "venv_bin_path": "false",
  "venv_copies": false,
  "venv_hermetic_scripts": true,
  "venv_site_packages_copies": false,
  "venv_system_site_packages": false,
}

Cookbook:

long running processes
- https://docs.pex-tool.org/recipes.html#long-running-pex-applications-and-daemons
- use https://github.com/dvarrazzo/py-setproctitle to alter process title when debugging running processes
environment variables
- https://docs.pex-tool.org/api/vars.html