Distributing sdist and Wheel Packages via PyPI#

Cantera has been distributed on PyPI as a source distribution (sdist) and a wheel since v2.6.0 in 2022. Wheels are Python’s binary distribution format, where source code is compiled to platform-specific libraries to eliminate the need for user machines to have compilers and dependencies installed. Source distributions are also uploaded so that platforms where a wheel is not available can still attempt to build Cantera from the source code.

This page describes how to build sdists and wheels for Cantera, and how these are built for distribution via PyPI. Some of the trade-offs in selecting the current build system are documented in the relevant enhancement issue. The source code to build and publish the wheels is in a specialized repository on GitHub.

The overall process is conducted in two steps:

  1. SCons is used to build an sdist from the git checkout with scons sdist

  2. cibuildwheel/build is used to build wheels from the sdist, not the git checkout

There are a few reasons for this two-stage process. First, it ensures that our sdist can be used to build a wheel on users’ systems. Second, it tremendously simplifies the cibuildwheel config, because installation of build dependencies is handled by cibuildwheel at build time.

TLDR#

This set of steps should build an sdist and wheel on a developer’s machine for local testing.

  1. Clone the repository (submodules are optional)

  2. Install SCons and build

  3. Run scons sdist a. scons clean sdist can also be used to automatically remove the build folder prior to building the `sdist.

  4. Building the wheel requires that Boost and libhdf5 are available on the build system. If these are not installed in standard locations, you can set BOOST_ROOT and HDF5_ROOT environment variables to point to the correct location.

  5. Change to the build/python_sdist/dist folder

  6. Unpack the sdist: tar xf cantera-3.1.0a2.tar.gz. Note the version 3.1.0a2 may be different.

  7. Change into the unpacked directory

  8. Build the wheel: python -m build --wheel . Note the trailing . which indicates the current directory. You can also add the -v or -vv flags to increase build verbosity.

The wheel should be located in the build/python_sdist/dist/cantera-3.1.0a2/dist folder. You can inspect the contents using any Zip-file reader. The CLI tool unzip can list the contents with the -l flag. You can install the wheel by running python -m pip install <path to the wheel>.

Source Distributions#

Sdists are built from the git checkout using SCons. Starting with v3.1, the only dependencies required to build the sdist are SCons and build. Dependencies such as NumPy, Boost, or any of the dependencies available as submodules are not necessary to build the sdist.

To build the sdist, run scons sdist from the repository root.

Note

For versions 2.6 and 3.0, all the usual dependencies to build Cantera from source are required. Starting with 3.1, the build process for the sdist was simplified to pull in external dependencies at wheel-build time, rather than copying code from the Cantera external source tree into the sdist.

The code for the sdist is located in the interfaces/python_sdist folder. Building the sdist takes two steps:

  1. SCons runs the build_sdist.py file to copy all the relevant source files into the build/python_sdist folder.

  2. SCons runs build to turn the source files into a compliant sdist tarball using the scikit-build-core package as the build backend.

Both of these steps are done at “configure” time in SCons, when SCons is first parsing the SConstruct and SConscript files. If the build is successful, SCons will exit before performing any configuration or platform checks. This is why the build is done at configure time, to avoid these time-consuming checks that aren’t necessary since the sdist just copies files into the build folder.

If successful, the .tar.gz file will be located in the build/python_sdist/dist folder.

Sdist configuration#

The configuration for the sdist is done in the pyproject.toml.in file. This is a template file that is filled in at build time by the build_sdist.py script. It needs to be filled in by SCons because that’s where we define the version of Cantera, in the SConstruct file.

You can read more about the standard fields in the pyproject.toml file in the PyPA documentation. You can read more about the scikit-build fields in the scikit-build-core documentation.

Wheels#

As mentioned above, wheels (.whl files) are Python’s standard binary distribution artifact. Similar to Conda packages, wheels include compiled libraries specific to a particular platform, so that end users don’t need to have compilers installed to use a package.

Building Cantera’s wheels is done using build, similar to the sdist. However, the wheel is built from the unpacked sdist tarball, and you must pass the --wheel flag to build. Then, build will install scikit-build-core to manage building the wheel from Cantera’s C++ and Cython source files.

scikit-build-core uses CMake to manage the compiling and linking process for the wheel. Most of the configuration for the wheel build is therefore done in the CMakeLists.txt files in the interfaces/python_sdist folder. These are copied into the build/python_sdist folder and from there into the actual sdist tarball.

Note

The minimum required version of CMake is 3.24 to support all the options we use with FetchContent. A compatible version should be installed by scikit-build-core if it isn’t available on the system.

There are essentially 4 stages to the wheel build:

  1. External dependencies (eigen, fmt, SUNDIALS, HighFive, yaml-cpp) are either located on the system, or downloaded and compiled using the FetchContent mechanism

  2. Cantera’s C++ source code is built and linked to the appropriate dependencies

  3. Cython runs and generates C++ source code for the Python interface

  4. The generated C++ is compiled, linked to libcantera, and packaged into the .whl file

Note

Cantera’s other dependencies, specifically Boost and libhdf5, are neither installed nor built during the wheel build and must be pre-installed on the build system. I couldn’t find a way to have eigen and HighFive, respectively, find those dependencies if they were also installed using FetchContent. If Boost and libhdf5 are installed in non-standard directories, you can use the Boost_ROOT or Boost_INCLUDE_DIRS and HDF5_ROOT variables to specific the correct locations. See the CMake documentation for find_package().

Steps 1 and 2 above are set up in the interfaces/python_sdist/src/CMakeLists.txt file. Steps 3 and 4 are set up in the interfaces/python_sdist/cantera/CMakeLists.txt file.

Publishing the sdist and Wheels#

When a version of Cantera is ready to be published to PyPI, the workflow in the pypi-packages repository must be executed. The workflow essentially conducts the steps in the TLDR section above.

The one big addition to the GitHub Action workflow is that there are a couple of shell scripts in the pypi-packages repository to build libhdf5 for macOS and Windows. On macOS, we also link to libaec so there’s a small patch for the CMakeLists.txt file in libaec.

For Linux builds, we have a custom-built manylinux image that already has the dependencies installed. This saves time compiling libhdf5 on Linux/ARM builds because those are emulated and already very slow.

As much of the configuration for cibuildwheel as possible is done statically in pyproject.toml.in. Only values that are dynamically determined when the workflow runs should be set in the workflow.

On this page
Show Source