Python Setup Tools

This post is focused on packaging distribution modules using "setuptools" and publish them onto PyPI. To better understand these concepts we will clarify the concept of module and, since most of the people use the term "package" in place of either "import package" and "distribution package", we will also clarify the  term "package" too to avoid confusion. In addition to that, we will highlight the differences, pros and cons of source, binary and wheel distribution packages. All of this taking care of "styling" things so that they can easily be used within a Continuous Integration environment.

This is the second of a set of three posts aimed at showing how to create a full featured Python3 project: so the requisite to go on reading it is having read the Python Full Featured Project post first.

The third and last post Packaging a Python Wheel as RPM is about how to pack this project into two RPM packages:

• the first installs the modules provided by the Python package
• the second installs the example script that implements a sample application that imports these modules. This last package also performs post installation tasks so as to reconfigure rsyslog

In addition to that, in the third and last post we  also see how to digitally sign these RPM packages, and in general how to set up an RPM development environment, with a good overview on how to exploit the RPM building tools.

If you have not read it yet, please start from that post, since you need to have the project with all of the code that is necessary to start with.

Modules

The very first thing is having a clear understanding of terms such as "module", "package", import and so on: it is very easy to mix-up things, since very often people uses the term package and module in an interchangeable way, and the term package is used both as the name of a container of modules as well as the cabinet used to package it.

Python Modules

Modules provide a handy way to split the code into more files, logically grouping related objects and functions within a namespace: the namespace has the same name of the file of the module itself - you can easily get the name of a module by accessing its __name__attribute.

This means that Python modules are nothing but  files containing Python code the main program can import if needed: this of course promotes maintainability and code re-usability.

Honestly, even the main file is a special module called "__main__": this means that it's code is within the "__main__" namespace.

We can easily see this by launching Python interactively:

python3

and print the value of the "__name__" attribute:

>>> print(__name__)
__main__

As an example let's create "foomodule.py" file with the following contents:

def cheer(name):
    print(f"Hello {name}")

def _say_goodbye(name):
    print(f"Goodbye {name}")

we can easily test it launching Python interactively:

python3

load the module and print its name:

>>> import foomodule
>>> print(foomodule.__name__)
foomodule

as expected the name is "foomodule"; let's list all of the objects and attributes defined within it by using the "dir()" function, (it is a function that enumerates the contents of the namespace - so the name of the module - provided as argument):

>>> dir(foomodule)
['__builtins__', '__cached__', '__doc__', '__file__', '__loader__', '__name__', '__package__', '__spec__', '_say_goodbye', 'cheer']

as you see besides several builtin objects, the module has also the "_say_goodbye(name)" and "cheer(name)" functions.

Here is an example application that loads the module and runs the cheer() function:

#!/usr/bin/env python3
import foomodule

foomodule.cheer("Marco Carcano")

if used within the main program, the above statement imports all of the objects defined within the "foomodule.py" file within "foomodule" namespace: we need to prepend "foomodule." so to be able to execute the "cheer(name)" function indeed.

We can even import all or a subset of the objects from the module directly within the "__main__" namespace:

#!/usr/bin/env python3
from foomodule import *

cheer("Marco")

as you see this time we can directly access the cheer(name) function without prepending "foomodule." namespace: let's have a look to the list of objects of the __main__ module: launch Python

python3

then load the module and list its contents:

>>> from foomodule import *
>>> dir()
['__annotations__', '__builtins__', '__doc__', '__loader__', '__name__', '__package__', '__spec__', 'cheer']

cheer is actually listed, but this time _say_goodbye() function is not among the list.

this means that so to be able to use the "_say_goodbye(name)" function, we must explicitly import it as follows:

#!/usr/bin/env python3
from foomodule import *
from foomodule import _say_goodbye

cheer("Marco Carcano")
_say_goodbye("Marco Carcano")

Modules Search Path

When an import statement is found, the interpreter first searches for a built-in module with that name of the module to import: if this is not found, the interpreter searches for a ".py" file with the same name of the module into a list of paths specified by sys.path variable.

sys.path defaults to these locations:

• the directory beneath the input script is stored into (or the current directory when no file is specified)
• PYTHONPATH (a list of directory names, with the same syntax as the shell variable PATH)
• the installation-dependent default (includes a site-packages directory, handled by the site module)

Note that the directory containing the script being run takes precedence over the standard library path: this is a handy way to perform overrides.

Extension Modules

Extension Modules are modules written in the low-level language of the Python implementation, so C/C++ for Python, Java for Jython. Typically they are contained into a single dynamically loadable pre-compiled object:

• shared object (".so") on Unix
• DLL on Windows - note that these DLL have the ".pyd" extension
• Java class file for Jython extensions.

Python Packages

Import Packages

Import packages are a way of structuring Python’s module namespace by using "dotted module names": this is a convenient way that avoids collisions of names caused by modules with the same name. Modules are grouped within a parent object - the package - the outcome is a "dotted" namespace with the package name as the first token and the module name as the second one. For example, the module name "foo.bar" designates a module named "bar" in a package named "foo" - the outcome is that the namespace of the objects provided by the packages is "foo.bar".

Python considers a directory as a package if it does contain an  "__init__.py" file: this file often contains the initialization code of the package, but it can also just be an empty file.

It can be used to set the "__all__" variable: as we previously saw,  when using "from package import *" Python does not import objects with a name starting with underscores. It is however possible to provide Python the list of objects that we want to be loaded when using such a statement: it is as simple as listing the objects into the __all__ list, for example:

__all__ = ["FoolistItem", "Foolist","_an_hidden_object"]

Distribution Packages

Distribution Packages are versioned archive files that contain import packages, modules, and other resource files that are used to distribute a Release.

Installing Distribution Packages using PIP

PIP is the official Python tool to download and install Python distribution packages. It downloads them from the online repository PyPI - we'll see more on this topic later on.

Installing a distribution package with pip is as easy as typing "pip install" followed by the name of the package you want to install.

For example:

pip install foopkg

when you have to deal with a lot of packages, you can list them into a requirements file like the following one:

###### Requirements without Version Specifiers ######
PyYAML

###### Requirements with Version Specifiers ######
jupyter == 1.0.0
nose >= 1.0
ansible >=2.8, <2.9

and install everything as a whole as follows:

sudo python3 -m pip install -r requirements.txt

as you see the syntax of the requirements file is really easy and recall the Python's syntax :

• # marks the beginning of a comment
• conditionals:
  • == equal
  • != different
  • < lower
  • > greater
  • <= lower or equal
  • >= greater or equal
  • ~= compatible release - for example ~= 1.1 is the same as >=1.1 or == 1.*
• two conditions can be specified by separating them with a comma (,)

Creating Distribution Packages using Setuptools

First and foremost let's get back to our Python3 project we created in the previous post - change directory to the root of the Carcano's foolist project:

cd ~/fooproject

Distribution packages are created using "setuptools", so the first thing to do is ensuring to have it installed an as up to date as possible:

sudo python3 -m pip install -U setuptools

Setuptools relies on "setup.py" settings file: we already hit this file on the first post of this trilogy, where we used it only to have setuptools instantiate unit tests.

Now, a clean and tidy approach is creating a Makefile so that we can handle the whole project using make - yes, there's still ancient people like me using them out there.

Create the "Makefile" file into "~/fooproject" directory with the following contents:

all: unittests

clean:
	$(info -> Makefile: cleanup previous builds ... )
	@(rm -rf src/test/__pycache__ src/carcano/foolist/__pycache__ src/bin/__pycache__)
	@(rm -rf src/carcano_foolist.egg-info src/.eggs)
	@(rm -rf src/dist src/build)

unittests:
	$(info -> Makefile: Launching unit tests ...)
	@(cd src; python3 setup.py nosetests >/dev/null)

Let's have a go - just issue:

make

since "unittests" is listed in the "all" target, it runs all of the integration tests we defined in the previous post:

-> Makefile: Launching unit tests ...
testFoolistAppend (test_foolist.Foolist) ... ok
testFoolistIsIterable (test_foolist.Foolist) ... ok
testFoolistRemove (test_foolist.Foolist) ... ok

----------------------------------------------------------------------
Ran 3 tests in 0.005s

OK

Now we explore it a little bit, adding the minimal metadata necessary to create a package: our new "src/setup.py" must look like as follows:

from setuptools import setup, find_namespace_packages

setup(
    name='carcano_foolist',
    version='0.0.1',
    author='Marco Antonio Carcano',
    author_email='myemailaddress@mydomain.tld',
    url= 'https://github.com/mac-grimoire/python-spells.git',
    packages=find_namespace_packages(include=['carcano.*']),
    setup_requires=['nose>=1.0'],
    test_suite="nose.collector",
    tests_require=["nose"],
)

here:

• we import find_namespace_packages function, since we use it to find the packages to we want to include in our distribution package (note that it does also exist find_packages if you are working without a namespace)
• add the minimum required metadata - I do not explain them since they are self-explanatory:
  • version
  • author
  • author_email
  • url

We are now ready to explore how to create distribution packages: the very first thing is to add the logic necessary to handle the version of the releases to the Makefile - it must look like as follows:

all: unittests

clean:
	$(info -> Makefile: cleanup previous builds ... )
	@(rm -rf src/test/__pycache__ src/carcano/foolist/__pycache__ src/bin/__pycache__)
	@(rm -rf src/carcano_foolist.egg-info src/.eggs)
	@(rm -rf src/dist src/build)

release:
ifndef RELEASE 
	$(error Makefile: RELEASE is not set - please set it with a value with the following format: major.minor[.patch][sub] - for example 0.0.1 or 1.0.1-a2)
endif
	$(info -> Makefile: validating RELEASE=${RELEASE} format)
	@(echo ${RELEASE} |grep -qE "[0-9]+\.[0-9]+\.*[0-9]*[a-zA-Z]*[0-9]*") || (echo " ${RELEASE} is not in compliance with our version format"; exit 1)	
	@(cd src; sed -i -E "s/version='[0-9]+.[0-9]+\.*[0-9]+[a-zA-Z]*[0-9]*'/version='${RELEASE}'/g" setup.py)

unittests:
	$(info -> Makefile: Launching unit tests ...)
	@(cd src; python3 setup.py nosetests >/dev/null)

we added:

• a conditional to check that the "RELEASE" environment variable has been set: this variable is used to set the version of the package we are about to build
• the "release" target: it's purpose is to validate the format of the value of the "RELEASE" environment variable

Source Distributions (sdist)

As the name itself suggests, a source distribution contains source code: this is not only Python code, but also the source code of any extension modules (usually in C or C++) bundled with the package.

This kind of package requires the user to compile extension modules on his side if they are necessary. This of course implies requirements, such as having installed development tools, that is not always a viable choice, especially when dealing with servers.

Source distributions also contain a bundle of metadata files sitting in a directory called "<package-name>.egg-info". These metadata help with building and installing the package, but user’s don’t really need to do anything with it.

From the developer’s perspective, a source distribution is what gets created when you run the following command:

python3 setup.py sdist

We can of course add a new target to the Makefile:

sdist: release clean
	$(info -> Makefile: building the sdist distribution package ...)
	@(cd src; python3 setup.py sdist)

let's make the "sdist" target:

export RELEASE="0.0.1"
make sdist

the output is as follows - I cut it to keep it short:

-> Makefile: validating RELEASE=0.0.1 format
-> Makefile: cleanup previous builds ... 
-> Makefile: building the sdist distribution package ...
running sdist
running egg_info
...
creating dist
Creating tar archive
removing 'carcano_foolist-0.0.1' (and everything under it)

so now we have our brand new "src/dist/carcano_foolist-0.0.1.tar.gz" sdist file.

Built Distributions (Bdist)

It is the "classic" distribution format: despite it’s not necessarily binary, though, because it might contain only Python source code and/or byte-code. This kind of distribution relies on the "bdist" family of command options - for example:

python3 setup.py bdist

bdist has several options that let you create platform specific packages, such as gztar on Linux, executable zip on Windows and so on. It even lets you generate an RPM package on Linux.

We can of course add a new target to the Makefile:

bdist: release clean
	$(info -> Makefile: building the bdist distribution package ...)
	@(cd src; python3 setup.py bdist)

let's make the "bdist" target:

export RELEASE="0.0.1"
make bdist

the output is as follows - I cut it to keep it short:

-> Makefile: validating RELEASE=0.0.1 format
-> Makefile: cleanup previous builds ...
-> Makefile: building the bdist distribution package ...
running bdist
running bdist_dumb
...
creating /home/grimoire/fooproject/src/dist
Creating tar archive
removing 'build/bdist.linux-x86_64/dumb' (and everything under it)

so now we have our brand new "src/dist/carcano_foolist-0.0.1.linux-x86_64.tar.gz" bdist file.

Wheel Distributions (Bdist Wheel)

Wheel is a built distribution format that lets you skip the build stage required with source distributions. The straightforward benefits are:

• increased delivery and installation speed
• avoidance of the requisite of having development tools and libraries installed on the system

For these reasons, unless there's a strict constraint to distribute packages as sources, it is always preferable to build wheel packages.

In order to build wheels it is needed to install the "wheel" package: since it's always good having an updated "setuptools" version, we install wheel and update "setuptools" within the same command:

sudo python3 -m pip install -U wheel setuptools

Building a wheel requires to use the "bdist_wheel" command option of setup.py:

python3 setup.py bdist_wheel

We can of course add a new target to the Makefile:

wheel: release clean
	$(info -> Makefile: building the wheel distribution package ...)
	@(cd src; python3 setup.py bdist_wheel)

let's build the wheel:

export RELEASE="0.0.1"
make wheel

the output is as follows - I cut it to keep it short:

-> Makefile: validating RELEASE=0.0.1 format
-> Makefile: cleanup previous builds ...
-> Makefile: building the wheel distribution package ...l
running bdist_wheel
running build
...
creating 'dist/carcano_foolist-0.0.1-py3-none-any.whl' and adding 'build/bdist.linux-x86_64/wheel' to it
adding 'carcano_foolist-0.0.1.dist-info/METADATA'
adding 'carcano_foolist-0.0.1.dist-info/WHEEL'
adding 'carcano_foolist-0.0.1.dist-info/top_level.txt'
adding 'carcano_foolist-0.0.1.dist-info/RECORD'
removing build/bdist.linux-x86_64/wheel

so now we have our brand new "src/dist/carcano_foolist-0.0.1-py3-none-any.whl" bdist file

This is a ZIP package with a name ending by ".whl": this name is specially crafted to tell installers what Python versions and platforms the wheel is geared to.

We can of course have a look at its contents:

unzip -l src/dist/carcano_foolist-0.0.1-py3-none-any.whl

the output looks like as follows:

Archive:  src/dist/carcano_foolist-0.0.1-py3-none-any.whl
  Length      Date    Time    Name
---------  ---------- -----   ----
      141  09-17-2021 17:44   carcano/foolist/__init__.py
     2857  09-17-2021 17:43   carcano/foolist/foolist.py
     2195  09-17-2021 17:43   carcano/foolist/foolistitem.py
      992  09-17-2021 17:53   carcano_foolist-0.0.1.data/data/bin/logging.conf
       76  09-17-2021 17:54   carcano_foolist-0.0.1.data/data/share/doc/fooapp/rsyslog/fooapp.conf
     1187  10-06-2021 20:52   carcano_foolist-0.0.1.data/scripts/fooapp.py
      674  10-06-2021 20:52   carcano_foolist-0.0.1.dist-info/METADATA
       92  10-06-2021 20:52   carcano_foolist-0.0.1.dist-info/WHEEL
        8  10-06-2021 20:52   carcano_foolist-0.0.1.dist-info/top_level.txt
      909  10-06-2021 20:52   carcano_foolist-0.0.1.dist-info/RECORD
---------                     -------
     9131                     10 files

thoroughly talking about wheels would require too much space, but I can at least tell you the "must know" concepts: the filename is broken down into parts separated by hyphens as follows:

{dist}-{version}(-{build})?-{python}-{abi}-{platform}.whl

For example the package name cryptography-2.9.2-cp35-abi3-macosx_10_9_x86_64.whl is made of the following tokens:

• cryptography: the package name
• 2.9.2: the package version (it must be a PEP 440-compliant string)
• p35: the Python tag - it claims the Python implementation and version required by this wheel (in this case CPython 3.5) - this means for example that this wheel won't work with Jython
• abi3:  the Application Binary Interface (ABI) tag - it claims the level of binary compatibility of the Python C API.
• macosx_10_9_x86_64 is the platform tag: it can be broken down in the following tokens:
  • macosx: the macOS operating system
  • 10_9: is the macOS developer tools SDK version used to compile the Python that in turn built this wheel
  • x86_64:  reference to x86-64 instruction set architecture.

As of the manylinux platform tag, be wary that it poses requisites on the target host (the host that installs the wheel built by you). This means that very often it is necessary to update the PIP version so to support it:

• manylinux1 requires pip 8.1.0 or later
• manylinux2010 requires pip 19.0 or later
• manylinux2014 requires pip 19.3 or later

More on Packaging with setuptools

it has eventually come the time to explore our "setup.py": here it is enriched with additional contents, by the way

from setuptools import setup, find_namespace_packages

setup(
    name='carcano_foolist',
    version='0.0.1',
    author='Marco Antonio Carcano',
    author_email='myemailaddress@mydomain.tld',
    description='An example list object that exploit Pyhon iteratable facilities',
    classifiers=[
        'License :: OSI Approved :: GNU Lesser General Public License v3 or later (LGPLv3+)',
        'Development Status :: 3 - Alpha',
        'Intended Audience :: Developers',
        'Programming Language :: Python :: 3'
    ],
    url= 'https://github.com/mac-grimoire/python-spells.git',
    license= 'GNU Lesser General Public License v3 or later',
    keywords = 'iteratable list',
    project_urls={
        'Bug Tracker': 'https://github.com/mac-grimoire/python-spells.git',
        'Documentation': 'https://github.com/mac-grimoire/python-spells.git',
        'Source Code': 'https://github.com/mac-grimoire/python-spells.git',
    },    
    packages=find_namespace_packages(include=['carcano.*']),
    scripts=["bin/fooapp.py"],
    data_files=[
        ("bin", ["bin/logging.conf"]),
        ("share/doc/fooapp/rsyslog", ["share/doc/fooapp/rsyslog/fooapp.conf"])
    ],
    setup_requires=['nose>=1.0'],
    test_suite="nose.collector",
    tests_require=["nose"]
)

the fields name, version, author, author_email, description and url are self-explanatory.

As for the others:

• classifiers: a list of classifiers that apply to the distribution package -  the full list of the available classifiers can be found here
• license: a custom string you can use to describe the license
• keywords: a string with a space separated list of keywords
• project_urls: a dictionary of additional relevant URLs
• packages: the list of packages we want to pack together: despite you can manually list them, it is far better having them automatically discovered by find_namespace_packages or find_packages.
• scripts: a list of scripts that have to be included with the package
• data_files: a list of dictionaries with the destination path where to copy files as key, and a list of files to be copied as the second element
• setup_requires: used to import a version higher or equal to 1.0 of the nose test suite
• test_suite and tests_require are used to state that unit tests are performed using nose

For the sake of completeness, note that the outcome distribution package from building the  "setup.py" file can be easily build and installed as a whole simply using pip as follows:

cd src;
sudo pip install -e .

Be wary however that this last method has more to deal to experimenting rather than a clean and tidy way of working method.

Listing requirements

if the scripts or packages have requisites, we can list them into the install_requires list:

    install_requires=[
        'PyYAML',
        'jupyter'
    ]

sometimes some of the requisites are optional: in this case use the extra_require dictionary.

For example, if "dump" is an optional feature provided by package "carcano.bar" that requires "PyYAML" >= 2.2.0 and  jupyter:

    extras_require={
        'dump': ['PyYAML>=5.3.1', 'jupyter']     }

create the package as previously shown: when it comes to install the "dump" option, all of its requirements will be automatically installed as well. The installation command to be issued is as follows:

python3 -m pip install carcano.bar[dump]

Automatically Generating SHELL Scripts

If needed, we can even have shell scripts automatically generated to wrap our Python scripts: just specify them using entry_point dictionary as follows:

    entry_points={
        'console_scripts': ['my-command=<code class="language-bash">carcano.bar:main'] } 

Additional Files

a list of additional files can be specified using:

• data_files: files related to the python package. eg. documentation, static image files, configurations
• package_data: files that will be installed system-wise, not in site-package directory. eg. desktop icons, fonts

For example:

 data_files= [
        ("/etc/rsyslog.d", ["<code class="language-python">hare/doc/fooapp/rsyslog/fooapp.conf"]), ]

Publishing Distribution packages onto a repository

Once the package, we must publish it into a repository, so that other people can easily know that it does exist and download it.

Pypi

Python's online package repository is called PyPI (Python Package Index): when it comes to install or update files, pip looks for them onto PyPI and grabs them from it.

There's also an instance of PyPI dedicated to testing: we can register an account for test purposes at the following page.

Twine

Twine is the tool that has to be used to publish packages on PyPI.

You can install it via "dnf" or "yum" from the "EPEL" repository:

sudo dnf install -y epel-release
sudo dnf install -y twine

by default it pushes to PyPI, but we can force it to publish onto the testing PyPI repository (TestPyPI) by specifying "--repository-url" parameter as follows: 

twine upload --repository-url https://test.pypi.org/legacy/ dist/*

as we should expect, it prompts us for our username and password.

As you can easily guess, the statement to push to PyPI is:

twine upload dist/*

Let's go on and complete our Makefile - we must add two new targets:

• push-test manages the push to TestPyPI
• push manages the push to PyPI

push-test:
ifndef TWINE_USERNAME
	$(error Makefile: TWINE_USERNAME is not set - please set it or you won't be able to  authenticate to the PyPI server)
endif
ifndef TWINE_PASSWORD
	$(error Makefile: TWINE_PASSWORD is not set - please set it or you won't be able to  authenticate to the PyPI server)
endif
	$(info -> Makefile: pushing to Test PyPI...)
	@(cd src; twine upload --repository-url https://test.pypi.org/legacy/ dist/*)

push:
ifndef TWINE_USERNAME
	$(error Makefile: TWINE_USERNAME is not set - please set it or you won't be able to  authenticate to the PyPI server)
endif
ifndef TWINE_PASSWORD
	$(error Makefile: TWINE_PASSWORD is not set - please set it or you won't be able to  authenticate to the PyPI server)
endif
	$(info -> Makefile: pushing to PyPI ...)
	@(cd src; twine upload dist/*)

since "twine" can read credentials from TWINE_USERNAME and TWINE_PASSWORD environment variables, avoiding to prompt for them, we added two additional conditionals to check them.

For your convenience, here is the full Makefile:

all: unittests

clean:
	$(info -> Makefile: cleanup previous builds ... )
	@(rm -rf src/test/__pycache__ src/carcano/foolist/__pycache__ src/bin/__pycache__)
	@(rm -rf src/carcano_foolist.egg-info src/.eggs)
	@(rm -rf src/dist src/build)

release:
ifndef RELEASE 
	$(error Makefile: RELEASE is not set - please set it with a value with the following format: major.minor[.patch][sub] - for example 0.0.1 or 1.0.1-a2)
endif
	$(info -> Makefile: validating RELEASE=${RELEASE} format)
	@(echo ${RELEASE} |grep -qE "[0-9]+\.[0-9]+\.*[0-9]*[a-zA-Z]*[0-9]*") || (echo " ${RELEASE} is not in compliance with our version format"; exit 1)	
	@(cd src; sed -i -E "s/version='[0-9]+.[0-9]+\.*[0-9]+[a-zA-Z]*[0-9]*'/version='${RELEASE}'/g" setup.py)

unittests:
	$(info -> Makefile: Launching unit tests ...)
	@(cd src; python3 setup.py nosetests >/dev/null)

sdist: release clean
	$(info -> Makefile: building the sdist distribution package ...)
	@(cd src; python3 setup.py sdist)

bdist: release clean
	$(info -> Makefile: building the bdist distribution package ...)
	@(cd src; python3 setup.py bdist)

wheel: release clean
	$(info -> Makefile: building the wheel distribution package ...)
	@(cd src; python3 setup.py bdist_wheel)

push-test:
ifndef TWINE_USERNAME
	$(error Makefile: TWINE_USERNAME is not set - please set it or you won't be able to  authenticate to the PyPI server)
endif
ifndef TWINE_PASSWORD
	$(error Makefile: TWINE_PASSWORD is not set - please set it or you won't be able to  authenticate to the PyPI server)
endif
	$(info -> Makefile: pushing to Test PyPI...)
	@(cd src; twine upload --repository-url https://test.pypi.org/legacy/ dist/*)

push:
ifndef TWINE_USERNAME
	$(error Makefile: TWINE_USERNAME is not set - please set it or you won't be able to  authenticate to the PyPI server)
endif
ifndef TWINE_PASSWORD
	$(error Makefile: TWINE_PASSWORD is not set - please set it or you won't be able to  authenticate to the PyPI server)
endif
	$(info -> Makefile: pushing to PyPI ...)
	@(cd src; twine upload dist/*)

we can finally have a go and push the wheel to TestPyPI:

first we have to export TWINE_USERNAME and TWINE_PASSWORD environment variables, or the Makefile will fail:

export TWINE_USERNAME='myusername'
export TWINE_PASSWORD='mypassword'

we can eventually push

make push-test

the output is as follows:

-> Makefile: pushing to Test PyPI...
Uploading distributions to https://test.pypi.org/legacy/
Uploading carcano_foolist-0.0.1-py3-none-any.whl
100%|█████████████████████████████████████████████████████████████████████████| 8.35k/8.35k [00:02<00:00, 4.22kB/s]

View at:
https://test.pypi.org/project/carcano-foolist/0.0.1/

Now that our package has been published on the test PyPI, anybody can easily install it using pip by supplying the "--index-url" command option. For example:

sudo python3 -m pip install -i https://test.pypi.org/simple/ carcano-foolist

the only drawback is that if a package has dependencies that cannot be resolved within TestPyPI, we must explicitly tell pip to fall-back to the regular PyPI repository: in this case we must also provide "--extra-index-url" parameter.

For example:

sudo python3 -m pip install -i https://test.pypi.org/simple/ --extra-index-url https://pypi.org/simple/ carcano-foolist

We can then go-on with the regular development cycle, fixing errors and adding features when needed.

Footnotes

Here it ends the second of this trilogy of posts dedicated to Python: I hope you enjoyed it. You now know how to deal with modules, import packages, distribution packages and their format. You now have the necessary skills to create distribution packages leveraging on "setuptools", to create unit tests and to put everything into a project managed by the "make" utility, so to easily include everything in a Continuous Delivery toolchain.

In the next post we see how to package this project into two RPM packages: the first installs the modules provided by the Python package, whereas the second installs the example script that implements a sample application that imports these modules. This last package also performs post installation tasks so as to reconfigure rsyslog.
In addition to that, we also see how to digitally sign these RPM packages, and in general how to set up an RPM development environment, with a good overview on how to exploit the RPM building tools. Don't miss it.