docs.pelican.bernatchez.net - python

Set up a personal python package index

2025-05-24T18:23:52+00:00

Overview

This document describes setting up a private package index as a more convenient way to distribute python packages.

The need to use something like this grows as we add maintainers and packages to maintain.

Our python packages need to be compatile with the standard python package index PyPi.

We use a private package index until we are ready to make a package available to the general public, then we switch to PyPi.

This tool we are using keeps our packages compatible with PyPi for when we do make it available there.

devpi

It is a conforming package index system.

We will use it to serve our private package index.

Configuring the SW for devpi

This howto is derived from the following tutorial:

<https://devpi.net/docs/devpi/devpi/stable/+doc/quickstart-releaseprocess.html>_

We are working with python3. We want to setup up a virtual environment that conforms to our assumptions, and then install within that. We call the environment devpi_venv. We create a directory for our setup first.

mkdir ${HOME}/devpi_setup
cd ${HOME}/devpi_setup

We install the bare essentials.

python3 -m venv devpi_venv
source devpi_venv/bin/activate
python -m pip install -U pip
pip install wheel
pip install setuptools
pip install twine

restview is A Restructured Text format viewer I find usefull. (optional)

pip install restview

You invoke it and it keeps re-rendering your file as you modify it.

Like this:

restview README.rst

rst2pdf is convenient for generating pdf copies of .rst documuents.

pip install rst2pdf
rst2pdf doc/privatepypi.rst -o reformatted_docs/privatepypi.pdf

Set up devpi server on our laptop

Install devpi client and web server The non web server - devpi-server - is installed as a consequence.

pip install -U devpi-web devpi-client

Initialize an empty index (at /var/devpi-server by default) Note I had a permissions problem with the init one. So first I manually do this:

sudo mkdir  /var/devpi-server
sudo chown ubuntu:ubuntu /var/devpi-server

devpi-init

We want a config file for supervisor daemon to use. This will generate a bunch of config files under the current directory, the one we want included.

cd ${HOME}/devpi_setup
devpi-gen-config

This gives us the file gen-config/supervisor-devpi.conf to copy to /etc/supervisor/conf so that we can start up the server. I started by editing it to say start=False so that it would need to be started up manually. Then I copied it to the right place for it.

sudo cp gen-config/supervisor-devpi.conf /etc/supervisor/conf.d/devpi-server.conf
sudo supervisorctl update
sudo supervisorctl start devpi-server

Create a user, login as him and create the 'dev' index

devpi user -c pbernatchez password=foobar
devpi login pbernatchez --password=foobar
devpi index -c dev bases=root/pypi

Use our dev index

devpi use testuser/dev

Now we can make use of the private index.

We are using flit to publish to our index and it relies on the file : ~/.pypirc.

So we make an entry there for our index.

I gave it the name 'mypypi':

[distutils]
index-servers =
   mypypi
   testpypi

[mypypi]
repository = http://localhost:3141/pbernatchez/dev
username = pbernatchez

[testpypi]
repository = https://test.pypi.org/legacy/
username = pbernatchez

From here on, using flit, we can refer it as 'mypypi'.

deactivate
cd /home/ubuntu/allrepos/animbboard
source ${HOME}/venv_animbboard/bin/activate
flit build
flit publish --repository mypypi
pip uninstall  animbboard
pip install -i http://localhost:3141/pbernatchez/dev  animbboard

Set up a private package index

2025-05-24T18:23:52+00:00

Overview

At this writing our current practise is to generate versioned tarballs and to 'pip install' from those.

This describes setting up a private package index as a more convenient alternative to using tarballs directly.

Our python software needs to be kept compatible with the standard python package index for when we desire to offer it to the general public.

To ensure we are staying compatible with pypi, we need to run it through the process, but with a conforming private packaging index.

devpi to the rescue ('https://www.devpi.net/') .

It is a conforming package index system we can use to serve our private packages whithout releasing them to the public.

Configuring the SW for devpi

This information is derived from the following tutorial:

<https://devpi.net/docs/devpi/devpi/stable/+doc/quickstart-releaseprocess.html>_

We start by creating a virtual environment called "venv_devpi" by following the steps described in the document virtualenv.

Then we activate that environment and continue with the steps below.

Set up devpi server on our laptop

Install a devpi client and web server
pip install -U devpi-web devpi-client
Note:
The non-web server - devpi-server - is also installed as a side effect.
We must avoid needing sudo to access the index, so instead we change ownership from root to the logged in user. We also need to make sure supervisor runs our server as that user.

First manually do this:
sudo mkdir -p /var/devpi-server
sudo chown -R $(logname):$(logname) /var/devpi-server
ls -ld /var/devpi-server
Initialize an empty index (at /var/devpi-server by default)
devpi-init
Make a place to put our configuration data files:
mkdir -p ${HOME}/devpi_setup
The following will generate a bunch of config files, one of which we will need, under the current directory. Putting that stuff in ${HOME}/devpi_setup.
pushd . >/dev/null
cd ${HOME}/devpi_setup
devpi-gen-config
popd  >/dev/null
Note:
This gave us the file  ${HOME}/devpi_setup/gen-config/supervisor-devpi.conf
as a config file to start up our pypi server.  Start by editing
it manually to say autostart=False so that it will need to be
started up manually.
Append a line like "user = username" to the configuration so it launches as that user, the same user as above.
echo user = $(logname) >>${HOME}/devpi_setup/gen-config/supervisor-devpi.conf
Copy the config where where it needs to be, then start our server.
sudo cp ${HOME}/devpi_setup/gen-config/supervisor-devpi.conf /etc/supervisor/conf.d/supervisor-devpi.conf
sudo supervisorctl update
sudo supervisorctl start devpi-server
Create a user (within the server), login as that user and create the 'bzdev' index
devpi use http://localhost:3141
devpi user -c pbernatchez password=dummy
devpi login pbernatchez --password=dummy
devpi index -c bzdev bases=root/pypi
Use our bzdev index
devpi use pbernatchez/bzdev
Now we can make use of our private index.

We are using flit to publish to our index and it relies on the file : ~/.pypirc.

For pip I named it 'bzdevpi'. So we make an entry there for our index:
[distutils]
index-servers =
   bzdevpi
   testpypi

[bzdevpi]
repository = http://localhost:3141/pbernatchez/bzdev
username = pbernatchez

[testpypi]
repository = https://test.pypi.org/legacy/
username = pbernatchez
From here on, using flit,

To publish,

flit publish --repository bzdevpi

To install,

pip install -i http://localhost:3141/pbernatchez/bzdev foobar

Example:
deactivate
cd /home/ubuntu/allrepos/wait_ssh
source ${HOME}/venv_generic/bin/activate
flit build
flit publish --repository bzdevpi
pip uninstall wait_ssh
pip install -i http://localhost:3141/pbernatchez/bzdev wait_ssh

Set up a python virtual environment

2025-05-24T18:23:52+00:00

Make sure ubuntu system version is recent enough

lsb_release -a

Output (or higher)

Distributor ID:     Ubuntu

Description:        Ubuntu 24.04.3 LTS

Release:            24.04

Codename:           noble

python3 -V

Output

::: Python 3.12.3

Make a base for all our virtual environments

mkdir ${HOME}/.venv
# Now we can create all our virtual environments  under ${HOME}/.venv/

Make a Python Virtual Environment

# Here we call our first environment devpi
cd ${HOME}/.venv/
python3 -m venv devpi

OR using venvwrapper

mkvenv devpi

Once we have our environment created. We can install the elements we need to get started.

Activate It

source ${HOME}/.venv/devpi/bin/activate

Or with venvwrapper

venv devpi

install the basics

# install
python -m pip install -U pip
pip install wheel
pip install setuptools
pip install twine
pip install rst2pdf
pip install flit

# de-activate
deactivate

Saving prerequisites

Once your virtual environment is complete, it is nice to save the context. Then you can quicky restore your virtual environment from scratch if need be.

While Activated

pip freeze >requirements.txt

# It may be a good idea to keep such files under version control.

Recovering While Activated

pip install -r requirements.txt

Automatic activation

In many cases we want to activate a project whenever we log in. We can do that automatically by adding something like this to .bashrc and/or .profile

#
# Added for automatic vitual environment activation
#
if [ -f "${HOME}/.venv/devpi/bin/activate" ] ; then
    source ${HOME}/.venv/devpi/bin/activate
fi

OR with venvwrapper

#
# Added for automatic vitual environment activation
#
VENV_HOME=${HOME}/.venv
venv_wrapper=$(which venvwrapper.sh)
if [[ -n $venv_wrapper ]]; then
    source $venv_wrapper
    venv devpi
fi

python virtual environment

2025-05-24T18:23:52+00:00

Python Virtual Environment Setup

To have a good control over pre-requisite library modules for our software, we run in a python virtual environment.

That means that whenever you want to use or maintain the software you must first activate that virtual environment like this: (replace 'generic' with an existing project environment name)

source ${HOME}/venv_generic/bin/activate

Once active you can deactivate it like this:

deactivate

Installation

We install any non pip packages we rely on, if any, first. Then we install virtualenv. Then we create a virtual environment. Then we activate it.

Install virtualenv itself.

sudo apt update
sudo apt install virtualenv

Install a .rst to .pdf converter.

snap install rst2pdf

Create a virtual environment for the project

virtualenv --prompt venv_generic ${HOME}/venv_generic

Then activate the created environment

source ${HOME}/venv_generic/bin/activate

Now we install modules our project will need, first pip itself and the tools it needs.

python3 -m pip install -U pip
pip install wheel
pip install setuptools
pip install twine

We are using flit for building and installling our software.

pip install flit

Saving prerequisites

We keep our dependencies under version control, so each time we install more components we need to refresh our requirements file.

pip freeze >${HOME}/allrepos/generic/requirements.txt

Reinstalling

Later, when we are migrating to a different workstation, or for some reason need to restablish our project from scratch, we can install all our prerequisites in one shot instead of installing items one at a time like we did above.

pip install -r ${HOME}/allrepos/generic/requirements.txt

Automatic activation

In many cases we want to activate a project whenver we log in, so lets do that automatically.

Append this snippet to the end of our "${HOME}/.bashrc" file.

#
# Added for automatic vitual environment activation
#
if [ -f "${HOME}/venv_generic/bin/activate" ] ; then
    source ${HOME}/venv_generic/bin/activate
fi

Good ignoring keeps flit happy

Flit works nicely with git, but it is quite strict, so we must ignore all irrelevant files properly or flit will reject builds.

Below is a good start for a project ".gitignore" file:

*~
\#*\#
dist/*
pdf_docs/*
html_docs/*

Set Up A Wrapper For Venv Environments

2025-05-24T18:23:52+00:00

To maintain more than one python project at a time we leverage the the builtin python venv tool for creating and using virtual environments.

The venvwrapper package helps to keep the maintenance of multiple virtual environments easier. In short venv is enough, but venvwrapper makes it easier.

Installation

Install the venvwrapper package into your default system Python 3. That is, with no virtual environment active. This installs venvwrapper.sh so that it is available in PATH

sudo pip3 install --break-system-packages venvwrapper

Check that it is indeed available

which venvwrapper.sh

Set up automatic activation of venvwrapper

Add the code snippet below to one or both of these two files.

${HOME}/.bashrc
${HOME}/.profile

Code snippet:

VENV_HOME=${HOME}/.venv
venv_wrapper=$(which venvwrapper.sh)
if [[ -n ${venv_wrapper} ]]; then
    source ${venv_wrapper}
fi

Now Once you open a terminal window you have access to these commands

mkvenv devpi  # create and activate a new virtual environment
venv   devpi  # activate an existing virtual environment
rmvenv devpi  # remove one or more virtual environments
lsvenv        # list existing virtual environments