Preserving environments with requirements.txt or environment.yml

Once you and your collaborators are comfortable managing different python virtual environments you are all set to start exporting and importing those environments for sharing.

The simplest way to share a python environment is to use a file which lists all the modules that are installed, and even their versions. Pip and conda have different ways of capturing dependencies.

pip – requirements.txt

For our example project we have used only one non-builtin python module: numpy. To specify that this is module is required we can create a file called requirements.txt and add the following:

numpy>=1.19

where the >=1.19 indicates that version 1.19 or greater will be fine. See the pip documentation for examples on how you can specify different version numbers, ranges, and exclusions. By default each module listed will be installed from the python package index (pypi), but you can also indicate other locations including local files/folders or github repositories.

If you don’t know what versions or dependencies you need for your particular project then you can get a hint by inspecting your current environment using pip freeze. This will list all the installed modules and versions for the current environment. The list will be much longer than you anticipate! To get a minimal list, start with an empty environment and then keep installing modules until your software will run, then run pip freeze >> requirements.txt.

To install all the modules within a requirements.txt file you run pip:

pip install -r requirements.txt

You now have a way of preserving and sharing your python environment with others. It is good practice to keep a requirements.txt file in your project directory, and to have this as part of your version control repository.

conda – environment.yml

Anaconda uses a slightly different file format to capture essentially the same information. For the above example our file would look like:

dependencies:
  - numpy>=1.19

Anaconda is able to reproduce not just the python modules, but the entire virtual environment including the versions of python and anaconda and also other non-python based codes. You can generate a good starting point using conda list --export.

In order to 1) not dictate your users’ virtual environment management choices, and 2) not duplicate information, the following method will link your environment.yml and requirements.txt files:

name: my-env
dependencies:
  - python>=3.6
  - anaconda
  - pip
  - pip:
    - -r requirements.txt

Conda understands pip formatted requirements, but pip doesn’t understand conda environment files, so there is no inverse version of the above linking.

Summary

In combination with a python virtual environment (previous lesson), we now have a greater expectation that our software will run the same on other people’s machines as is does on our own (test) machine. There will be some localisation (win/linux/osx) or language (en, ch, fr, etc) difference that might cause some problems down the line, but we have solved one of the most common sources of error.