Using Multiple Requirements Files

What is a requirements.txt?

A requirements.txt file is a standard way to list Python dependencies for a project. It specifies the packages your project needs and optionally locks their versions. This file is used by tools like pip to install dependencies consistently across environments.

Why Use Multiple Requirements Files?

Using multiple requirements.txt files allows you to:

• Keep environment-specific dependencies separate.
• Avoid unnecessary packages in production environments.
• Reduce the risk of version conflicts.

Structuring Multiple Requirements Files

In my tutorial, Structuring Python Project Directories, we set up a dedicated folder to store the source code. Let us use the Gen EMR project as an example. Open up your terminal and navigate to the directory below:

1
2
3

| - gen_emr/
    | - gen_emr_src/  --> Open up this directory.
    | - gen_emr_venv/

1. Create a requirements directory

In your project’s source code directory, create a folder named requirements to house your requirements files.

1
2
3
4

| - gen_emr/
    | - gen_emr_src/
        | - requirements/  --> New
    | - gen_emr_venv/

2. Create the following files inside the requirements directory

1
2
3
4
5
6
7
8

| - gen_emr/
    | - gen_emr_src/
        | - requirements/
            | - base.txt  --> New
            | - local.txt  --> New
            | - staging.txt  --> New
            | - production.txt  --> New
    | - gen_emr_venv/

3. Populate base.txt with common dependencies

Add all dependencies shared across all environments to base.txt. For example:

1

Django==5.1.4

In relation to this topic,pip freeze is a command that outputs a list of all installed Python packages in the current environment along with their exact versions. This command can be used to generate the requirements.txt file.

Lesson learned: Instead of using pip freeze, manually specify the main packages your project requires in your requirements.txt.

I previously used the pip freeze command to generate my requirements file. However, this approach also includes the dependencies of my main packages.

For instance, installing Django automatically installs its dependencies, such as asgiref, sqlparse, and tzdata. When you run pip freeze > requirements.txt, the generated requirements.txt file will include these dependencies as well:

1
2
3
4

asgiref==3.8.1
Django==5.1.4
sqlparse==0.5.3
tzdata==2024.2 

This approach has several drawbacks. For instance, I previously encountered a package that had different dependencies for Windows and Linux. When I tried installing the packages using the requirements file, it led to errors. Additionally, this method introduces significant complexity when tracking the main packages for your project. For example, if Django is one of your primary packages, you likely don’t need to worry about dependencies like asgiref, sqlparse, or tzdata.

4. Customize Other Files

In environment-specific files (local.txt, staging.txt, production.txt), include base.txt and add dependencies unique to that environment. For example:

For local.txt:

1
2
3
4
5

# Include base dependencies
-r base.txt

# Local-specific tools
django-debug-toolbar==4.4.6

For staging.txt:

1
2
3
4
5

# Include base dependencies
-r base.txt

# Staging-specific tools
pytest-django==4.5.2

For production.txt

1
2
3
4
5
6

# Include base dependencies
-r base.txt

# Production-specific tools
django-cors-headers==4.6.0
gunicorn==23.0.0

Installing From Multiple Requirements Files

To install dependencies for a specific environment, use the -r flag followed by the file path. Make sure to activate your virtual environment before proceeding with the installation.

For local environment:

1

pip install -r requirements/local.txt

For staging environment:

1

pip install -r requirements/staging.txt

For production environment:

1

pip install -r requirements/production.txt