Skip to main content

Develop in Linux

Makefile for Python Projects

A project is more than just code. There is the virtual environment, static analysis, tests, packaging and uploading to PiPy. All of these involve some command line interaction. These commands are drilled into my head as I use them but fade after time. When I need to use them again, I have to either look back at my notes or google search. I wanted a different, better way.

There are a number of different options. Both manage.py and click by the Pocoo team are command line builder tools in python. I have tried using manage.py in the past and ran into issues. It just took too much fussing to make it work for me. I kept having to modify the file as I was testing. I wanted to spend my time testing, not modifying my python file.

For a while, I used bash scripts. They ended up proliferating and I ended up making special case scripts. It just got out of hand.

I finally saw a Makefile which had some good ideas. After using it for several projects and going through a rewrite of most it recently, I thought I would share it. You can find it at https://github.com/jidn/python-Makefile.

Features

  • Integrated help

  • Creation of virtual environment

  • Multiple requirements.txt files supported

  • Static code analysis using flake8 and pydocstyle

  • Testing and coverage with additional command line arguments.

  • PiPy packaging and upload

Install

While you could clone it from GitHub, I would really recommend just grabbing the current file and copying it into your project directory using either wget or curl. The following should work for most people.

$ curl https://raw.githubusercontent.com/jidn/python-Makefile/master/Makefile > Makefile

Now there are some variables specific to your project you can change at the beginning. I recommend you examine the README document. It has the latest information.

Integrated Help

When you specify the help target/command, the common targets are printed to the console. I tried to make them concise and easy to remember.

$ make help
env Create virtualenv and install requirements
python=PYTHON_EXE interpreter to use, default=python
check Run style checks
test TEST_RUNNER on 'tests'
args="-x --pdb --ff" optional arguments
coverage Get coverage information, optional 'args' like test
upload Upload package to PyPI
clean clean-all Clean up and clean up removing virtualenv

Create Virtual Environment

Create the virtual environment in the .env directory within the current directory. No more stomping on other virtual environments in some global directory. Let's keep everything local to make environment debugging easier. This environment is also where all the requirements are installed.

You can specify the python interpreter version by adding python=PYTHON_EXE on the command-line or you can change the python = python line in Makefile to consistently use a specific version. The default python interpreter is the current python interpreter.

$ make env python=python2.7
$ make env python=python3.4m

If you want to put the environment into a different directory, you can either use the command line argument ENV

$ make env ENV=env

or modify the line in the Makefile.

ENV := env

Requirements

If a requirements.txt file exists, those files will be installed at the time the environment is created. You can always add or modify it later and make will add any additional files.

So you want to be different and not call the file requirements.txt but prefer requires.txt? We have you covered. Just modify Makefile and change the REQUIRE from requirements.txt to the file name of your choice.

REQUIRE := require.txt

You can specify it on the command line, but I don't recommend it. It won't automatically add any new changes and you have to remember to add it each time you recreate your environment.

I like to keep two different requirements files. The first is the package, what most everyone thinks of as a requirements file. I add a second one to my tests directory for my testing suite, but which is not required by my package. Below you can see my boilerplate tests/requirements.txt file. It contains packages used by default for checking code style, testing, and coverage.:

pytest-cov

Checking Code Style

You can check your code using both flake8 and pydocstyle using the check target. Additionally, you can run the checks separately as seen below.

$ make check
$ make flake8
$ make pep257

Testing & Coverage

Testing and repeated testing. I seem to spend a lot of time here and with all the different tests, I want to use different options against my test runner, py.test. You can change the test runner by modifying the TEST_RUNNER in the Makefile. Again, like all the other variables, you can use a command line argument to change it for a single execution.

$ make test TEST_RUNNER=nosetests
$ make test args="-v"
$ make test args="-x --pdb"

Coverage, or more specifically pytest-cov works the same way as testing. You can add command line arguments through args= option. If you don't want to use pytest-cov or change things up, change the coverage target in the Makefile; it's just a command line.

$ make coverage args="--cov-report html:cov_html"

PiPy Packaging and Upload

The upload target will take your package and send it off to PiPy. These are currently untested and I haven't used them in some time.

Final Thoughts

I hope this helps someone out there. If you have any thoughts, ideas, or changes, I would like to know.