Python

Python is the "dynamic language of choice" of the Netherlands eScience Center.

Currently, there are two Python versions: 2 and 3. Should I use Python 2 or Python 3 for my development activity? Generally, Python 2.x is legacy, Python 3.x is the present and future of the language. However, not all Python libraries are compatible with Python 3.

  • Six: Python 2 and 3 Compatibility Library
  • 2to3: Automated Python 2 to 3 code translation
  • python-modernize: wrapper around 2to3

The philosophy of Python is summarized in the Zen of Python. In Python, this text can be retrieved with the import this command.

Dependencies and package management

Use pip or conda (note that pip and conda can be used side by side, see also what is the difference between pip and conda?).

If you are planning on distributing your code at a later stage, be aware that your choice of package management may affect your packaging process. See Building and packaging for more info.

Pip + virtualenv

Create isolated Python environments with virtualenv. Very much recommended for all Python projects since it:

  • installs Python modules when you are not root,
  • contains all Python dependencies so the environment keeps working after an upgrade, and
  • lets you select the Python version per environment, so you can test code compatibility between Python 2.x and 3.x.

To manage multiple virtualenv environments and reference them only by name, use virtualenvwrapper. To create a new environment, run mkvirtualenv environment_name, to start using it, run workon environment_name and to stop working with it, run deactivate.

If you are using Python 3 only, you can also make use of the standard library venv module. Creating a virtual environment with it is as easy as running python3 -m venv /path/to/environment. Run . /path/to/environment/bin/activate to start using it and deactivate to deactivate.

With virtualenv and venv, pip is used to install all dependencies. An increasing number of packages are using wheel, so pip downloads and installs them as binaries. This means they have no build dependencies and are much faster to install. If the installation of a package fails because of its native extensions or system library dependencies and you are not root, you have to revert to Conda (see below).

To keep a log of the packages used in your package, run pip freeze > requirements.txt in the root of your package. If some of the packages listed in requirements.txt are needed during testing only, use an editor to move those lines to test_requirements.txt. Now your package can be installed with

pip install -r requirements.txt
pip install -e .

Conda

Conda can be used instead of virtualenv and pip. It easily installs binary dependencies, like Python itself or system libraries. Installation of packages that are not using wheel but have a lot of native code is much faster than pip because Conda does not compile the package, it only downloads compiled packages. The disadvantage of Conda is that the package needs to have a Conda build recipe. Many Conda build recipes already exist, but they are less common than the setup.py that generally all Python packages have.

There are two main distributions of Conda: Anaconda and Miniconda. Anaconda is large and contains a lot of common packages, like numpy and matplotlib, whereas Miniconda is very lightweight and only contains Python. If you need more, the conda command acts as a package manager for Python packages.

Use conda install to install new packages and conda update to keep your system up to date. The conda command can also be used to create virtual environments.

For environments where you do not have admin rights (e.g. DAS-5) either Anaconda or Miniconda is highly recommended, since the install is very straightforward. The installation of packages through Conda seems very robust. If you want to add packages to the (Ana)conda repositories, please check Build using conda. A possible downside of Anaconda is the fact that this is offered by a commercial supplier, but we don't foresee any vendor lock-in issues.

Editors and IDEs

  • Every major text editor supports Python, either natively or through plugins. At the Netherlands eScience Center, often used editors are atom, Sublime Text and vim.
  • PyDev is an open source IDE. The source code is available here. It has debugging, unit testing, and reporting(code analysis, code coverage) support.
  • For those seeking an IDE, JetBrains PyCharm is the Python IDE of choice. PyCharm Community Edition is open source. The source code is available here. It has visual debugger, unit testing and code coverage support, profiler. List of other tools can be found here.

Coding style conventions

The style guide for Python is PEP8. The autopep8 package can automatically format most Python code to conform to the PEP 8 style guide. The more comprehensive yapf tool can automatically format code for optimal readability according to a chosen style (PEP 8 is the default). The isort package automatically formats and groups all imports in a standard, readable way.

The pep8 package is a tool to check your Python code against some of the style conventions in PEP 8. The pyflakes program checks for semantic errors and some style issues that pep8 doesn't pick up. Even more comprehensive than pyflakes, pylint is a configurable tool that checks for style, good coding practices, and some common mistakes.

Most of these tools can be integrated in text editors and IDEs for convenience.

Building and packaging code

For packaging your code, you can either use pip or conda. Neither of them is better than the other -- they are different; use the one which is more suitable for your project. pip may be more suitable for distributing pure python packages, and it provides some support for binary dependencies using wheels. conda may be more suitable when you have external dependencies which cannot be packaged in a wheel.

  • Use twine to upload your package to PyPI (so it can be installed with pip) (tutorial)
    • Packages should be uploaded to PyPI using the nlesc account
    • When distributing code through PyPI, non-python files (such as requirements.txt) will not be packaged automatically, you need to add them to a MANIFEST.in file.
    • To test whether your distribution will work correctly before uploading to PyPI, you can run python setup.py sdist in the root of your repository. Then try installing your package with pip install dist/<your_package>tar.gz.
  • Build using conda
    • If possible, add packages to conda-forge. Use BioConda or custom channels (hosted on GitHub) as alternatives if need be.
  • Python wheels are the new standard for distributing Python packages. For pure python code, without C extensions, use bdist_wheel with a Python 2 and Python 3 setup, or use bdist_wheel --universal if the code is compatible with both Python 2 and 3. If C extensions are used, each OS needs to have its own wheel. The manylinux docker images can be used for building wheels compatible with multiple Linux distributions. See the manylinux demo for an example. Wheel building can be automated using Travis (for pure python, Linux and OS X) and Appveyor (for Windows).

Testing

Please use pytest.

Code coverage

When you have tests it is also a good to see which source code is exercised by the test suite. Code coverage can be measured with the coverage Python package. The coverage package can also generate html reports which show which line was covered. Most test runners have have the coverage package integrated.

The code coverage reports can be published online in code quality service or code coverage services. Preferred is to use one of the code quality service which also handles code coverage listed below. If this is not possible or does not fit then use one of the generic code coverage service list in the software guide.

Code quality analysis tools and services

Code quality service is explained in the Generic software guide. There are multiple code quality services available for Python. There is not a best one, below is a short list of services with their different strenghts.

Codacy

Code quality and coverage grouped by file. Can setup goals to improve quality or coverage by file or category. For example project see https://www.codacy.com/app/3D-e-Chem/kripodb/dashboard

Scrutinizer

Code quality and coverage grouped by class and function. For example project see https://scrutinizer-ci.com/g/NLeSC/eEcology-Annotation-WS/

Landscape

Dedicated for Python code quality. Celery, Django and Flask specific behaviors. The Landscape analysis tool called prospector can be run locally. For example project see https://landscape.io/github/NLeSC/MAGMa

Debugging and profiling

Debugging

Profiling

There are a number of available profiling tools that are suitable for different situations.

  • cProfile measures number of function calls and how much CPU time they take. The output can be further analyzed using the pstats module.
  • For more fine-grained, line-by-line CPU time profiling, two modules can be used:
    • line_profiler provides a function decorator that measures the time spent on each line inside the function.
    • pprofile is less intrusive; it simply times entire Python scripts line-by-line. It can give output in callgrind format, which allows you to study the statistics and call tree in kcachegrind (often used for analyzing c(++) profiles from valgrind).

Logging

Writing Documentation

Python uses Docstrings for function level documentation. You can read a detailed description of docstring usage in PEP 257. The default location to put HTML documentation is Read the Docs. You can connect your account at Read the Docs to your GitHub account and let the HTML be generated automatically using Sphinx.

Autogenerating the documentation

There are several tools that automatically generate documentation from docstrings. These are the most used:

We recommend using Sphinx and Google documentation style.

General scientific

IPython and Jupyter notebooks (aka IPython notebooks)

IPython is an interactive Python interpreter -- very much the same as the standard Python interactive interpreter, but with some extra features (tab completion, shell commands, in-line help, etc).

Jupyter notebooks (formerly know as IPython notebooks) are browser based interactive Python enviroments. It incorporates the same features as the IPython console, plus some extras like in-line plotting. Look at some examples to find out more. Within a notebook you can alternate code with Markdown comments (and even LaTeX), which is great for reproducible research. Notebook extensions adds extra functionalities to notebooks. JupyterLab is a web-based environment with a lot of improvements and integrated tools. JupyterLab is still under development and may not be suitable if you need a stable tool.

Visualization

  • Matplotlib has been the standard in scientific visualization. It supports quick-and-dirty plotting through the pyplot submodule. Its object oriented interface can be somewhat arcane, but is highly customizable and runs natively on many platforms, making it compatible with all major OSes and environments. It supports most sources of data, including native Python objects, Numpy and Pandas.
    • Seaborn is a Python visualisation library based on Matplotlib and aimed towards statistical analysis. It supports numpy, pandas, scipy and statmodels.
  • Web-based:
    • Bokeh is Interactive Web Plotting for Python.
    • Plotly is another platform for interactive plotting through a web browser, including in Jupyter notebooks.
    • altair is a grammar of graphics style declarative statistical visualization library. It does not render visualizations itself, but rather outputs Vega-Lite JSON data. This can lead to a simplified workflow.
    • ggplot is a plotting library imported from R.

Database Interface

Parallelisation

CPython (the official and mainstream Python implementation) is not built for parallel processing due to the global interpreter lock.

Having said that, there are many packages that circumvent this constraint.

Web Frameworks

There are a lot web frameworks for Python that are very easy to run.

We recommend flask.

NLP/text mining

  • nltk Natural Language Toolkit
  • Pattern: web/text mining module
  • gensim: Topic modeling

Available templates

The recommended layout for files/modules etc in Python is described by Kenneth Reitz here

It is also available as a example repo on GitHub here.

Ben made a version of this repository that brings it closer to our ideal, but it is stil not perfect (e.g., needs replacing of project name).

An alternative might be to create a project template using (for example) cookiecutter. This solution needs more research.