Python Packaging Notes

After writing exciting code, you may want to publish and share your code with others. It seems pretty intuitive to do so - organize the python file folder, write some installation instructions, and upload them to Github and PyPI. However, this process is usually tricker than we’ve expected: We may encounter various issues like hidden bugs in the code, incompatible environments, etc.

There are many good resources available online, but they usually appear in separate posts or individual code snippets. And some important tricks are seldom mentioned in the popular tutorials.

To this end, I cleaned up the Python package building pipeline, and include some important posts that help a lot when I am making the package layoutparser. Though this post is majorly for personal reference, I hope it could also be useful to you.

Starter Reading

• python-packaging
• [Highly Recommended] Packaging a python library

Pipeline for Building a Library

1. Write the code and package them into a folder, add the __init__.py file when necessary → Project Structure
2. Test them and ensure the there weren’t bugs → Code Quality Management & CI
3. Add documentation if necessary → Write Documentation
   1. Create an account on the readthdocs website
4. Extra document work
   1. Choose a license
   2. Add a readme file
5. Prepare to distribute your work → Dependency and Distribution
   1. Add a [setup.py](http://setup.py) file listing dependencies and installation instructions
   2. Upload your package to pypi

Project Structure

• The src layout for creating the package

• Stitch functions in multiple files in the __init__.py file

  https://python3-cookbook.readthedocs.io/zh_CN/latest/c10/p04_split_module_into_multiple_files.html#id1

Code Quality Management and CI

• Using pytest and tox to test your code
  • pytest: https://docs.pytest.org/en/stable/getting-started.html
  • pytest + tox: https://tox.readthedocs.io/en/latest/example/pytest.html

Write Documentation

• Generate doc using sphinx
  • There are two parts of documentation you would like to add in your project:
    • A series of stitching pages, including index.rst, and other pages that
      • have some necessary introduction and explanation of your libraries
      • tell the sphinx software how to render code docstrings
    • Code docstrings with appropriate formats, e.g.:
      • Google Style: https://sphinxcontrib-napoleon.readthedocs.io/en/latest/example_google.html
  • Procedures
    • Generate an example project following https://docs.readthedocs.io/en/stable/intro/getting-started-with-sphinx.html
    • Modify the conf.py
    • Modify the index.rst
  • Reference
    • https://www.writethedocs.org/guide/writing/beginners-guide-to-docs/#id1
    • https://www.ericholscher.com/blog/2016/mar/15/dont-use-markdown-for-technical-docs/
• Configuring readthedocs to host your documentation and automatically build the documentation when the repo is updated
  • Getting started https://docs.readthedocs.io/en/stable/intro/getting-started-with-sphinx.html
  • Connect your project with readthedocs
  • Specify documentation building dependency https://docs.readthedocs.io/en/stable/guides/specifying-dependencies.html#specifying-dependencies
  • Add webhooks so that the doc could be automatically updated when you push to Github https://docs.readthedocs.io/en/stable/webhooks.html

Dependency and Distribution

• General Introduction for packaging python libraries
  • https://packaging.python.org/tutorials/packaging-projects/
• Choose a good version system
  • PEP 440 – Version Identification and Dependency Specification
  • There will be three locations that your version number should appear: 1) your package, 2) the setup.py file, and 3) the documentation file configuration.
  • How to coordinate the version numbers? Please check Jina.ai’s solution:
    • version number in __init__.py file
    • version number in setup.py file
    • version number in conf.py file in documentation configuration
• Add conditional requirements with extra_requirements in setup.py
  • Requirements: Some of the dependencies of the library are only necessary for certain scenarios.
  • Solution: conditional requirements https://hynek.me/articles/conditional-python-dependencies/
• Include/exclude non-code files in your library using MANIFEST
  • https://docs.python.org/2/distutils/sourcedist.html#the-manifest-in-template

• How to write setup.py to include a git repo as a dependency

  • https://stackoverflow.com/a/54794506

  • How to state in requirements.txt a direct Github source

  • Note: however, if you want to publish the library to pypi, all dependencies in setup.py should also be pypi packages.

• Use twine to publish your library to pypi

  • https://twine.readthedocs.io/en/latest/

  • https://packaging.python.org/tutorials/packaging-projects/#generating-distribution-archives

Automation

• Github Workflows
• Automatic build and release
  • Github CI for Python
    • https://gist.github.com/mwouts/9842452d020c08faf9e84a3bba38a66f
  • Github Python Auto Release on PyPI
    • Creating and storing encrypted secrets https://docs.github.com/en/actions/configuring-and-managing-workflows/creating-and-storing-encrypted-secrets
    • https://github.com/actions/starter-workflows/blob/master/ci/python-publish.yml

Misc

• Add Badges in the readme of your project
  • You can generate your own badges: https://github.com/google/pybadges
  • Or using existing badges from shields.io:
    • https://naereen.github.io/badges/
    • https://shields.io