hmz007
36ed224bac
Signed-off-by: hmz007 <hmz007@gmail.com> Change-Id: Ib87fdbe7a0be7dc961af3500fe9ea4589a127f9f
577 lines
18 KiB
ReStructuredText
577 lines
18 KiB
ReStructuredText
.. _`package_discovery`:
|
||
|
||
========================================
|
||
Package Discovery and Namespace Package
|
||
========================================
|
||
|
||
.. note::
|
||
a full specification for the keyword supplied to ``setup.cfg`` or
|
||
``setup.py`` can be found at :doc:`keywords reference <keywords>`
|
||
|
||
.. note::
|
||
the examples provided here are only to demonstrate the functionality
|
||
introduced. More metadata and options arguments need to be supplied
|
||
if you want to replicate them on your system. If you are completely
|
||
new to setuptools, the :doc:`quickstart section <quickstart>` is a good
|
||
place to start.
|
||
|
||
``Setuptools`` provide powerful tools to handle package discovery, including
|
||
support for namespace package.
|
||
|
||
Normally, you would specify the package to be included manually in the following manner:
|
||
|
||
.. tab:: setup.cfg
|
||
|
||
.. code-block:: ini
|
||
|
||
[options]
|
||
#...
|
||
packages =
|
||
mypkg1
|
||
mypkg2
|
||
|
||
.. tab:: setup.py
|
||
|
||
.. code-block:: python
|
||
|
||
setup(
|
||
# ...
|
||
packages=['mypkg1', 'mypkg2']
|
||
)
|
||
|
||
.. tab:: pyproject.toml (**EXPERIMENTAL**) [#experimental]_
|
||
|
||
.. code-block:: toml
|
||
|
||
# ...
|
||
[tool.setuptools]
|
||
packages = ["mypkg1", "mypkg2"]
|
||
# ...
|
||
|
||
|
||
If your packages are not in the root of the repository you also need to
|
||
configure ``package_dir``:
|
||
|
||
.. tab:: setup.cfg
|
||
|
||
.. code-block:: ini
|
||
|
||
[options]
|
||
# ...
|
||
package_dir =
|
||
= src
|
||
# directory containing all the packages (e.g. src/mypkg1, src/mypkg2)
|
||
# OR
|
||
package_dir =
|
||
mypkg1 = lib1
|
||
# mypkg1.mod corresponds to lib1/mod.py
|
||
# mypkg1.subpkg.mod corresponds to lib1/subpkg/mod.py
|
||
mypkg2 = lib2
|
||
# mypkg2.mod corresponds to lib2/mod.py
|
||
mypkg2.subpkg = lib3
|
||
# mypkg2.subpkg.mod corresponds to lib3/mod.py
|
||
|
||
.. tab:: setup.py
|
||
|
||
.. code-block:: python
|
||
|
||
setup(
|
||
# ...
|
||
package_dir = {"": "src"}
|
||
# directory containing all the packages (e.g. src/mypkg1, src/mypkg2)
|
||
)
|
||
|
||
# OR
|
||
|
||
setup(
|
||
# ...
|
||
package_dir = {
|
||
"mypkg1": "lib1", # mypkg1.mod corresponds to lib1/mod.py
|
||
# mypkg1.subpkg.mod corresponds to lib1/subpkg/mod.py
|
||
"mypkg2": "lib2", # mypkg2.mod corresponds to lib2/mod.py
|
||
"mypkg2.subpkg": "lib3" # mypkg2.subpkg.mod corresponds to lib3/mod.py
|
||
# ...
|
||
)
|
||
|
||
.. tab:: pyproject.toml (**EXPERIMENTAL**) [#experimental]_
|
||
|
||
.. code-block:: toml
|
||
|
||
[tool.setuptools]
|
||
# ...
|
||
package-dir = {"" = "src"}
|
||
# directory containing all the packages (e.g. src/mypkg1, src/mypkg2)
|
||
|
||
# OR
|
||
|
||
[tool.setuptools.package-dir]
|
||
mypkg1 = "lib1"
|
||
# mypkg1.mod corresponds to lib1/mod.py
|
||
# mypkg1.subpkg.mod corresponds to lib1/subpkg/mod.py
|
||
mypkg2 = "lib2"
|
||
# mypkg2.mod corresponds to lib2/mod.py
|
||
"mypkg2.subpkg" = "lib3"
|
||
# mypkg2.subpkg.mod corresponds to lib3/mod.py
|
||
# ...
|
||
|
||
This can get tiresome really quickly. To speed things up, you can rely on
|
||
setuptools automatic discovery, or use the provided tools, as explained in
|
||
the following sections.
|
||
|
||
|
||
.. _auto-discovery:
|
||
|
||
Automatic discovery
|
||
===================
|
||
|
||
.. warning:: Automatic discovery is an **experimental** feature and might change
|
||
(or be completely removed) in the future.
|
||
See :ref:`custom-discovery` for a stable way of configuring ``setuptools``.
|
||
|
||
By default ``setuptools`` will consider 2 popular project layouts, each one with
|
||
its own set of advantages and disadvantages [#layout1]_ [#layout2]_ as
|
||
discussed in the following sections.
|
||
|
||
Setuptools will automatically scan your project directory looking for these
|
||
layouts and try to guess the correct values for the :ref:`packages <declarative
|
||
config>` and :doc:`py_modules </references/keywords>` configuration.
|
||
|
||
.. important::
|
||
Automatic discovery will **only** be enabled if you **don't** provide any
|
||
configuration for ``packages`` and ``py_modules``.
|
||
If at least one of them is explicitly set, automatic discovery will not take place.
|
||
|
||
**Note**: specifying ``ext_modules`` might also prevent auto-discover from
|
||
taking place, unless your opt into :doc:`pyproject_config` (which will
|
||
disable the backward compatible behaviour).
|
||
|
||
.. _src-layout:
|
||
|
||
src-layout
|
||
----------
|
||
The project should contain a ``src`` directory under the project root and
|
||
all modules and packages meant for distribution are placed inside this
|
||
directory::
|
||
|
||
project_root_directory
|
||
├── pyproject.toml
|
||
├── setup.cfg # or setup.py
|
||
├── ...
|
||
└── src/
|
||
└── mypkg/
|
||
├── __init__.py
|
||
├── ...
|
||
└── mymodule.py
|
||
|
||
This layout is very handy when you wish to use automatic discovery,
|
||
since you don't have to worry about other Python files or folders in your
|
||
project root being distributed by mistake. In some circumstances it can be
|
||
also less error-prone for testing or when using :pep:`420`-style packages.
|
||
On the other hand you cannot rely on the implicit ``PYTHONPATH=.`` to fire
|
||
up the Python REPL and play with your package (you will need an
|
||
`editable install`_ to be able to do that).
|
||
|
||
.. _flat-layout:
|
||
|
||
flat-layout
|
||
-----------
|
||
*(also known as "adhoc")*
|
||
|
||
The package folder(s) are placed directly under the project root::
|
||
|
||
project_root_directory
|
||
├── pyproject.toml
|
||
├── setup.cfg # or setup.py
|
||
├── ...
|
||
└── mypkg/
|
||
├── __init__.py
|
||
├── ...
|
||
└── mymodule.py
|
||
|
||
This layout is very practical for using the REPL, but in some situations
|
||
it can be can be more error-prone (e.g. during tests or if you have a bunch
|
||
of folders or Python files hanging around your project root)
|
||
|
||
To avoid confusion, file and folder names that are used by popular tools (or
|
||
that correspond to well-known conventions, such as distributing documentation
|
||
alongside the project code) are automatically filtered out in the case of
|
||
*flat-layout*:
|
||
|
||
.. autoattribute:: setuptools.discovery.FlatLayoutPackageFinder.DEFAULT_EXCLUDE
|
||
|
||
.. autoattribute:: setuptools.discovery.FlatLayoutModuleFinder.DEFAULT_EXCLUDE
|
||
|
||
.. warning::
|
||
If you are using auto-discovery with *flat-layout*, ``setuptools`` will
|
||
refuse to create :term:`distribution archives <Distribution Package>` with
|
||
multiple top-level packages or modules.
|
||
|
||
This is done to prevent common errors such as accidentally publishing code
|
||
not meant for distribution (e.g. maintenance-related scripts).
|
||
|
||
Users that purposefully want to create multi-package distributions are
|
||
advised to use :ref:`custom-discovery` or the ``src-layout``.
|
||
|
||
There is also a handy variation of the *flat-layout* for utilities/libraries
|
||
that can be implemented with a single Python file:
|
||
|
||
single-module distribution
|
||
^^^^^^^^^^^^^^^^^^^^^^^^^^
|
||
|
||
A standalone module is placed directly under the project root, instead of
|
||
inside a package folder::
|
||
|
||
project_root_directory
|
||
├── pyproject.toml
|
||
├── setup.cfg # or setup.py
|
||
├── ...
|
||
└── single_file_lib.py
|
||
|
||
|
||
.. _custom-discovery:
|
||
|
||
Custom discovery
|
||
================
|
||
|
||
If the automatic discovery does not work for you
|
||
(e.g., you want to *include* in the distribution top-level packages with
|
||
reserved names such as ``tasks``, ``example`` or ``docs``, or you want to
|
||
*exclude* nested packages that would be otherwise included), you can use
|
||
the provided tools for package discovery:
|
||
|
||
.. tab:: setup.cfg
|
||
|
||
.. code-block:: ini
|
||
|
||
[options]
|
||
packages = find:
|
||
#or
|
||
packages = find_namespace:
|
||
|
||
.. tab:: setup.py
|
||
|
||
.. code-block:: python
|
||
|
||
from setuptools import find_packages
|
||
# or
|
||
from setuptools import find_namespace_packages
|
||
|
||
.. tab:: pyproject.toml (**EXPERIMENTAL**) [#experimental]_
|
||
|
||
.. code-block:: toml
|
||
|
||
# ...
|
||
[tool.setuptools.packages]
|
||
find = {} # Scanning implicit namespaces is active by default
|
||
# OR
|
||
find = {namespace = false} # Disable implicit namespaces
|
||
|
||
|
||
Finding simple packages
|
||
-----------------------
|
||
Let's start with the first tool. ``find:`` (``find_packages()``) takes a source
|
||
directory and two lists of package name patterns to exclude and include, and
|
||
then return a list of ``str`` representing the packages it could find. To use
|
||
it, consider the following directory::
|
||
|
||
mypkg
|
||
├── setup.cfg # and/or setup.py, pyproject.toml
|
||
└── src
|
||
├── pkg1
|
||
│ └── __init__.py
|
||
├── pkg2
|
||
│ └── __init__.py
|
||
├── aditional
|
||
│ └── __init__.py
|
||
└── pkg
|
||
└── namespace
|
||
└── __init__.py
|
||
|
||
To have setuptools to automatically include packages found
|
||
in ``src`` that starts with the name ``pkg`` and not ``additional``:
|
||
|
||
.. tab:: setup.cfg
|
||
|
||
.. code-block:: ini
|
||
|
||
[options]
|
||
packages = find:
|
||
package_dir =
|
||
=src
|
||
|
||
[options.packages.find]
|
||
where = src
|
||
include = pkg*
|
||
exclude = additional
|
||
|
||
.. note::
|
||
``pkg`` does not contain an ``__init__.py`` file, therefore
|
||
``pkg.namespace`` is ignored by ``find:`` (see ``find_namespace:`` below).
|
||
|
||
.. tab:: setup.py
|
||
|
||
.. code-block:: python
|
||
|
||
setup(
|
||
# ...
|
||
packages=find_packages(
|
||
where='src',
|
||
include=['pkg*'],
|
||
exclude=['additional'],
|
||
),
|
||
package_dir={"": "src"}
|
||
# ...
|
||
)
|
||
|
||
|
||
.. note::
|
||
``pkg`` does not contain an ``__init__.py`` file, therefore
|
||
``pkg.namespace`` is ignored by ``find_packages()``
|
||
(see ``find_namespace_packages()`` below).
|
||
|
||
.. tab:: pyproject.toml (**EXPERIMENTAL**) [#experimental]_
|
||
|
||
.. code-block:: toml
|
||
|
||
[tool.setuptools.packages.find]
|
||
where = ["src"]
|
||
include = ["pkg*"]
|
||
exclude = ["additional"]
|
||
namespaces = false
|
||
|
||
.. note::
|
||
When using ``tool.setuptools.packages.find`` in ``pyproject.toml``,
|
||
setuptools will consider :pep:`implicit namespaces <420>` by default when
|
||
scanning your project directory.
|
||
To avoid ``pkg.namespace`` from being added to your package list
|
||
you can set ``namespaces = false``. This will prevent any folder
|
||
without an ``__init__.py`` file from being scanned.
|
||
|
||
.. important::
|
||
``include`` and ``exclude`` accept strings representing :mod:`glob` patterns.
|
||
These patterns should match the **full** name of the Python module (as if it
|
||
was written in an ``import`` statement).
|
||
|
||
For example if you have ``util`` pattern, it will match
|
||
``util/__init__.py`` but not ``util/files/__init__.py``.
|
||
|
||
The fact that the parent package is matched by the pattern will not dictate
|
||
if the submodule will be included or excluded from the distribution.
|
||
You will need to explicitly add a wildcard (e.g. ``util*``)
|
||
if you want the pattern to also match submodules.
|
||
|
||
.. _Namespace Packages:
|
||
|
||
Finding namespace packages
|
||
--------------------------
|
||
``setuptools`` provides the ``find_namespace:`` (``find_namespace_packages()``)
|
||
which behaves similarly to ``find:`` but works with namespace package.
|
||
|
||
Before diving in, it is important to have a good understanding of what
|
||
:pep:`namespace packages <420>` are. Here is a quick recap.
|
||
|
||
When you have two packages organized as follows:
|
||
|
||
.. code-block:: bash
|
||
|
||
/Users/Desktop/timmins/foo/__init__.py
|
||
/Library/timmins/bar/__init__.py
|
||
|
||
If both ``Desktop`` and ``Library`` are on your ``PYTHONPATH``, then a
|
||
namespace package called ``timmins`` will be created automatically for you when
|
||
you invoke the import mechanism, allowing you to accomplish the following:
|
||
|
||
.. code-block:: pycon
|
||
|
||
>>> import timmins.foo
|
||
>>> import timmins.bar
|
||
|
||
as if there is only one ``timmins`` on your system. The two packages can then
|
||
be distributed separately and installed individually without affecting the
|
||
other one.
|
||
|
||
Now, suppose you decide to package the ``foo`` part for distribution and start
|
||
by creating a project directory organized as follows::
|
||
|
||
foo
|
||
├── setup.cfg # and/or setup.py, pyproject.toml
|
||
└── src
|
||
└── timmins
|
||
└── foo
|
||
└── __init__.py
|
||
|
||
If you want the ``timmins.foo`` to be automatically included in the
|
||
distribution, then you will need to specify:
|
||
|
||
.. tab:: setup.cfg
|
||
|
||
.. code-block:: ini
|
||
|
||
[options]
|
||
package_dir =
|
||
=src
|
||
packages = find_namespace:
|
||
|
||
[options.packages.find]
|
||
where = src
|
||
|
||
``find:`` won't work because timmins doesn't contain ``__init__.py``
|
||
directly, instead, you have to use ``find_namespace:``.
|
||
|
||
You can think of ``find_namespace:`` as identical to ``find:`` except it
|
||
would count a directory as a package even if it doesn't contain ``__init__.py``
|
||
file directly.
|
||
|
||
.. tab:: setup.py
|
||
|
||
.. code-block:: python
|
||
|
||
setup(
|
||
# ...
|
||
packages=find_namespace_packages(where='src'),
|
||
package_dir={"": "src"}
|
||
# ...
|
||
)
|
||
|
||
When you use ``find_packages()``, all directories without an
|
||
``__init__.py`` file will be disconsidered.
|
||
On the other hand, ``find_namespace_packages()`` will scan all
|
||
directories.
|
||
|
||
.. tab:: pyproject.toml (**EXPERIMENTAL**) [#experimental]_
|
||
|
||
.. code-block:: toml
|
||
|
||
[tool.setuptools.packages.find]
|
||
where = ["src"]
|
||
|
||
When using ``tool.setuptools.packages.find`` in ``pyproject.toml``,
|
||
setuptools will consider :pep:`implicit namespaces <420>` by default when
|
||
scanning your project directory.
|
||
|
||
After installing the package distribution, ``timmins.foo`` would become
|
||
available to your interpreter.
|
||
|
||
.. warning::
|
||
Please have in mind that ``find_namespace:`` (setup.cfg),
|
||
``find_namespace_packages()`` (setup.py) and ``find`` (pyproject.toml) will
|
||
scan **all** folders that you have in your project directory if you use a
|
||
:ref:`flat-layout`.
|
||
|
||
If used naïvely, this might result in unwanted files being added to your
|
||
final wheel. For example, with a project directory organized as follows::
|
||
|
||
foo
|
||
├── docs
|
||
│ └── conf.py
|
||
├── timmins
|
||
│ └── foo
|
||
│ └── __init__.py
|
||
└── tests
|
||
└── tests_foo
|
||
└── __init__.py
|
||
|
||
final users will end up installing not only ``timmins.foo``, but also
|
||
``docs`` and ``tests.tests_foo``.
|
||
|
||
A simple way to fix this is to adopt the aforementioned :ref:`src-layout`,
|
||
or make sure to properly configure the ``include`` and/or ``exclude``
|
||
accordingly.
|
||
|
||
.. tip::
|
||
After :ref:`building your package <building>`, you can have a look if all
|
||
the files are correct (nothing missing or extra), by running the following
|
||
commands:
|
||
|
||
.. code-block:: bash
|
||
|
||
tar tf dist/*.tar.gz
|
||
unzip -l dist/*.whl
|
||
|
||
This requires the ``tar`` and ``unzip`` to be installed in your OS.
|
||
On Windows you can also use a GUI program such as 7zip_.
|
||
|
||
|
||
Legacy Namespace Packages
|
||
=========================
|
||
The fact you can create namespace package so effortlessly above is credited
|
||
to `PEP 420 <https://www.python.org/dev/peps/pep-0420/>`_. It use to be more
|
||
cumbersome to accomplish the same result. Historically, there were two methods
|
||
to create namespace packages. One is the ``pkg_resources`` style supported by
|
||
``setuptools`` and the other one being ``pkgutils`` style offered by
|
||
``pkgutils`` module in Python. Both are now considered deprecated despite the
|
||
fact they still linger in many existing packages. These two differ in many
|
||
subtle yet significant aspects and you can find out more on `Python packaging
|
||
user guide <https://packaging.python.org/guides/packaging-namespace-packages/>`_
|
||
|
||
|
||
``pkg_resource`` style namespace package
|
||
----------------------------------------
|
||
This is the method ``setuptools`` directly supports. Starting with the same
|
||
layout, there are two pieces you need to add to it. First, an ``__init__.py``
|
||
file directly under your namespace package directory that contains the
|
||
following:
|
||
|
||
.. code-block:: python
|
||
|
||
__import__("pkg_resources").declare_namespace(__name__)
|
||
|
||
And the ``namespace_packages`` keyword in your ``setup.cfg`` or ``setup.py``:
|
||
|
||
.. tab:: setup.cfg
|
||
|
||
.. code-block:: ini
|
||
|
||
[options]
|
||
namespace_packages = timmins
|
||
|
||
.. tab:: setup.py
|
||
|
||
.. code-block:: python
|
||
|
||
setup(
|
||
# ...
|
||
namespace_packages=['timmins']
|
||
)
|
||
|
||
And your directory should look like this
|
||
|
||
.. code-block:: bash
|
||
|
||
foo
|
||
├── setup.cfg # and/or setup.py, pyproject.toml
|
||
└── src
|
||
└── timmins
|
||
├── __init__.py
|
||
└── foo
|
||
└── __init__.py
|
||
|
||
Repeat the same for other packages and you can achieve the same result as
|
||
the previous section.
|
||
|
||
``pkgutil`` style namespace package
|
||
-----------------------------------
|
||
This method is almost identical to the ``pkg_resource`` except that the
|
||
``namespace_packages`` declaration is omitted and the ``__init__.py``
|
||
file contains the following:
|
||
|
||
.. code-block:: python
|
||
|
||
__path__ = __import__('pkgutil').extend_path(__path__, __name__)
|
||
|
||
The project layout remains the same and ``setup.cfg`` remains the same.
|
||
|
||
|
||
----
|
||
|
||
|
||
.. [#experimental]
|
||
Support for specifying package metadata and build configuration options via
|
||
``pyproject.toml`` is experimental and might change (or be completely
|
||
removed) in the future. See :doc:`/userguide/pyproject_config`.
|
||
.. [#layout1] https://blog.ionelmc.ro/2014/05/25/python-packaging/#the-structure
|
||
.. [#layout2] https://blog.ionelmc.ro/2017/09/25/rehashing-the-src-layout/
|
||
|
||
.. _editable install: https://pip.pypa.io/en/stable/cli/pip_install/#editable-installs
|
||
.. _7zip: https://www.7-zip.org
|