3

I'm making my 1st official project. It's written in Python, is open-sources, and I'd like people to be able to freely and easily fork and modify the code. The project name is "shelf" and the main script/executable is shelf.py.

I'm trying to make everything modular. The main script shelf.py should be the only thing the end-user manually interact with.

My file structure and import hierarchy:

shelf.py
├── src
│   ├── config
│   │   └── configure.py
│   ├── data
│   │   └── database.py
│   ├── downloading
│   │   └── download.py
│   ├── system
│   │   ├── fileio.py
│   │   ├── rw
│   │   │   ├── open.py
│   │   │   └── save.py
│   │   └── reader.py
│   └── ui
│       ├── cli.py
│       ├── gtk.py
│       └── events
│           └── user_input.py
└── tests

Here, shelf.py is in the root of the directory.

"""shelf.py"""

import src.config.configure as configure
import src.data.database as database
import src.downloading.download as download
import src.system.fileio as fileio
import src.system.reader as reader
import src.ui.cli as cli
import src.ui.events as events


...


"""fileio.py"""

import rw.open
import rw.save

# ...

The rule I've made is that a script can only ever import scripts (modules) in a subdirectory. open.py may not import save.py, and vice-versa. Coupling of these modules is all done in the parent module, which is fileio.py for open.py and save.py.

This rule is made to avoid circular dependencies and modularize all the functions as much as possible. Most functions will be pure functions.

Is this rule of "import only modules in a subdirectory" viable for development?

ChocolateOverflow
  • 151
  • 2

1 Answers1

2

Keeping a clear dependency graph is a good thing, because the results can be a bit unexpected when two modules import each other. However:

  • Only importing child modules is probably a bit restrictive. You might have one module that is imported by many other modules.

  • Only importing child modules will likely require doing a lot of dependency injection to make objects from sibling modules available to each other. In a reasonalby-sized project, this effort isn't worth it.

Your proposed layout also clashes a bit with Python best practices, and with how import works. When you import a module, the module name is searched in the directories in the sys.path list, which can be controlled with the PYTHONPATH environment variable. The import does not load relative paths! So when src/system/fileio.py runs import rw.open this might find /usr/lib/python/rw/open.py or src/rw/open.py but won't generally find src/system/rw/open.py.

Python does support relative imports with the from .rw.open import * syntax, but this requires that you are importing from a module (not the top level script) and you can only use the from ... import ... syntax.

The best solution would likely be to learn about setuptools, which can help with managing lookup paths and providing entrypoints into your program. This would likely lead to a project layout like this:

setup.py   # metadata
shelf/
  __init__.py
  __main__.py  # provides an entrypoint
  config.py
  database.py
  download.py
  system/
    __init__.py
    fileio.py
    rw/
      __init__.py
      open.py
      save.py
    reader.py
  ui/
    __init__.py
    cli.py
    gtk.py
    events/
      __init__.py
      user_inputs.py
  tests/
    ...

What is that __init__.py? In Python, a directory can act as a module like foo.py when it contains an init file: foo/__init__.py. This file can be empty, but is necessary for nesting modules.

What is that __main__.py? This file will be executed if you want to be able to run your program like python3 -m shelf. This has the advantage that you can use relative imports, which is not possible when you execute a file directly like python3 shelf.py.

If you want to run your program with a simple command like shelf, then you should add an entrypoint to your setup.py and then pip install -e . your project. This will generate a suitable wrapper script. A simple setup.py file might look like this:

from setuptools import setup

setup(name='shelf',
      version='0.1.0',
      packages=['shelf'],
      install_requires=[
          'click',  # list dependencies here, can also specify versions
      ],
      entry_points={
          # Instruct setuptools to generate wrapper scripts.
          # This script creates a shelf script that does something like
          # import shelf.__main__; self.__mail__.main()
          # General syntax: name = module : function
          'console_scripts': [
              'shelf=shelf.main:main',
          ],
      },
)
amon
  • 135,795