Step 1: Understanding the Error
You're working on a Python project with multiple packages, and suddenly you encounter this frustrating error:
$ python -m myapp.main
Traceback (most recent call last):
File "/usr/local/lib/python3.9/runpy.py", line 197, in _run_module_as_main
return _run_code(code, main_globals, None,
File "/usr/local/lib/python3.9/runpy.py", line 87, in _run_code
exec(code, run_globals)
File "/Users/dev/project/myapp/main.py", line 2, in <module>
from myapp.utils import helper
ImportError: cannot import name 'helper' from 'myapp.utils' (unknown location)
Or perhaps you're seeing this variation:
$ python setup.py install
error: package directory 'myapp/plugins' does not exist
These errors typically occur when Python's namespace package system gets confused about your package structure. Namespace packages (PEP 420) allow you to split a single Python package across multiple directories or distributions, but they're notorious for breaking in subtle ways.
Let's reproduce a common namespace package error:
# Project structure that causes the error
project/
├── myapp/
│ ├── core/
│ │ └── engine.py
│ └── main.py
├── plugins/
│ └── myapp/
│ └── plugins/
│ └── custom.py
└── setup.py
# myapp/main.py
from myapp.core import engine # This works
from myapp.plugins import custom # This fails!
def run():
print("Starting application")
engine.start()
custom.load()
if __name__ == "__main__":
run()
Running this code produces:
$ python -m myapp.main
ImportError: No module named 'myapp.plugins'
Step 2: Identifying the Cause
Namespace package errors occur for several reasons. The most common culprits are:
Missing __init__.py files in regular packages. When Python can't determine if a directory should be treated as a namespace package or regular package, import errors occur. This confusion happens because PEP 420 introduced implicit namespace packages that don't require __init__.py files.
Mixed package styles in the same project. You can't mix regular packages (with __init__.py) and namespace packages (without __init__.py) at the same level in your package hierarchy.
Incorrect sys.path configuration. Python needs to know where to look for your packages. If the parent directories aren't in sys.path, namespace packages won't be discovered properly.
Let's diagnose the issue with some debugging code:
# debug_namespace.py
import sys
import pkgutil
import importlib.util
# Check what Python sees as packages
print("Python path entries:")
for path in sys.path:
print(f" {path}")
# Check if namespace packages are detected
print("\nNamespace packages found:")
for importer, modname, ispkg in pkgutil.iter_modules():
if ispkg and '.' not in modname:
spec = importlib.util.find_spec(modname)
if spec and not spec.origin: # No origin means namespace package
print(f" {modname}: {spec.submodule_search_locations}")
# Try to find our problematic package
print("\nSearching for 'myapp' package:")
try:
spec = importlib.util.find_spec('myapp')
print(f" Found: {spec}")
print(f" Locations: {spec.submodule_search_locations}")
except ImportError as e:
print(f" Not found: {e}")
Running this diagnostic reveals the problem:
$ python debug_namespace.py
Python path entries:
/Users/dev/project
/usr/local/lib/python3.9/site-packages
Namespace packages found:
myapp: ['/Users/dev/project/myapp']
Searching for 'myapp' package:
Found: ModuleSpec(name='myapp', loader=None, origin=None, submodule_search_locations=['/Users/dev/project/myapp'])
Locations: ['/Users/dev/project/myapp']
The plugins directory isn't being recognized as part of the namespace package because it's in a different location.
Step 3: Implementing the Solution
Solution 1: Convert to Proper Namespace Package Structure
The cleanest fix is restructuring your project to follow PEP 420 namespace package conventions:
# Corrected project structure
project/
├── src/
│ └── myapp/ # No __init__.py here for namespace package
│ ├── core/
│ │ ├── __init__.py
│ │ └── engine.py
│ └── main.py
├── plugins/
│ └── myapp/ # No __init__.py here either
│ └── plugins/
│ ├── __init__.py
│ └── custom.py
└── setup.py
Update your setup.py to handle namespace packages correctly:
# setup.py
from setuptools import setup, find_namespace_packages
setup(
name='myapp',
version='1.0.0',
# Use find_namespace_packages instead of find_packages
packages=find_namespace_packages(where='src'),
package_dir={'': 'src'},
# Include plugin packages separately if needed
namespace_packages=['myapp'], # Declare namespace package
python_requires='>=3.3', # PEP 420 requires Python 3.3+
)
Now modify your import paths:
# Fix sys.path to include both directories
import sys
import os
# Add both source roots to Python path
project_root = os.path.dirname(os.path.dirname(os.path.abspath(__file__)))
sys.path.insert(0, os.path.join(project_root, 'src'))
sys.path.insert(0, os.path.join(project_root, 'plugins'))
# Now imports work correctly
from myapp.core import engine
from myapp.plugins import custom
Solution 2: Using pkgutil-style Namespace Packages
For backward compatibility with Python 2, use pkgutil-style namespace packages:
# src/myapp/__init__.py
# This line declares a namespace package
__path__ = __import__('pkgutil').extend_path(__path__, __name__)
# plugins/myapp/__init__.py
# Same declaration in the plugins directory
__path__ = __import__('pkgutil').extend_path(__path__, __name__)
This approach requires __init__.py files but makes them namespace-aware. Here's a complete working example:
# working_namespace_example.py
import sys
import os
# Setup paths for demonstration
base_dir = os.path.dirname(os.path.abspath(__file__))
sys.path.insert(0, os.path.join(base_dir, 'src'))
sys.path.insert(0, os.path.join(base_dir, 'plugins'))
# Create the package structure programmatically for testing
import tempfile
import shutil
def setup_namespace_package():
"""Create a working namespace package structure"""
temp_dir = tempfile.mkdtemp()
# Create src/myapp structure
src_myapp = os.path.join(temp_dir, 'src', 'myapp')
os.makedirs(src_myapp)
# Create namespace __init__.py
with open(os.path.join(src_myapp, '__init__.py'), 'w') as f:
f.write("__path__ = __import__('pkgutil').extend_path(__path__, __name__)\n")
# Create core module
core_dir = os.path.join(src_myapp, 'core')
os.makedirs(core_dir)
with open(os.path.join(core_dir, '__init__.py'), 'w') as f:
f.write("")
with open(os.path.join(core_dir, 'engine.py'), 'w') as f:
f.write("def start():\n print('Engine started')\n")
# Create plugins/myapp structure
plugins_myapp = os.path.join(temp_dir, 'plugins', 'myapp')
os.makedirs(plugins_myapp)
# Create namespace __init__.py
with open(os.path.join(plugins_myapp, '__init__.py'), 'w') as f:
f.write("__path__ = __import__('pkgutil').extend_path(__path__, __name__)\n")
# Create plugins module
plugins_dir = os.path.join(plugins_myapp, 'plugins')
os.makedirs(plugins_dir)
with open(os.path.join(plugins_dir, '__init__.py'), 'w') as f:
f.write("")
with open(os.path.join(plugins_dir, 'custom.py'), 'w') as f:
f.write("def load():\n print('Custom plugin loaded')\n")
return temp_dir
# Test the solution
temp_dir = setup_namespace_package()
sys.path.insert(0, os.path.join(temp_dir, 'src'))
sys.path.insert(0, os.path.join(temp_dir, 'plugins'))
# Now imports work!
from myapp.core import engine
from myapp.plugins import custom
engine.start() # Output: Engine started
custom.load() # Output: Custom plugin loaded
# Cleanup
shutil.rmtree(temp_dir)
Solution 3: Using PYTHONPATH Environment Variable
For development environments, you can set PYTHONPATH to include all namespace package locations:
# Set multiple directories in PYTHONPATH
$ export PYTHONPATH="/Users/dev/project/src:/Users/dev/project/plugins:$PYTHONPATH"
# Now run your Python script
$ python -m myapp.main
Starting application
Engine started
Custom plugin loaded
Create a development script to automate this:
# run_dev.py
#!/usr/bin/env python
import os
import sys
import subprocess
# Get the project root
project_root = os.path.dirname(os.path.abspath(__file__))
# Build PYTHONPATH with all namespace package locations
namespace_dirs = [
os.path.join(project_root, 'src'),
os.path.join(project_root, 'plugins'),
os.path.join(project_root, 'extensions'), # Add more as needed
]
# Set environment variable
env = os.environ.copy()
env['PYTHONPATH'] = ':'.join(namespace_dirs) + ':' + env.get('PYTHONPATH', '')
# Run the main application
subprocess.run([sys.executable, '-m', 'myapp.main'], env=env)
Step 4: Working Code Example
Here's a complete, working namespace package setup that handles common edge cases:
# complete_namespace_solution.py
"""
Complete solution for namespace package import errors
Handles PEP 420 namespace packages with multiple distributions
"""
import os
import sys
import importlib.util
from pathlib import Path
class NamespacePackageManager:
"""Manages namespace packages across multiple directories"""
def __init__(self, package_name):
self.package_name = package_name
self.search_paths = []
def add_search_path(self, path):
"""Add a directory to search for namespace package components"""
path = Path(path).resolve()
if path.exists() and path not in self.search_paths:
self.search_paths.append(path)
# Add to sys.path if not already there
str_path = str(path)
if str_path not in sys.path:
sys.path.insert(0, str_path)
def validate_namespace_package(self):
"""Check if namespace package is properly configured"""
issues = []
for search_path in self.search_paths:
package_path = search_path / self.package_name
if not package_path.exists():
issues.append(f"Package directory missing: {package_path}")
continue
init_file = package_path / '__init__.py'
# Check for mixed package styles
if init_file.exists():
content = init_file.read_text()
# Check if it's a proper namespace package init
if '__path__' not in content and 'pkgutil' not in content:
issues.append(
f"Regular package found at {package_path}. "
f"For namespace packages, either remove __init__.py "
f"or add namespace declaration"
)
return issues
def fix_imports(self):
"""Attempt to fix namespace package imports"""
# For PEP 420 namespace packages (no __init__.py)
for search_path in self.search_paths:
package_path = search_path / self.package_name
init_file = package_path / '__init__.py'
if package_path.exists() and not init_file.exists():
# This is a PEP 420 namespace package component
print(f"Found PEP 420 namespace component: {package_path}")
# Force reload of the package to pick up all components
if self.package_name in sys.modules:
del sys.modules[self.package_name]
# Try to import and check what was found
try:
spec = importlib.util.find_spec(self.package_name)
if spec and spec.submodule_search_locations:
print(f"Namespace package '{self.package_name}' configured successfully")
print(f"Search locations: {spec.submodule_search_locations}")
return True
except ImportError as e:
print(f"Failed to configure namespace package: {e}")
return False
# Example usage
def setup_and_test_namespace_packages():
"""Demonstration of fixing namespace package issues"""
# Initialize the manager
manager = NamespacePackageManager('myapp')
# Add all directories containing parts of the namespace package
project_root = Path(__file__).parent
manager.add_search_path(project_root / 'src')
manager.add_search_path(project_root / 'plugins')
manager.add_search_path(project_root / 'extensions')
# Validate the setup
issues = manager.validate_namespace_package()
if issues:
print("Found issues with namespace package setup:")
for issue in issues:
print(f" - {issue}")
# Attempt to fix
print("\nAttempting to fix imports...")
if manager.fix_imports():
print("✓ Namespace package fixed successfully")
else:
print("✗ Could not fix namespace package automatically")
print("\nManual fix required:")
print("1. Remove all __init__.py files from namespace package roots")
print("2. Or add namespace declaration to existing __init__.py files")
print("3. Ensure all parent directories are in sys.path")
else:
print("✓ No namespace package issues detected")
manager.fix_imports()
if __name__ == "__main__":
setup_and_test_namespace_packages()
Step 5: Additional Tips & Related Errors
Watch out for these common namespace package pitfalls:
ImportError with "No module named" for nested packages. This happens when parent directories aren't in sys.path:
# This structure causes issues
company/
backend/
myapp/
api/
# Fix: Add company/backend to sys.path, not just company/backend/myapp
sys.path.insert(0, '/path/to/company/backend')
AttributeError when importing from namespace packages. Sometimes the package is found but submodules aren't:
# Instead of this (which might fail)
import myapp
myapp.plugins.custom.load() # AttributeError: 'module' object has no attribute 'plugins'
# Do this (explicit imports)
from myapp.plugins import custom
custom.load()
Editable installs breaking namespace packages. When using pip install -e ., namespace packages might not work correctly:
# This might break namespace packages
$ pip install -e .
# Better approach for development
$ pip install -e . --no-build-isolation
# Or use setup.py develop
$ python setup.py develop
Test your namespace package configuration with this diagnostic script:
# test_namespace_health.py
import sys
import pkgutil
import importlib
def check_namespace_package_health(package_name):
"""Comprehensive namespace package health check"""
print(f"Checking namespace package: {package_name}")
print("=" * 50)
# Check if package is importable
try:
pkg = importlib.import_module(package_name)
print(f"✓ Package {package_name} is importable")
# Check package path
if hasattr(pkg, '__path__'):
print(f" Package paths: {list(pkg.__path__)}")
# Check for namespace package indicators
if not hasattr(pkg, '__file__'):
print(f" ✓ Confirmed as namespace package (no __file__ attribute)")
else:
print(f" ⚠ May be regular package (__file__ = {pkg.__file__})")
# List all submodules
print(f"\n Submodules found:")
for importer, modname, ispkg in pkgutil.walk_packages(
pkg.__path__,
prefix=package_name + "."
):
pkg_type = "package" if ispkg else "module"
print(f" - {modname} ({pkg_type})")
except ImportError as e:
print(f"✗ Cannot import {package_name}: {e}")
print("\n Debugging steps:")
print(" 1. Check sys.path includes parent directories")
print(" 2. Verify no __init__.py in namespace package root")
print(" 3. Ensure Python 3.3+ for PEP 420 support")
# Run the health check
check_namespace_package_health('myapp')
Remember that namespace packages require Python 3.3 or later for PEP 420 support. For Python 2 compatibility, stick with pkgutil-style namespace packages. When distributing namespace packages, each component can be installed separately, making them ideal for plugin systems and large modular applications.