How to Fix Python Module Not Found Error in Crontab: PATH Environment Variable Solutions

You've tested your Python script dozens of times. It works perfectly when you run it manually. But the moment you add it to crontab, boom - ModuleNotFoundError. Your carefully installed packages suddenly don't exist. Sound familiar?

Here's what your crontab entry looks like:

# Crontab entry that fails
* * * * * python3 /home/user/scripts/data_processor.py

And here's the error flooding your logs:

Traceback (most recent call last):
  File "/home/user/scripts/data_processor.py", line 2, in <module>
    import pandas as pd
ModuleNotFoundError: No module named 'pandas'

Step 1: Understanding the Error

Let's reproduce this error systematically. First, create a simple Python script that uses an external module:

#!/usr/bin/env python3
# test_cron.py
import pandas as pd
import numpy as np
from datetime import datetime

print(f"Script running at {datetime.now()}")
data = pd.DataFrame({'values': np.random.rand(5)})
print(f"Data generated: {data.shape}")

When you run this manually:

$ python3 test_cron.py
Script running at 2025-01-21 10:15:30.123456
Data generated: (5, 1)

Perfect. Now add it to crontab:

$ crontab -e
# Add this line
*/5 * * * * python3 /Users/username/test_cron.py >> /Users/username/cron.log 2>&1

Check your log file after 5 minutes:

$ tail -f /Users/username/cron.log
Traceback (most recent call last):
  File "/Users/username/test_cron.py", line 2, in <module>
    import pandas as pd
ModuleNotFoundError: No module named 'pandas'

The script that worked perfectly moments ago now fails completely. The pandas module has vanished into thin air.

Step 2: Identifying the Cause

Cron runs in a minimal environment. It doesn't load your shell configuration files (.bashrc, .zshrc, .bash_profile) that normally set up your PATH, PYTHONPATH, and virtual environment variables.

Let's prove this. Create a diagnostic script:

#!/usr/bin/env python3
# diagnose_env.py
import sys
import os
from datetime import datetime

print(f"=== Environment Diagnosis at {datetime.now()} ===")
print(f"Python executable: {sys.executable}")
print(f"Python version: {sys.version}")
print(f"Python path: {sys.path}")
print(f"PATH variable: {os.environ.get('PATH', 'NOT SET')}")
print(f"PYTHONPATH: {os.environ.get('PYTHONPATH', 'NOT SET')}")
print(f"Virtual env: {os.environ.get('VIRTUAL_ENV', 'NOT SET')}")

Run it manually first:

$ python3 diagnose_env.py
=== Environment Diagnosis at 2025-01-21 10:20:15.123456 ===
Python executable: /Users/username/.pyenv/versions/3.11.0/bin/python3
Python version: 3.11.0 (main, Oct 24 2022, 18:26:48)
Python path: ['/Users/username', '/Users/username/.pyenv/versions/3.11.0/lib/python311.zip', ...]
PATH variable: /Users/username/.pyenv/shims:/usr/local/bin:/usr/bin:/bin:/usr/sbin:/sbin
PYTHONPATH: /Users/username/projects/lib
Virtual env: NOT SET

Now add it to crontab and compare:

# Crontab entry
* * * * * python3 /Users/username/diagnose_env.py >> /Users/username/diagnose.log 2>&1

The cron output reveals the problem:

=== Environment Diagnosis at 2025-01-21 10:21:00.123456 ===
Python executable: /usr/bin/python3
Python version: 3.9.6 (default, May  7 2023, 23:32:44)
Python path: ['/Users/username', '/Library/Python/3.9/site-packages', ...]
PATH variable: /usr/bin:/bin
PYTHONPATH: NOT SET
Virtual env: NOT SET

Notice the differences? Cron is using the system Python (/usr/bin/python3) instead of your pyenv or virtual environment Python. The PATH is minimal, PYTHONPATH is missing entirely.

Step 3: Implementing the Solution

Solution 1: Set Environment Variables in Crontab

The quickest fix is to define environment variables at the top of your crontab file:

# Edit crontab
$ crontab -e

# Add these lines at the top
SHELL=/bin/bash
PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/Users/username/.pyenv/shims
PYTHONPATH=/Users/username/projects/lib:/Users/username/.local/lib/python3.11/site-packages

# Now your cron job
*/5 * * * * python3 /Users/username/scripts/data_processor.py >> /Users/username/cron.log 2>&1

This approach works but has limitations. You're still not activating virtual environments properly, and the PATH might differ between development environments.

Solution 2: Use Absolute Python Path

Instead of relying on PATH resolution, specify the exact Python executable:

# For system Python with user-installed packages
*/5 * * * * /usr/bin/python3 /Users/username/scripts/data_processor.py

# For pyenv Python
*/5 * * * * /Users/username/.pyenv/versions/3.11.0/bin/python /Users/username/scripts/data_processor.py

# For virtual environment Python
*/5 * * * * /Users/username/venv/bin/python /Users/username/scripts/data_processor.py

This method is more reliable but still doesn't handle complex environment setups.

Solution 3: Create a Wrapper Shell Script

The most robust solution is creating a shell script that sets up your entire environment:

#!/bin/bash
# run_python_script.sh

# Load shell profile to get all environment variables
source /Users/username/.bash_profile

# Or if using zsh on macOS
# source /Users/username/.zshrc

# Activate virtual environment if needed
source /Users/username/projects/myproject/venv/bin/activate

# Set any additional environment variables
export PYTHONPATH="/Users/username/projects/lib:$PYTHONPATH"
export MY_API_KEY="your-api-key-here"

# Log the environment for debugging
echo "=== Cron Job Started at $(date) ==="
echo "Python: $(which python3)"
echo "PATH: $PATH"

# Run your Python script
python3 /Users/username/scripts/data_processor.py

# Log completion
echo "=== Cron Job Completed at $(date) ==="

Make the script executable:

$ chmod +x /Users/username/scripts/run_python_script.sh

Then in crontab, simply call this wrapper:

*/5 * * * * /bin/bash /Users/username/scripts/run_python_script.sh >> /Users/username/cron.log 2>&1

Solution 4: Using bash -lc for Login Shell

Another approach is forcing cron to use a login shell, which loads your profile:

# This loads your .bash_profile or .zprofile
*/5 * * * * /bin/bash -lc 'python3 /Users/username/scripts/data_processor.py' >> /Users/username/cron.log 2>&1

The -l flag makes bash act as a login shell, loading your usual environment setup. The -c flag runs the command string that follows.

Solution 5: Handling Virtual Environments

For virtualenv or venv environments, you have several options:

Option A: Direct activation in crontab

*/5 * * * * source /Users/username/myproject/venv/bin/activate && python /Users/username/scripts/data_processor.py

Option B: Use the virtual environment's Python directly

*/5 * * * * /Users/username/myproject/venv/bin/python /Users/username/scripts/data_processor.py

Option C: For conda environments

# First, find your conda path
$ which conda
/Users/username/miniconda3/bin/conda

# In crontab, activate conda environment
*/5 * * * * /Users/username/miniconda3/bin/conda run -n myenv python /Users/username/scripts/data_processor.py

Or with explicit activation:

#!/bin/bash
# conda_wrapper.sh
source /Users/username/miniconda3/etc/profile.d/conda.sh
conda activate myenv
python /Users/username/scripts/data_processor.py

Solution 6: For pyenv Users

If you're using pyenv, the wrapper script approach works best:

#!/bin/bash
# pyenv_cron_wrapper.sh

# Initialize pyenv
export PYENV_ROOT="$HOME/.pyenv"
export PATH="$PYENV_ROOT/bin:$PATH"
eval "$(pyenv init --path)"
eval "$(pyenv init -)"

# Set the Python version
pyenv shell 3.11.0

# Or use pyenv local if you have a .python-version file
cd /Users/username/projects/myproject
pyenv local 3.11.0

# Run your script
python /Users/username/scripts/data_processor.py

Working Code Example

Here's a complete, production-ready setup that handles all common scenarios:

#!/usr/bin/env python3
# robust_cron_script.py
import sys
import os
import logging
from datetime import datetime

# Set up logging to track execution
logging.basicConfig(
    level=logging.INFO,
    format='%(asctime)s - %(levelname)s - %(message)s',
    handlers=[
        logging.FileHandler('/Users/username/logs/cron_script.log'),
        logging.StreamHandler()
    ]
)

def main():
    """Main function with error handling"""
    try:
        # Log environment details
        logging.info(f"Script started with Python {sys.version}")
        logging.info(f"Python executable: {sys.executable}")
        
        # Import modules with error handling
        try:
            import pandas as pd
            import numpy as np
            logging.info("Successfully imported required modules")
        except ImportError as e:
            logging.error(f"Module import failed: {e}")
            logging.error(f"Python path: {sys.path}")
            raise
        
        # Your actual script logic
        data = pd.DataFrame({
            'timestamp': [datetime.now()],
            'value': [np.random.rand()]
        })
        
        # Save results
        output_path = '/Users/username/data/output.csv'
        data.to_csv(output_path, mode='a', header=not os.path.exists(output_path))
        logging.info(f"Data saved to {output_path}")
        
    except Exception as e:
        logging.error(f"Script failed: {e}", exc_info=True)
        sys.exit(1)
    
    logging.info("Script completed successfully")

if __name__ == "__main__":
    main()

And the corresponding wrapper script:

#!/bin/bash
# cron_wrapper.sh

# Exit on error
set -e

# Set up logging
LOG_FILE="/Users/username/logs/cron_wrapper.log"
exec 1>> "$LOG_FILE"
exec 2>&1

echo "========================================="
echo "Cron job started at $(date)"
echo "========================================="

# Load environment
source /Users/username/.bash_profile

# Activate virtual environment
source /Users/username/projects/venv/bin/activate

# Set Python path
export PYTHONPATH="/Users/username/projects/lib:$PYTHONPATH"

# Debug information
echo "Python location: $(which python3)"
echo "Python version: $(python3 --version)"
echo "Pip list (first 5 packages):"
pip list | head -5

# Run the Python script
python3 /Users/username/scripts/robust_cron_script.py

echo "Cron job completed at $(date)"

Finally, the crontab entry:

# Run every hour
0 * * * * /bin/bash /Users/username/scripts/cron_wrapper.sh

Additional Tips & Related Errors

Debugging Cron Jobs

Always redirect both stdout and stderr to a log file:

*/5 * * * * /path/to/script.py >> /path/to/logfile.log 2>&1

The 2>&1 redirects stderr (file descriptor 2) to stdout (file descriptor 1), capturing all output.

Common Related Errors

PermissionError: Make your scripts executable:

$ chmod +x /path/to/your/script.py

No such file or directory: Always use absolute paths in crontab:

# Wrong
*/5 * * * * python3 ~/scripts/myscript.py

# Right
*/5 * * * * python3 /Users/username/scripts/myscript.py

ImportError with C extensions: Some Python packages with C extensions need additional library paths:

# In your wrapper script
export LD_LIBRARY_PATH="/usr/local/lib:$LD_LIBRARY_PATH"
export DYLD_LIBRARY_PATH="/usr/local/lib:$DYLD_LIBRARY_PATH"  # macOS specific

Testing Your Cron Environment

Create this test script to verify your cron environment matches your development environment:

#!/usr/bin/env python3
# test_cron_env.py
import subprocess
import sys

def test_module_import(module_name):
    """Test if a module can be imported"""
    try:
        __import__(module_name)
        return f"✓ {module_name}"
    except ImportError:
        return f"✗ {module_name}"

# Test common modules
modules = ['pandas', 'numpy', 'requests', 'matplotlib', 'scipy']
print("Module availability:")
for mod in modules:
    print(test_module_import(mod))

# Test pip list
try:
    result = subprocess.run([sys.executable, '-m', 'pip', 'list'], 
                          capture_output=True, text=True)
    print(f"\nInstalled packages: {len(result.stdout.splitlines())} found")
except Exception as e:
    print(f"Could not list packages: {e}")

Platform-Specific Considerations

On macOS, you might encounter SIP (System Integrity Protection) issues. The system Python at /usr/bin/python3has restrictions. Using homebrew Python or pyenv avoids these issues:

# Install Python via homebrew
$ brew install python@3.11

# Use homebrew Python in crontab
*/5 * * * * /opt/homebrew/bin/python3 /path/to/script.py

On Linux systems, the approach is similar but paths differ:

# Typical Linux paths
PATH=/usr/local/sbin:/usr/local/bin:/sbin:/bin:/usr/sbin:/usr/bin
# Virtual environment might be at
/home/username/venv/bin/python

Remember that cron's environment is intentionally minimal for security. Every path, every module, every environment variable you rely on needs explicit configuration. The wrapper script approach gives you the most control and makes debugging easier when things go wrong.

Testing is crucial. Before adding any script to crontab, run it through the wrapper script manually to ensure all dependencies are properly configured. Your future self will thank you when that 3 AM cron job actually runs successfully.

sCoding

Search This Blog