Luigi

What Luigi is

Luigi is a free Python package for building batch-job pipelines and managing task dependencies, workflow execution, and failures. Workflows and dependency graphs are defined in Python, though tasks can also trigger work outside Python. The package supports command-line use and use from Python code. Its file abstractions cover HDFS and local storage, with atomic operations intended to prevent pipelines from being left with partial data.

The documentation describes task support for Hadoop, Hive, Pig, and Spark. Luigi’s server provides a web interface for searching and filtering tasks and viewing dependency graphs. It can manage pipelines with thousands of tasks that take days or weeks, including Hadoop jobs, database transfers, and machine-learning algorithms. The latest stable version installs with `pip install luigi`; adding `[toml]` provides TOML configuration support. Documentation lists Python 3.10, 3.11, 3.12, and 3.13 as tested versions. Luigi is self-hosted, and Spotify’s Data Team maintains it. A mailing list is available for discussions and questions.