Pathway Live Data Framework

What Pathway Live Data Framework is

Pathway Live Data Framework is a Python framework for building analytics and AI data pipelines over stored data and live streams. The same pipeline can handle batch data and streaming ETL or retrieval-augmented generation (RAG) workflows. Its Rust engine runs Python code with multithreading and multiprocessing, while Differential Dataflow supports real-time handling of out-of-order data. AI tooling includes online machine learning, RAG pipelines, and vector indexes for fresh data. Documentation lists more than 350 connectors, including SharePoint, and users can implement their own. Pathway provides Python and SQL programming APIs, plus a REST endpoint for serving queries, answers, and real-time features. Deployments can use Docker or Kubernetes, with OpenTelemetry compatibility, and the framework can save computation state to resume after a failure or pipeline update. The free Community tier is self-hosted and supports up to 8 GB RAM and 4 cores. Scale is self-hosted, requires a license key, and supports up to 16 GB RAM and 4 cores; it can be free or paid. Enterprise licensing is available, with self-hosted or managed deployment.