Ray
Ray is an open-source framework for scaling Python applications, including distributed compute, machine learning training, and hyperparameter tuning, across clusters of machines.
Overview
Ray is a distributed execution framework for Python that makes it straightforward to scale arbitrary Python functions and classes across a cluster. At its core, Ray provides simple primitives — @ray.remote tasks and actors — for turning ordinary Python functions and objects into units of work that run in parallel across many CPUs or nodes. On top of that core, the Ray ecosystem includes higher-level libraries: Ray Train for distributed model training, Ray Tune for hyperparameter search, Ray Serve for model serving, and Ray Data for distributed data loading and preprocessing.
Basic usage
Copy code
import ray
ray.init()
@ray.remote
def process(x):
return x * x
futures = [process.remote(i) for i in range(10)]
results = ray.get(futures)
Ray handles scheduling these remote calls across available workers, whether that's cores on your laptop or hundreds of nodes in a cluster.
Where Ray fits versus a query engine
Ray is a general-purpose distributed computing framework, not a SQL engine or a database. It's typically used for machine learning workloads: distributed training, large-scale batch inference, hyperparameter tuning, and reinforcement learning. It differs from analytical engines like DuckDB, which are purpose-built for fast, vectorized SQL and dataframe queries over structured data on a single node. In practice these tools are often complementary in a pipeline: DuckDB (or a warehouse) can be used to select, join, and filter down a large dataset into the feature set actually needed for training, and Ray then takes over to distribute the compute-heavy model training or inference step across a cluster.
Related terms
Apache Spark is an open-source distributed processing engine for large-scale data workloads, using in-memory computation across a cluster of machines.
Dask →Dask is a Python library for parallel and distributed computing that scales pandas-like DataFrame, NumPy-like array, and general task-graph workloads across multiple cores or a cluster.
Apache Hadoop →Apache Hadoop is an open-source framework for distributed storage and processing of very large datasets across clusters of commodity servers, built around HDFS for storage and MapReduce (or later, YARN-managed engines) for computation.
Python →Python is a high-level, interpreted programming language known for its simplicity and readability.
FAQS
Primarily machine learning and general distributed Python compute — training, tuning, and serving models — rather than SQL-style analytics. Ray Data adds distributed data loading, but it's not a query engine in the way a database is.
No. ray.init() with no arguments starts a local Ray instance that parallelizes across the cores of your own machine, and the same code scales to a real cluster later without changes.
