Sail OSS and the LakeSail Platform

LakeSail is building around a direct premise: keep the Spark developer interface, but replace the runtime beneath it. The open-source core is Sail, a Rust-native compute engine designed to run Spark-compatible workloads without the operational and performance costs that often come with JVM-based Spark clusters.

That is a sharper claim than "Spark accelerator." Spark is not only a query engine; it is an ecosystem interface. PySpark jobs, Spark SQL, notebooks, catalogs, schedulers, and years of data engineering muscle memory all sit above the runtime. Sail's bet is that teams can keep that interface while moving execution onto a modern, Arrow-native, Rust-based engine.

What Sail OSS Is

Sail is an Apache 2.0 open-source project from LakeSail. The project describes Sail as a multimodal compute engine for single-host and distributed execution, with a mission to unify batch processing, stream processing, and compute-intensive AI workloads.

The key compatibility choice is Spark Connect. Sail runs as a Spark Connect server, so a PySpark client can submit work over gRPC and receive results through the familiar Spark client path. In practice, the migration story is not "rewrite your pipelines for a new engine"; it is closer to "point your existing Spark client at a new endpoint."

spark = (
    SparkSession.builder
    .remote("sc://sail-host:50051")
    .getOrCreate()
)

LakeSail's technical deck frames this as the protocol boundary that lets Sail replace the whole engine. The client still owns the API surface; Sail decodes the Spark Connect protobuf plan and performs planning and execution in Rust.

Why Replace the Engine

LakeSail's argument is that JVM-era Spark hits ceilings that are structural rather than incidental. Short queries can be dominated by cluster startup, garbage collection, and JVM memory behavior. Python UDFs run out of process, which means batches cross serialization and IPC boundaries. AI and ML workloads often live in Python while the core execution engine lives elsewhere, so the cost of crossing that boundary becomes a core design problem.

Sail takes the replacement path: Rust-native execution end to end, columnar data through Apache Arrow, query planning and execution through Apache DataFusion, async runtime support through Tokio, and embedded CPython through PyO3 for Python UDFs.

This is where Sail differs from projects that accelerate Spark by offloading pieces of the physical plan to native code. LakeSail's technical materials contrast that approach with replacing the JVM runtime outright. The intended benefit is not just faster operators; it is removing the repeated row/column conversion, serialization, and out-of-process Python costs that sit around those operators.

Python UDFs Are the Tell

Python is where Sail's design becomes most visible. In traditional Spark, Python UDF execution crosses from the JVM to a separate Python worker and back. Each batch must be serialized, shipped, deserialized, executed, reserialized, and returned. That path is especially painful for feature engineering, embeddings, inference, tokenization, vectorization, and custom transforms.

Sail embeds CPython in the engine with PyO3. The technical deck describes UDFs as running in process, sharing Arrow memory with the Rust engine. That lets Sail treat Python UDFs as part of the execution plan rather than as an external worker boundary.

That does not mean every Python workload becomes free. Model loading, network calls, vectorization choices, and Python code quality still matter. But it changes the baseline cost model: Python becomes an integrated execution path instead of a second-class side channel.

How Sail Runs

Sail has two execution modes.

In local mode, Sail runs as a single process. A PySpark client connects to the Sail server over Spark Connect, and a local job runner executes the optimized physical plan using available CPU cores.

In cluster mode, the Sail server coordinates distributed work across Sail workers. A driver schedules stages and tasks; workers execute partitions; control-plane communication uses Sail's internal gRPC protocol; and the data plane uses Arrow Flight for shuffle data and result transfer.

The technical deck adds a useful detail about the control plane: Sail uses an actor model on Tokio. Drivers and workers process messages through single-threaded event loops, keeping state owned by actors rather than guarded through shared locks. The runtime organizes work as jobs, stages, tasks, and attempts, with task regions used as scheduling and failure-recovery units.

Lakehouse Formats Are Engine Work

Sail is also designed for open lakehouse tables rather than plain file listings alone. LakeSail highlights native Apache Iceberg and Delta Lake support, with catalogs including AWS Glue, Iceberg REST, Unity Catalog, OneLake, Hive Metastore, and in-memory options.

The deck makes the architectural point clearly: lakehouse formats require deep engine integration. Listing tables can infer schema and partitions from files and directories. Lakehouse tables use manifests, metadata logs, table statistics, atomic writes, schema evolution, and DML operations such as MERGE, UPDATE, and DELETE.

Sail's Delta Lake implementation uses custom logical and physical plan nodes for DML. The MERGE planning pipeline moves from SQL or Spark Connect protobuf into Sail's internal representation, then through custom logical extensions, optimizer rules, physical planning, and a final optimized physical plan.

That matters because table formats are not only storage metadata. They affect pruning, planning, conflict handling, commit behavior, and observability. Sail's position is that those concerns belong inside the engine, not only in a connector wrapper around it.

Observability Through SQL

One useful Sail design choice is the system catalog. The technical deck shows internal execution state exposed through SQL tables such as sessions, jobs, stages, tasks, and workers. A query like SELECT * FROM system.session.sessions WHERE status = 'RUNNING' can inspect runtime state through the same declarative interface used for data.

This complements OpenTelemetry metrics, traces, and logs. It also fits the larger Sail theme: keep operational and execution concepts close to the query engine instead of scattering them across unrelated tools.

Benchmarks: Useful, but Directional

LakeSail publishes benchmark claims that are worth treating as directional rather than universal. In the Sail docs, a derived TPC-H benchmark on 100 GB of Parquet data reports total query time of 102.75 seconds for Sail versus 387.36 seconds for Spark, along with lower peak memory use and no shuffle spill to disk. LakeSail summarizes that result as nearly 4x faster execution and up to 94% cost reduction when Sail can run on smaller infrastructure.

The technical deck also presents per-query speedups, resource utilization comparisons, and ClickBench comparisons against Spark, Comet, and Databricks. Those results support the same thesis: avoid JVM overhead, reduce memory pressure, and eliminate shuffle spill for workloads that fit Sail's execution strengths.

The caveat is the usual one: benchmark shape matters. File format, cluster sizing, hot versus cold data, query mix, UDF behavior, catalog latency, and production reliability requirements all affect whether the published results translate. The right reading is not "Sail is always faster"; it is "Sail changes enough of the runtime architecture that real Spark workloads are worth testing against it."

The LakeSail Platform

Sail OSS is the engine. LakeSail Platform is the managed product for running that engine in production.

The platform is deployed into the customer's AWS account using a bring-your-own-cloud model. LakeSail says it provisions Kubernetes clusters and Karpenter-driven compute inside a network it creates in that account, while credentials remain in the customer's account. On top of Sail, the platform adds the operational pieces most teams need before they trust a compute engine with production workloads:

• managed cluster lifecycle;
• scheduled and on-demand SQL or Python jobs;
• interactive Spark Connect sessions;
• notebooks;
• catalog connections;
• identity, teams, roles, SSO, MFA, and fine-grained access policies;
• notifications for job failures;
• billing, monitoring, and observability.

LakeSail's product page describes three ways to run Sail:

1. Sail OSS: self-host the Apache 2.0 engine with community support.
2. LakeSail Platform: managed Sail in your AWS account, with upgrades, monitoring, autoscaling, scheduling, and per-second billing.
3. Enterprise: platform capabilities plus enterprise support, SSO, custom licensing, and private deployment options.

That separation is clean: open source for teams that want direct control of the engine, platform for teams that want the Spark replacement without owning Kubernetes operations, autoscaling, scheduling, identity, and observability themselves.

The Practical Migration Story

The most compelling LakeSail message is operational continuity. The product copy repeatedly emphasizes zero rewrites: keep PySpark and Spark SQL, update the Spark Connect endpoint, and run the same workloads against Sail.

A minimal local trial looks like this:

pip install "pysail==0.6.6"
sail spark server --port 50051

Then existing Spark Connect code can target:

sc://127.0.0.1:50051

For production, the decision becomes whether to self-host Sail on your own infrastructure or use LakeSail Platform to handle Kubernetes, autoscaling, scheduling, access control, billing, and observability.

Takeaway

Sail is best understood as a Rust-native Spark replacement with an open-source engine and a managed platform around it. The OSS project is aimed at compatibility, performance, lakehouse execution, and Python-native AI workloads. The LakeSail Platform is aimed at production operations: running Sail in a customer's cloud account with managed clusters, jobs, sessions, catalogs, identity, billing, and monitoring.

For Spark-heavy teams, the appeal is obvious. If Sail's compatibility holds for a real workload, it offers a migration path that changes the engine before it changes the codebase. That is a pragmatic wedge into one of the hardest parts of data infrastructure: improving runtime economics without forcing every pipeline owner to relearn their tools.

Sources

• LakeSail homepage
• LakeSail product overview
• LakeSail Platform introduction
• Sail documentation
• Sail architecture
• Sail installation
• Sail benchmark results
• ~/storage/downloads/Sail Technical.pdf

Headerboard image credit: Angel Alvares Pascua

published with omnighost · SHA-256 cc7c659402d95739f642e7cc3c4636b4a1c14bc62461227ad3aca0c0ad1e2f5d