Managed Service for Apache Spark runtime 4.0 GA, with Spark 4.2.0 and Java 25

Managed Service for Apache Spark (formerly Google Cloud Serverless for Apache Spark): Runtime version 4.0 is generally available (GA). This version includes the following components:

• Apache Spark 4.2.0
• BigQuery Connector 0.46.0
• Cloud Storage Connector 4.0.7
• Java 25
• Python 3.13
• Scala 2.13.17

Runtime 4.0 also introduces the following changes:

• Batch workloads run in the premium tier and use Lightning Engine by default.
• Memory shapes determine the memory of drivers and executors. The default highmem memory shape provides 7.75 GiB of memory per vCPU.
• Premium tier workloads that use the default driver configuration are eligible to run the driver on a fast-startup VM, which reduces startup time.
• The Lakehouse runtime catalog is the default Spark catalog for batch workloads and interactive sessions.
• Workloads that don't use GPUs consume the Data Compute Units quota instead of Compute Engine CPU and disk quotas.

Runtime 4.0 doesn't support some features of earlier runtimes, such as SparkR batches, Jupyter sessions, and the Persistent History Server. For more information, see Unsupported functionality and behavior changes.