Mohammad Gufran Jahangir September 30, 2025 0

Here’s a simple line-by-line summary of the important points from the Serverless Compute release notes (09/24/2025):

Perfect 👍 Here’s a one-page cheat sheet table for the Serverless Compute release notes (09/24/2025):

Azure Databricks Serverless Compute – Release Notes Cheat Sheet

VersionDateKey New FeaturesKey Behavior Changes
17.2Sept 25, 2025ST_ExteriorRing (polygon boundary); TEMPORARY metric views; Faster LokiFileSystem on S3; Auto Loader auto-detects partitionsDESCRIBE CONNECTION shows JDBC env; Option to truncate history in Uniform migration; split() fix; Better url_decode handling; Shared env for Python UDTFs
17.1Aug 19, 2025Photon uses less memory on wide schemasInvalid CHECK → error; Pulsar hides Bouncy Castle; Auto Loader uses file events by default; Teradata connector = case-sensitive
Env v4Aug 13, 2025Library & API upgrades–
17.0Jul 24, 2025SQL Procedures; Default collation for SQL funcs; Recursive CTEs; df.mergeInto API; SHOW SCHEMAS ALL CATALOGS; Better liquid clustering; MERGE allows random valuesAuto Loader incremental listing deprecated; CREATE VIEW stricter errors
Perf Targets GAJun 10, 2025Choose Performance optimized (fast) or Standard (cheaper)–
16.4May 28, 2025Auto Loader clean processed files; Type widening (Preview); IDENTIFIER in catalog ops; listagg/string_agg; Python filter pushdown; Better UDF tracebacksCached reads respect options; MERGE must materialize source (if flag enabled)
Perf Mode ConfigApr 14, 2025Jobs can toggle Performance optimized vs Standard mode–
16.3Apr 9, 2025State reader GA; Delta downgrade GA; SQL scripting (Preview); Table/view default collation; H3 funcs; Multi-column ALTER TABLEKafka errors clearer
16.2Mar 13, 2025timestampdiff/timestampadd in generated columns; DESCRIBE TABLE AS JSON; Blank-insensitive collationsDelta Sharing WITH HISTORY default; Credential mismatch → error
High MemoryFeb 7, 2025High-memory notebooks (Preview)–
16.1Feb 5, 2025Recursive Avro; Confluent Schema Registry; OPTIMIZE FULL; Delta identity cols; New SQL funcs (try_url_decode, randstr, etc.); VACUUM LITE; Built-in Oracle JDBCRStudio removed; Type widening restricted; Regex parsing fixed
15.4Oct 28, 2024UTF-8 funcs; Enable Iceberg on existing tables; try_url_decode; RELY on FKs; Faster COPY INTOViews tolerate schema change; Removed ! (NOT) syntax outside boolean; Consistent Base64 errors
JDK 17Aug 15, 2024Upgrade from JDK 8 → JDK 17Regex parsing fixed
15.1Jul 23, 2024Star (*) syntax in WHERE; Better JSON error recovery–
14.3 (Initial)Apr 15, 2024First serverless release; input_file funcs deprecatedSpark config limits; UDF execution improved; CHAR fields padded

General Info

  • Serverless compute is automatically updated by Azure Databricks.
  • Updates bring performance, security, and bug fixes without user code changes.
  • Each environment version fixes Python versions and package sets for stability.

Version 17.2 (Sept 25, 2025)

New features

  • New SQL function ST_ExteriorRing (get polygon outer boundary).
  • Metric views can be TEMPORARY (exist only in the current session).
  • LokiFileSystem on S3 now uses faster native I/O.
  • Auto Loader can auto-detect partition columns in singleVariantColumn mode.

Behavior changes

  • DESCRIBE CONNECTION now shows JDBC environment settings.
  • Option to truncate uniform history when migrating managed tables.
  • split() function fixed for empty regex + positive limit.
  • url_decode and try_url_decode fixed in Photon (return NULL on invalid input).
  • Unity Catalog Python UDTFs share execution environment (STRICT ISOLATION optional).

Version 17.1 (Aug 19, 2025)

New features

  • Photon engine uses less memory for wide schemas.

Behavior changes

  • Invalid CHECK constraints now throw errors.
  • Pulsar connector no longer exposes Bouncy Castle (install manually if needed).
  • Auto Loader now defaults to using file events when available (faster).
  • Teradata connector now defaults to case-sensitive string comparison.

Serverless Environment Version 4 (Aug 13, 2025)

  • Includes library upgrades and API updates.

Version 17.0 (July 24, 2025)

New features

  • SQL procedures supported (CREATE PROCEDURE / CALL).
  • Can set a default collation for SQL functions.
  • Recursive CTEs (rCTEs) supported.
  • PySpark and Spark Connect support df.mergeInto API.
  • SHOW SCHEMAS can list schemas across ALL CATALOGS.
  • Liquid clustering optimizes deletion vectors better.
  • MERGE now allows non-deterministic expressions (e.g., rand()).
  • Python MERGE APIs return DataFrames instead of Unit.

Behavior changes

  • Deprecated Auto Loader option cloudFiles.useIncrementalListing is now always false.
  • CREATE VIEW with materialized-view-only clauses now throws errors.

Serverless Performance Targets GA (June 10, 2025)

  • Performance mode is GA:
    • Performance optimized → faster.
    • Standard → cheaper, slower start.

Version 16.4 (May 28, 2025)

New features

  • Auto Loader can clean processed files (move/delete).
  • Type widening supported in streaming and Delta Sharing (preview).
  • IDENTIFIER supported for catalog operations.
  • listagg / string_agg functions added.
  • Filter pushdown support for Python data sources.
  • Better Python UDF traceback messages.

Behavior changes

  • Cached table reads respect options (not just first query).
  • Source materialization in MERGE is now required (if flag enabled).

Performance Mode Configurable (Apr 14, 2025)

  • Jobs can run in Performance optimized (fast) or Standard mode (cost-saver).

Version 16.3 (Apr 9, 2025)

New features

  • State reader for Structured Streaming is GA.
  • Delta protocol downgrade (DROP FEATURE) is GA.
  • SQL scripting (procedural logic) in Public Preview.
  • Default collation for tables and views.
  • New H3 functions.
  • Can alter multiple columns in one ALTER TABLE.

Behavior changes

  • Better Kafka error messages.

Version 16.2 (Mar 13, 2025)

New features

  • timestampdiff / timestampadd in Delta Lake generated columns.
  • DESCRIBE TABLE AS JSON returns structured metadata.
  • Blank-insensitive collations supported.

Behavior changes

  • Delta Sharing history enabled by default.
  • Credential mismatch now returns errors.

Bug fixes

  • Fix for incremental CLONE re-copying files.

High Memory Setting (Feb 7, 2025)

  • Serverless notebooks can now request higher memory (Public Preview).

Version 16.1 (Feb 5, 2025)

New features (highlights only)

  • Recursive Avro schema support.
  • Confluent Schema Registry support.
  • OPTIMIZE FULL for liquid clustering.
  • Identity columns in Delta via Python APIs.
  • New SQL functions: try_url_decode, zeroifnull, nullifzero, dayname, uniform, randstr.
  • Automatic schema evolution in MERGE.
  • Collations in Spark + Delta (Preview).
  • VACUUM LITE mode.
  • Built-in Oracle JDBC driver.

Behavior changes

  • Hosted RStudio removed.
  • No longer allowed: type widening for integer → decimal/double.

Version 15.4 (Oct 28, 2024)

New features

  • UTF-8 validation functions.
  • Enable Iceberg on existing tables.
  • try_url_decode function.
  • RELY keyword for foreign keys.
  • Parallelized selective overwrites.
  • Faster COPY INTO queries.

Behavior changes

  • Views now tolerate schema changes.
  • Removed support for undocumented ! (NOT) syntax outside boolean.
  • More consistent Base64 error handling.

JDK Upgrade (Aug 15, 2024)

  • Serverless moved from JDK 8 → JDK 17.
  • Regex parsing corrected.

Version 15.1 (July 23, 2024)

  • Star (*) syntax allowed in WHERE clause.
  • JSON parser recovers better from malformed JSON.

Version 14.3 (Apr 15, 2024 – first release)

  • Initial serverless release.
  • Removed support for many Spark configs.
  • input_file functions deprecated.
  • Various fixes (e.g., unhex, auto-aliases, padding CHAR fields).
  • PySpark UDF execution now matches dedicated compute.

Tags:
Category: 
guest
0 Comments
Oldest
Newest Most Voted
Inline Feedbacks
View all comments