Best practices for serverless compute
Big picture
- Serverless can make your work faster, cheaper, and more reliable if you follow these tips.
Before you migrate
- Serverless uses Secure Standard (shared) access mode for safety.
- Some old code/features won’t work on serverless. Check Serverless limitations.
- Easiest to migrate if:
- Your data is in Unity Catalog.
- Your code already works on Standard access mode.
- You can run on DBR 14.3+.
- Test first: run your job on non-serverless, Standard access mode, DBR 14.3+. If it passes, you’re ready.
- Don’t try to fix everything at once. Make new workloads serverless-friendly first; migrate old ones later.
Ingesting data (getting data in)
- You cannot install JARs on serverless → no JDBC/ODBC drivers for ingestion.
- Use these alternatives:
- COPY INTO and streaming tables (SQL-based).
- Auto Loader to pick up new files automatically.
- Partner Connect ingestion tools.
- Add Data UI to upload files directly.
Querying external data (without moving it)
- Want no copies and freshest data? Try Delta Sharing.
- Want quick POCs / ad-hoc reports across external DBs? Try Lakehouse Federation (governed by Unity Catalog).
- Test one or both and see if the speed meets your needs.
Spark configurations
- Serverless blocks most manual Spark configs.
- Only a short list of Spark settings is allowed (see the support list).
- If you set an unsupported config, the job fails.
Watch your costs
- Use system tables for dashboards, alerts, and quick queries.
- Set budget alerts at the account level.
- Import the prebuilt usage dashboard to track spend fast.
Quick checklist (copy/paste)
- Data in Unity Catalog
- Runs on Standard access mode, DBR 14.3+
- No JAR-based JDBC/ODBC ingestion
- Use COPY INTO / Streaming / Auto Loader / Partner Connect / Add Data UI
- For external queries: Delta Sharing or Lakehouse Federation
- Only supported Spark configs used
- Cost dashboards + alerts set up
