Monday, 28 September 2026

Delta Lake Backup for Databricks on Azure Data Lake Storage Gen2

The problem I kept hitting

If you run a Delta Lake on ADLS Gen2, you have probably asked the same question I did: how do I actually back this up? Azure Backup has protected blob storage for a while, but the moment you turn on hierarchical namespace, you were on your own.

Every article I found said the same thing: build an ADF pipeline, schedule AzCopy to copy containers to another account, or rely on Gen2 soft delete. Each of these works up to a point. With a copy pipeline, I own the schedule, the retention clean-up, the monitoring and the access model. Soft delete only helps with accidental deletes, and it's no use when a bad job overwrites data. On top of that, we run multiple Delta Lake environments, and refreshing UAT from PROD or restoring between environments was getting more complex with every release. None of it ever felt like a real backup.

That changed in November 2025, when Microsoft released vaulted backup for Azure Data Lake Storage. Here is what I learned trying it on a Synapse lakehouse.

How it works, in short

You point a Backup vault at your HNS storage account. Behind the scenes, Azure Backup creates its own storage account inside the vault and replicates your containers into it. Each backup is a recovery point you can restore later.

The bit I liked: there is no Spark, no pipeline and no integration runtime to pay for. You set a policy, and retention of up to 10 years is handled for you.

Does it work for Delta tables?

Yes, and the reason is simple. A Delta table is Parquet files plus a _delta_log folder. A commit only counts once its log file is written. So even if a backup catches a job halfway through, the restored table just opens at the last committed version. Stray Parquet files from the unfinished job are ignored.

That said, a few things caught my attention:

  • Keep each table in one container. Backup is container-scoped, and moving data between containers is not supported. If you promote tables by moving folders from bronze to silver, rethink that.
  • It is a daily backup, not point-in-time. One scheduled run a day, plus a few on-demand ones. I still rely on Delta time travel for "undo that bad MERGE" moments.
  • Restores go to a different account. Your ABFSS paths change, so tables in the lake database or metastore need re-pointing.
  • Soft-delete undelete stops working while vaulted backup is on. The vault becomes your recovery path.

The full list of supported and unsupported scenarios is in the support matrix. Read it before you enable this on production.

Setting it up

The setup took me about 15 minutes:

  1. Create a Backup vault in the same region as the lake. I put mine in a separate, locked-down subscription.
  2. Create a backup policy with your schedule and retention.
  3. Give the vault's managed identity the Storage Account Backup Contributor role on the storage account.
  4. In the vault, choose + Backup, pick the account and containers, and let validation check the roles.

The first recovery point takes a while to appear, so don't panic if it's not there straight away. Microsoft's configure guide has the portal, PowerShell and CLI steps.

Restoring and checking a table

I restored one table into a scratch storage account using a prefix filter on the table folder, making sure _delta_log came back with it. Steps are in the restore guide.

Then I checked it from a notebook:

path = "abfss://silver@stlakerestore.dfs.core.windows.net/sales/fact_orders"

from delta.tables import DeltaTable
DeltaTable.forPath(spark, path).history(3).show(truncate=False)
spark.read.format("delta").load(path).count()

If the last version and row count look right, you can re-point the table or copy it back to production.

My take

For Delta on ADLS Gen2, this is now my default safety net. I layer it like this:

  • Time travel for recent mistakes.
  • Vaulted backup for ransomware, accidental deletes and long-term retention.
  • A small ADF or AzCopy copy to another region for the few gold tables I can't afford to lose, since restores stay in-region.

If you are still running a home-grown copy pipeline as your only backup, try this on a non-production account first. Restore a table, time it, and you'll have a real recovery number to share with your team.

Useful links

Copyright © 2026 Vinoth N Manoharan. The information provided in this post is provided "as is" with no implied warranties or guarantees.

at

No comments:

Post a Comment

Subscribe to: Post Comments (Atom)