Integrating Azure Databricks notebooks with Azure Data Factory (ADF) involves several steps. The integration allows you to run Databricks notebooks…

Enabling Azure Log Analytics for Azure Databricks involves several steps, as you need to integrate Databricks with Azure Monitor, which…

Logging is a means of tracking events. Logging is important for software developing, debugging, and running. Information Levels: Below are…

To Setup Unity Catalog you should have account global admin privilege’s to the subscription. At Subscription level it should have…

It is generally administrate the access and also audit the database access. It is also know as Data Governance. Unity…

We can find the Ganglia at Databricks Clusters > Metrics, and it is shown below. Figure 1: Ganglia metrics and their interpretation The…

Data Shuffling Explained Data shuffling happens when tasks like joining two datasets, aggregating data, or window operations require data to…

Verifying a memory issue These error messages, however, are often generic and can be caused by other issues. So, if…

To View compute information in the Apache Spark UI You can view detailed information about Spark jobs by selecting the Spark…

View compute To view your compute, click  Compute in the workspace sidebar. Pin a compute Edit a compute You can edit…

Azure Databricks Cluster A cluster is a group of machines called “nodes” that work together to process your data and…

Init scripts in Databricks are scripts that run during the startup of a cluster or when a new node is…

Azure Databricks Architecture How Azure Databricks Architecture works?