Enabling Azure Log Analytics for Azure Databricks involves several steps, as you need to integrate Databricks with Azure Monitor, which underlies Log Analytics. Here’s a step-by-step guide on how to set this up and how to query logs from the Log Analytics workspace.

Step 1: Prepare Azure Databricks and Azure Log Analytics

1. Create an Azure Databricks Workspace if you don’t already have one:
   • Navigate to the Azure Portal.
   • Click on “Create a resource”, search for “Azure Databricks”, and follow the prompts to create it.
2. Create an Azure Log Analytics Workspace:
   • In the Azure Portal, create a resource and search for “Log Analytics”.
   • Choose “Log Analytics Workspace” and provide the necessary details like name, subscription, resource group, and location.

Step 2: Enable Diagnostic Settings in Azure Databricks

1. Open Azure Databricks workspace settings:
   • Go to the Azure portal.
   • Find your Databricks workspace, click on it, and then select “Diagnostic settings” under the Monitoring section.
2. Add diagnostic setting:
   • Click on “+ Add diagnostic setting”.
   • Enter a name for the setting.
   • Choose the logs and metrics you wish to send to Log Analytics. For logs, you might include dbfs, clusters, accounts, etc.
   • Under “Destination details”, select “Send to Log Analytics workspace”.
   • Choose the Log Analytics workspace you created earlier.
   • Save the settings.

Step 3: Configure Databricks to Send Logs

• Ensure that logging is enabled in your Databricks clusters:
  • When creating or editing a Databricks cluster, enable the logging to the workspace by selecting the appropriate checkboxes under the “Advanced Options” section.

Step 4: Query Logs from Log Analytics Workspace

Once your logs are being sent to the Log Analytics workspace, you can query them using Kusto Query Language (KQL).

1. Navigate to your Log Analytics workspace:
   • Go to the Azure Portal.
   • Find and click on your Log Analytics workspace.
2. Open Logs:
   • Inside the Log Analytics workspace, select “Logs” from the general menu to open the log query tool.
3. Write and Run Queries:
   • Use KQL to query the data. Here’s an example query to get you started:

DatabricksCluster

Mohammad Gufran Jahangir