Error: INTEG001 – API Rate Limit Exceeded in Databricks

Table of Contents

Introduction

The INTEG001: API Rate Limit Exceeded error occurs when you exceed the allowed number of API requests per time window in Databricks or an integrated service. This error can appear in various scenarios such as:

Databricks REST API calls (jobs, clusters, secrets, etc.).
Cloud Storage APIs (AWS S3, Azure ADLS, Google Cloud Storage).
Third-party APIs (external data services like Salesforce, OpenAI, Twitter, etc.).

This guide covers the root causes, troubleshooting steps, and best practices to resolve the INTEG001 – API rate limit exceeded error.

Common Causes of the INTEG001 Error

1. Exceeding Databricks API Rate Limits

Frequent job submissions, cluster management, or notebook executions via the Databricks REST API.
Automated scripts continuously polling for updates without retry mechanisms.

Databricks API Limits:

50 requests per second per workspace for most REST API endpoints.
Higher limits available for premium customers.

2. Cloud Storage API Quotas Exceeded (AWS S3, ADLS, GCS)

Too many concurrent read/write operations to cloud storage.
S3, ADLS, or GCS throttles requests due to API usage spikes.

3. External Third-Party API Limits (Salesforce, OpenAI, Twitter, etc.)

External services impose rate limits on API calls to prevent abuse.
Example: Twitter API allows 900 requests per 15-minute window for standard access.

How to Troubleshoot INTEG001: API Rate Limit Exceeded

1. Identify the API Being Throttled

Determine if the error originates from:

Databricks REST API
Cloud storage (S3, ADLS, GCS)
External third-party APIs (e.g., Salesforce, OpenAI)

Databricks REST API Example:

curl -X GET -H "Authorization: Bearer <TOKEN>" https://<databricks-instance>/api/2.0/clusters/list

Cloud Storage Example (AWS S3):

Check AWS CloudWatch logs for API throttling events.

aws s3api list-objects --bucket my-bucket --prefix data/

2. Implement Exponential Backoff for Retry Logic

Python Example for Databricks API

import time
import requests

def call_api_with_retry(url, headers, retries=5):
    for i in range(retries):
        response = requests.get(url, headers=headers)
        if response.status_code == 200:
            return response.json()
        elif response.status_code == 429:  # Rate limit exceeded
            wait_time = 2 ** i  # Exponential backoff
            print(f"Rate limit exceeded. Retrying in {wait_time} seconds...")
            time.sleep(wait_time)
        else:
            response.raise_for_status()

api_url = "https://databricks-instance/api/2.0/clusters/list"
headers = {"Authorization": "Bearer YOUR_TOKEN"}
data = call_api_with_retry(api_url, headers)

3. Reduce API Request Frequency

Batch multiple API requests into a single call when possible.
Use caching to reduce repeated API calls for the same data.
Increase the polling interval to avoid hitting the rate limit.

Example: Use caching for frequent S3 reads:

df.cache()  # Cache data to reduce repeated S3 reads
df.show()

4. Monitor API Quotas and Set Alerts

Use Databricks monitoring tools or cloud provider monitoring services to track API usage.
Set alerts for nearing API limits and throttle requests accordingly.

AWS Example:

aws cloudwatch put-metric-alarm --alarm-name "S3-API-Limit" --metric-name "S3RequestCount" --namespace "AWS/S3" --threshold 1000

Azure Example:
Use Azure Monitor to track API throttling events in ADLS.

5. Check API Rate Limit Documentation for Third-Party Services

Twitter API: 900 requests per 15 minutes for standard access.
OpenAI API: 60 requests per minute for free-tier users.
Salesforce API: 100,000 API calls per day for enterprise users.

Use Retry-After Headers for Third-Party APIs:

response = requests.get("https://api.twitter.com/2/tweets")
if response.status_code == 429:
    retry_after = int(response.headers.get("Retry-After", 60))
    print(f"Rate limit exceeded. Retrying in {retry_after} seconds...")
    time.sleep(retry_after)

Best Practices to Prevent INTEG001: API Rate Limit Exceeded

✅ Implement Exponential Backoff and Retry Logic

Start with a small wait time and double the delay for each retry.

✅ Batch and Optimize API Requests

Reduce the number of requests by batching multiple operations into a single request.

✅ Use Event-Driven Workflows Instead of Frequent Polling

For Databricks, use Job Completion Callbacks or Event Notifications.

✅ Monitor API Usage and Set Alerts

Use CloudWatch (AWS), Azure Monitor, or Databricks Metrics to track API requests.

✅ Leverage API Rate Limit Documentation

Check the rate limits for your specific service and design your integration accordingly.

Conclusion

The INTEG001: API Rate Limit Exceeded error is a common issue when working with Databricks REST APIs, cloud storage services, or third-party APIs. By implementing retry logic, reducing API request frequency, and monitoring usage, you can prevent rate-limiting issues and ensure stable API performance.

Mohammad Gufran Jahangir