Introduction

The S3001 error in Databricks indicates that your IAM role does not have the required permissions to access AWS S3. This issue commonly occurs when reading from or writing to S3 using Spark or Delta Lake.

🚨 Common Symptoms of S3001 Error:

• Error message: “S3001: Access Denied when accessing S3 bucket.”
• Data ingestion jobs fail when trying to read from S3.
• Write operations to S3 fail, even though the bucket exists.
• S3 path listing or file access fails during Delta table operations.

This guide will cover the causes, troubleshooting steps, and solutions for fixing the S3001 error.

Causes of S3001 – S3 Access Denied

1. IAM Role Missing Required S3 Permissions

If the IAM role attached to your Databricks cluster does not have the required S3 permissions, you will get an S3001 Access Denied error.

2. Bucket Policy Restricts Access

Even if your IAM role has sufficient permissions, the S3 bucket policy might block access. This happens if the bucket policy restricts public or cross-account access.

3. Wrong IAM Role Attached to the Cluster

If your cluster is running with an incorrect or insufficient IAM role, it will not have the necessary permissions to access S3.

4. Encryption or VPC Endpoint Issues

• Server-side encryption (SSE) or VPC endpoint misconfigurations can also trigger the S3001 error.

How to Fix the S3001 Error (Access Denied)

1. Verify and Update IAM Role Permissions

Ensure that your IAM role has the correct S3 permissions to access the bucket.

Example Policy for Full S3 Access:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": [
        "s3:GetObject",
        "s3:PutObject",
        "s3:ListBucket"
      ],
      "Resource": [
        "arn:aws:s3:::your-bucket-name",
        "arn:aws:s3:::your-bucket-name/*"
      ]
    }
  ]
}

Steps to Update IAM Role:

1. Go to AWS Console → IAM → Roles.
2. Select the role attached to your Databricks cluster.
3. Add or modify the policy to include the required S3 permissions.

2. Check the S3 Bucket Policy

Ensure the S3 bucket policy allows access for the IAM role used by your Databricks cluster.

Example Bucket Policy:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Principal": {
        "AWS": "arn:aws:iam::123456789012:role/your-iam-role"
      },
      "Action": "s3:GetObject",
      "Resource": "arn:aws:s3:::your-bucket-name/*"
    }
  ]
}

Verify the Bucket Policy:

1. Go to AWS Console → S3 → Bucket Permissions.
2. Check the Bucket Policy and ensure your IAM role has the required access.

3. Verify IAM Role Attached to Databricks Cluster

Ensure that your Databricks cluster is using the correct IAM role with sufficient permissions.

Steps to Check the IAM Role:

1. Go to Databricks UI → Clusters → Configuration.
2. Under Advanced Options, verify the Instance Profile (IAM role).
3. Ensure the role has the necessary S3 permissions.

If incorrect, attach the correct IAM role:

databricks clusters configure --cluster-id <cluster-id> --instance-profile-arn <arn-of-your-iam-role>

4. Verify S3 Bucket Encryption Settings

If your S3 bucket is using server-side encryption (SSE), ensure that your IAM role has the required permissions to access encrypted objects.

Example Policy for SSE Access:

{
  "Effect": "Allow",
  "Action": [
    "s3:GetObject",
    "s3:PutObject"
  ],
  "Resource": "arn:aws:s3:::your-encrypted-bucket/*",
  "Condition": {
    "StringEquals": {
      "s3:x-amz-server-side-encryption": "aws:kms"
    }
  }
}

5. Verify VPC Endpoint Configuration (If Applicable)

If your Databricks cluster is in a private VPC, ensure that the VPC endpoint for S3 is configured properly.

Steps to Verify:

1. Go to AWS Console → VPC → Endpoints.
2. Ensure a VPC endpoint exists for com.amazonaws.<region>.s3.
3. Check the security groups and routing rules to allow traffic to S3.

Step-by-Step Troubleshooting Guide

1. Verify S3 Access Using Databricks Notebook

Run the following code in a Databricks notebook to check S3 access:

dbutils.fs.ls("s3://your-bucket-name/")

• If the S3001 error appears, the IAM role or bucket policy is misconfigured.

2. Check Cluster Logs for Detailed Errors

Go to Databricks UI → Clusters → Driver Logs and look for access denied or authentication errors.

3. Verify Permissions and Bucket Policy in AWS Console

Check the IAM policy and bucket policy to ensure the necessary permissions.

4. Test S3 Access with AWS CLI

aws s3 ls s3://your-bucket-name --profile <your-aws-profile>

• If you receive an Access Denied error, update the IAM role or bucket policy.

Best Practices for Avoiding S3001 Errors

1. Grant Least Privilege Access: Only allow the required S3 actions (GetObject, PutObject, ListBucket).
2. Use Instance Profiles for secure and automatic credential management.
3. Regularly Audit IAM Policies: Ensure permissions are up to date and secure.
4. Enable Logging for S3 Access: Use AWS CloudTrail to monitor S3 access patterns.
5. Avoid Hardcoding S3 Credentials: Use IAM roles and instance profiles instead.

Conclusion

The S3001 – S3 Access Denied error in Databricks is typically caused by insufficient IAM permissions, restrictive bucket policies, or encryption settings. By following the steps in this guide, you can:
✅ Verify and update IAM role permissions.
✅ Adjust the S3 bucket policy to allow access.
✅ Ensure the correct IAM role is attached to your Databricks cluster.
✅ Check encryption and VPC configurations.