Data Factory limitations in Microsoft Fabric

Posted by

There are certain limitations to the current Data Factory in Microsoft Fabric features. 

Data pipeline limitations in Microsoft Fabric

  1. Some features like tumbling window and event triggers are not yet available in Fabric pipelines.
  2. Continuous Integration and Continuous Delivery (CI/CD) aren’t supported in pipelines.
  3. Connectors don’t support OAuth, Azure Key Vault (AKV), Managed System Identity (MSI), or parameters.
  4. Certain activities like GetMetaData and Script can’t use Fabric KQL databases as a source.
  5. Copy activity uses a Web connector, while Web/Webhook activities use a richer Web v2 connector.
  6. Custom activities are not supported; use Azure Batch activity instead.
  7. Data pipelines are limited to their workspace and can’t interact with items in other workspaces.
  8. On-premises data gateway can be used with Dataflow Gen2 for ingesting on-premises data.
  9. Managed VNet can’t be used in pipelines.
  10. Web activity doesn’t support service principal based authentication.

Data pipeline resource limits

The following table describes the resource limitations for pipelines in Data Factory in Microsoft Fabric.

Pipeline ResourceDefault limitMaximum limit
Total number of entities, such as pipelines, activities, triggers within a workspace5,0005,000
Concurrent pipeline runs per data factory that’s shared among all pipelines in workspace10,00010,000
External activities like stored procedure, Web, Web Hook, and others3,0003,000
Pipeline activities execute on integration runtime, including Lookup, GetMetadata, and Delete1,0001,000
Concurrent authoring operations, including test connection, browse folder list and table list, preview data, and so on200200
Maximum activities per pipeline, which includes inner activities for containers4040
Maximum parameters per pipeline5050
ForEach items100,000100,000
ForEach parallelism2050
Lookup Activity item count50005000
Maximum queued runs per pipeline100100
Characters per expression8,1928,192
Maximum timeout for pipeline activity runs24 hours24 hours
Bytes per object for pipeline objects200 KB200 KB
Bytes per payload for each activity run896 KB896 KB
Data Integration Units per copy activity runAuto256
Write API calls1,200/h1,200/h
Read API calls12,500/h12,500/h
Maximum time of data flow debug session8 hrs8 hrs
Meta Data Entity Size limit in a factory2 GB2 GB

Data Factory Dataflow Gen2 limitations

The following list describes the limitations for Dataflow Gen2 in Data Factory in Microsoft Fabric.

  1. Data Factory Fast Copy: It’s not ready yet.
  2. Data destination to Lakehouse:
    • Avoid using spaces or special characters in column or table names.
    • Don’t use duration or binary columns in Dataflow Gen2 dataflows.
    • Make sure you have the latest gateway version installed for Dataflow Gen2.
    • Long refreshes (over an hour) with OAuth2 credentials may fail due to token expiration.
  3. Incremental refresh: Not available in Dataflow Gen2.
  4. Delta Lake specification: Avoid using case-sensitive column names to prevent errors.
  5. Gateway and data destination: Limited to one-hour evaluation or refresh time.
  6. Column nullability: It’s based on the behavior specified in the Power Query used for Dataflow Gen2.

The following table indicates the supported data types in specific storage locations.

Supported data types per storage location:DataflowStagingLakehouseAzure DB (SQL) OutputAzure Data Explorer OutputFabric Lakehouse (LH) OutputFabric Warehouse (WH) Output
Structured (List, Record, Table)NoNoNoNoNo

Inline Feedbacks
View all comments
Would love your thoughts, please comment.x