There are certain limitations to the current Data Factory in Microsoft Fabric features.
Data pipeline limitations in Microsoft Fabric
- Some features like tumbling window and event triggers are not yet available in Fabric pipelines.
- Continuous Integration and Continuous Delivery (CI/CD) aren’t supported in pipelines.
- Connectors don’t support OAuth, Azure Key Vault (AKV), Managed System Identity (MSI), or parameters.
- Certain activities like GetMetaData and Script can’t use Fabric KQL databases as a source.
- Copy activity uses a Web connector, while Web/Webhook activities use a richer Web v2 connector.
- Custom activities are not supported; use Azure Batch activity instead.
- Data pipelines are limited to their workspace and can’t interact with items in other workspaces.
- On-premises data gateway can be used with Dataflow Gen2 for ingesting on-premises data.
- Managed VNet can’t be used in pipelines.
- Web activity doesn’t support service principal based authentication.
Data pipeline resource limits
The following table describes the resource limitations for pipelines in Data Factory in Microsoft Fabric.
| Pipeline Resource | Default limit | Maximum limit |
|---|---|---|
| Total number of entities, such as pipelines, activities, triggers within a workspace | 5,000 | 5,000 |
| Concurrent pipeline runs per data factory that’s shared among all pipelines in workspace | 10,000 | 10,000 |
| External activities like stored procedure, Web, Web Hook, and others | 3,000 | 3,000 |
| Pipeline activities execute on integration runtime, including Lookup, GetMetadata, and Delete | 1,000 | 1,000 |
| Concurrent authoring operations, including test connection, browse folder list and table list, preview data, and so on | 200 | 200 |
| Maximum activities per pipeline, which includes inner activities for containers | 40 | 40 |
| Maximum parameters per pipeline | 50 | 50 |
| ForEach items | 100,000 | 100,000 |
| ForEach parallelism | 20 | 50 |
| Lookup Activity item count | 5000 | 5000 |
| Maximum queued runs per pipeline | 100 | 100 |
| Characters per expression | 8,192 | 8,192 |
| Maximum timeout for pipeline activity runs | 24 hours | 24 hours |
| Bytes per object for pipeline objects | 200 KB | 200 KB |
| Bytes per payload for each activity run | 896 KB | 896 KB |
| Data Integration Units per copy activity run | Auto | 256 |
| Write API calls | 1,200/h | 1,200/h |
| Read API calls | 12,500/h | 12,500/h |
| Maximum time of data flow debug session | 8 hrs | 8 hrs |
| Meta Data Entity Size limit in a factory | 2 GB | 2 GB |
Data Factory Dataflow Gen2 limitations
The following list describes the limitations for Dataflow Gen2 in Data Factory in Microsoft Fabric.
- Data Factory Fast Copy: It’s not ready yet.
- Data destination to Lakehouse:
- Avoid using spaces or special characters in column or table names.
- Don’t use duration or binary columns in Dataflow Gen2 dataflows.
- Make sure you have the latest gateway version installed for Dataflow Gen2.
- Long refreshes (over an hour) with OAuth2 credentials may fail due to token expiration.
- Incremental refresh: Not available in Dataflow Gen2.
- Delta Lake specification: Avoid using case-sensitive column names to prevent errors.
- Gateway and data destination: Limited to one-hour evaluation or refresh time.
- Column nullability: It’s based on the behavior specified in the Power Query used for Dataflow Gen2.
The following table indicates the supported data types in specific storage locations.
| Supported data types per storage location: | DataflowStagingLakehouse | Azure DB (SQL) Output | Azure Data Explorer Output | Fabric Lakehouse (LH) Output | Fabric Warehouse (WH) Output |
|---|---|---|---|---|---|
| Action | No | No | No | No | No |
| Any | No | No | No | No | No |
| Binary | No | No | No | No | No |
| Currency | Yes | Yes | Yes | Yes | No |
| DateTimeZone | Yes | Yes | Yes | No | No |
| Duration | No | No | Yes | No | No |
| Function | No | No | No | No | No |
| None | No | No | No | No | No |
| Null | No | No | No | No | No |
| Time | Yes | Yes | No | Yes | Yes |
| Type | No | No | No | No | No |
| Structured (List, Record, Table) | No | No | No | No | No |
