- Data Factory in Microsoft Fabric: It’s a new version of Azure Data Factory that offers big-scale data movement and transformation services in the cloud. It’s designed to handle even the toughest ETL tasks, making it user-friendly, powerful, and perfect for businesses.
- Comparison Article: This article looks at the distinctions between Azure Data Factory and Data Factory in Microsoft Fabric.
Difference between Azure Data Factory and Data Factory in Microsoft Fabric
Some different features concepts of Data Factory in Fabric compared to Azure Data Factory.
| Azure Data Factory | Data Factory in Fabric | Description |
|---|---|---|
| Pipeline | Data pipeline | Data pipeline in Fabric is better integrated with the unified data platform including Lakehouse, Datawarehouse, and more. |
| Mapping dataflow | Dataflow Gen2 | Dataflow Gen2 provides easier experience to build transformation. We are in progress of letting more functions of mapping dataflow supported in Dataflow Gen2 |
| Activities | Activities | We are in progress to make more activities of ADF supported in Data Factory in Fabric. Data Factory in Fabric also has some newly attracted activities like Office 365 Outlook activity. |
| Dataset | Not Applicable | Data Factory in Fabric doesn’t have dataset concepts. Connection will be used for connecting each data source and pull data. |
| Linked Service | Connections | Connections have similar functionality as linked service, but connections in Fabric have more intuitive way to create. |
| Triggers | Schedules (other triggers are in progress) | Fabric can use the schedule to automatically run pipeline. We are adding more triggers supported by ADF in Microsoft Fabric. |
| Publish | Save, Run | For pipeline in Fabric, you don’t need to publish to save the content. Instead, you can use Save button to save the content directly. When you click Run button, it will save the content before running pipeline. |
| Autoresolve and Azure Integration runtime | Not Applicable | In Fabric, we don’t have the concept of Integration runtime. |
| Self-hosted integration runtimes | On-premises Data Gateway(in design) | The capability in Fabric is still in progress of design. |
| Azure-SSIS integration runtimes | To be determined | The capability in Fabric hasn’t confirmed the roadmap and design. |
| MVNet and Private End Point | To be determined | The capability in Fabric hasn’t confirmed the roadmap and design. |
| Expression language | Expression language | Expression language is similar in ADF and Fabric. |
| Authentication type in linked service | Authentication kind in connection | Authentication kind in Fabric pipeline already supported popular authentication types in ADF, and more authentication kinds will be added. |
| CI/CD | CI/CD | CI/CD capability in Fabric Data Factory will be coming soon. |
| Export and Import ARM | Save as | Save as is available in Fabric pipeline to duplicate a pipeline. |
| Monitoring | Monitoring, Run history | The monitoring hub in Fabric has more advanced functions and modern experience like monitoring across different workspaces for better insights. |
Data pipeline of data factory in Microsoft Fabric
There are many exciting features of data pipeline in Data Factory of Microsoft Fabric.
Lakehouse/Datawarehouse integration
- In Fabric’s pipeline, you can use Lakehouse and Data Warehouse as both sources and destinations.
- This makes it really easy for you to create your own projects that connect with Lakehouse and Data Warehouse.
- Whether you’re pulling data from these sources or sending data to them, Fabric’s pipeline makes it simple to work with Lakehouse and Data Warehouse.
Office 365 outlook activity
- Office 365 Outlook activity: This is a feature in Office 365 that lets you use Outlook to do things automatically.
- Sends customized email notifications: It sends emails that you can personalize to include information from your pipeline and its activities.
- Easy configuration: You can set it up quickly and without much hassle.
Get data experience
- Modern and easy Get Data experience: You can quickly and easily get the data you need.
- Data Factory in Fabric: This feature is available in Data Factory, a tool in Microsoft Fabric.
- Super-fast for you to set up your copy pipeline: It’s very quick for you to create a pipeline to copy your data.
- Create a new connection: You can easily make a connection to the data source you want to copy from.
Data pipeline monitoring experience
- The monitoring hub, along with Data Factory tools like data flows and pipelines, gives us a complete view of all the tasks.
- We can easily investigate any task within Data Factory.
- It’s also handy for analyzing data across different workspaces through the monitoring hub.
Activity continuity between Azure Data Factory (ADF) and Data Factory in Fabric
| Activity | ADF | Data Factory in Fabric |
|---|---|---|
| Append Variable | Y | Y |
| Copy | Y | Y |
| Delete | Y | Y |
| Execute/Invoke Pipeline | Y | Y |
| Fail | Y | Y |
| Filter | Y | Y |
| For Each | Y | Y |
| GetMetadata | Y | Y |
| If | Y | Y |
| Lookup | Y | Y |
| Script | Y | Y |
| Set Variable | Y | Y |
| Sproc | Y | Y |
| Switch | Y | Y |
| Until | Y | Y |
| Wait | Y | Y |
| Web | Y | Y |
| Webhook | Y | Y |
| Synapse Notebook | Y | Y |
| Azure Function/Functions | Y | Y |
| ADX/KQL | Y | Y |
| Azure Machine Learning | Y | Y |
| Azure Batch | Y | Y |
| Azure Databricks (3 activities in ADF) | Y | Y |
| Validation | Y | N |
| HDInsight (5 activities in ADF) | Y | N |
| SSIS | Y | N |
| Mapping Dataflow | Y | N |
| Dataflow Gen2 | N/A | Y |
| Office 365 Outlook | N/A | Y |
| Teams | N/A | Y |
| Dataset Refresh | N/A | N |
| Azure ML Batch Execution | Deprecated | N/A |
| Azure ML Update Resource | Deprecated | N/A |
| Power Query (ADF only – Wrangling Dataflow) | Deprecated | N/A |
| USQL | Deprecated | N/A |
New activities in Fabric Data Factory
In addition to maintaining activity continuity, Data Factory in Fabric introduces some new activities to meet your richer orchestration needs. These new activities are:
- Outlook: Available in Fabric Data Factory to facilitate integration with Outlook services.
- Teams: Available in Fabric Data Factory to enable orchestration of Microsoft Teams activities.
- Dataset Refresh: In progress in Fabric Data Factory to enhance dataset refresh capabilities.
- Dataflow Gen2: Available in Fabric Data Factory to empower data orchestration with advanced dataflow capabilities.
Connector continuity between Azure Data Factory (ADF) and Data Factory in Fabric
| Connector Type | ADF (Source/Sink) | Fabric Data Factory (Source/Sink) |
|---|---|---|
| Azure & Fabric | ||
| Blob Storage | ✓/✓ | ✓/✓ |
| Cognitive Search Index | -/✓ | N |
| Cosmos DB – SQL API | ✓/✓ | ✓/✓ |
| Cosmos DB – MongoDB API | ✓/✓ | N |
| Azure Data Explorer | ✓/✓ | ✓/✓ |
| ADLS Gen1 | ✓/✓ | ✓/✓ |
| ADLS Gen2 | ✓/✓ | ✓/✓ |
| Database for MariaDB | ✓/- | N |
| Database for MySQL | ✓/✓ | N |
| Databricks Delta Lake | ✓/✓ | N |
| Azure File Storage | ✓/✓ | N |
| Database for PostgreSQL | ✓/✓ | ✓/✓ |
| Azure Cosmos DB for PostgreSQL | ✓/✓ | N |
| Azure Table Storage | ✓/✓ | ✓/✓ |
| SQL Database | ✓/✓ | ✓/✓ |
| SQL Database MI | ✓/✓ | ✓/✓ |
| Synapse Analytics | ✓/✓ | ✓/✓ |
| Fabric Lakehouse ​ | ✓/✓ | ✓/✓ |
| Fabric DW | N | ✓/✓ |
| Fabric KQL | N | ✓/✓ |
| Database & DW | ||
| Amazon Redshift | ✓/- | ✓/- |
| DB2 | ✓/- | N |
| Drill | ✓/- | N |
| Google Big Query | ✓/- | N |
| Greenplum | ✓/- | N |
| HBase | ✓/- | N |
| Hive | ✓/- | N |
| Impala | ✓/- | N |
| Informix | ✓/✓ | N |
| MariaDB | ✓/- | N |
| Microsoft Access | ✓/✓ | N |
| MySQL | ✓/- | N |
| Netezza | ✓/- | N |
| Oracle | ✓/✓ | N |
| Amazon RDS for Oracle | ✓/- | N |
| Phoenix | ✓/- | N |
| PostgreSQL | ✓/- | ✓/- |
| Presto | ✓/- | N |
| SAP BW Open Hub | ✓/- | N |
| SAP BW MDX | ✓/- | N |
| SAP HANA | ✓/✓ | N |
| SAP Table | ✓/- | N |
| Snowflake | ✓/✓ | ✓/✓ |
| Amazon RDS for SQL Server | ✓/- | ✓/- |
| SQL Server | ✓/✓ | ✓/✓ |
| Spark | ✓/- | N |
| Sybase | ✓/- | N |
| Teradata | ✓/- | N |
| Vertica | ✓/- | N |
| File Storage | ||
| Amazon S3 | ✓/- | ✓/- |
| Amazon S3 Compatible | ✓/- | ✓/- |
| File System | ✓/✓ | N |
| FTP | ✓/- | ✓/- |
| Google Cloud Storage | ✓/- | ✓/- |
| HDFS | ✓/- | N |
| Oracle Cloud Storage | ✓/- | N |
| SFTP | ✓/✓ | ✓/✓ |
| File Formats | ||
| AVRO | ✓/✓ | ✓/✓ |
| Binary | ✓/✓ | ✓/✓ |
| Delimited Text | ✓/✓ | ✓/✓ |
| Excel | ✓/- | ✓/- |
| JSON | ✓/✓ | ✓/✓ |
| ORC | ✓/✓ | ✓/✓ |
| Parquet | ✓/✓ | ✓/✓ |
| XML | ✓/- | ✓/- |
| NoSQL | ||
| Cassandra | ✓/- | N |
| Couchbase | ✓/- | N |
| MongoDB | ✓/✓ | ✓/✓ |
| MongoDB Atlas | ✓/✓ | ✓/✓ |
| Services & Apps | ||
| Amazon MWS | ✓/- | N |
| Concur | ✓/- | N |
| Dataverse | ✓/✓ | ✓/✓ |
| Dynamics 365 | ✓/✓ | N |
| Dynamics AX | ✓/- | N |
| Dynamics CRM | ✓/✓ | ✓/✓ |
| Google AdWords | ✓/- | N |
| HubSpot | ✓/- | N |
| Jira | ✓/- | N |
| Magento | ✓/- | N |
| Marketo | ✓/- | N |
| Microsoft 365 | ✓/- | ✓/- |
| Oracle Eloqua | ✓/- | N |
| Oracle Responsys | ✓/- | N |
| Zoho | ✓/- | N |
| Oracle Service Cloud | ✓/- | N |
| PayPal | ✓/- | N |
| QuickBooks | ✓/- | N |
| Salesforce | ✓/✓ | N |
| SF Service Cloud | ✓/✓ | N |
| SF Marketing Cloud | ✓/- | N |
| SAP C4C | ✓/✓ | N |
| SAP ECC | ✓/- | N |
| ServiceNow | ✓/- | N |
| SharePoint Online List | ✓/- | ✓/- |
| Shopify | ✓/- | N |
| Square | ✓/- | N |
| Web Table | ✓/- | N |
| Xero | ✓/- | N |
| Generic | ||
| HTTP | ✓/- | ✓/- |
| OData | ✓/- | ✓/- |
| ODBC | ✓/✓ | N |
| REST | ✓/✓ | ✓/✓ |
