Flatten Multiple Arrays with a Single Step in Azure Data Factory (ADF)

Unroll Multiple Arrays in a Single Flatten Step in Azure Data Factory
Introduction
Azure Data Factory (ADF) is a managed data integration service that simplifies the development, execution, and monitoring of extract, transform, and load (ETL) pipelines. ADF provides a visual authoring experience for creating data pipelines, and it allows for the orchestration of complex data processing workflows. In this article, we’ll look at how to use the Flatten Activity in ADF to unroll multiple arrays in a single step.

What is the Flatten Activity?
The Flatten Activity is part of the Data Transformation activities available in ADF. It is used to flatten or expand nested structures or arrays in a dataset. This is useful when you need to transform data from a nested structure into a flat structure, or vice versa. For example, if you have a dataset with an array of objects that contains multiple records, the Flatten Activity can be used to expand those records into individual rows.

Unrolling Multiple Arrays in a Single Flatten Step
The Flatten Activity can be used to unroll multiple arrays in a single step. This can be useful when you have a dataset with multiple nested arrays that you need to expand into individual rows. To do this, you’ll need to define the structure of the nested array, which is done by setting the Flatten Settings property. This property defines the structure of the nested array by specifying the array name and the name of the fields within the array.

Example: Unrolling Multiple Arrays
In this example, we’ll look at how to unroll two nested arrays using the Flatten Activity in ADF. We’ll assume that we have a dataset with two arrays, “orders” and “customers”, which contain multiple records. To unroll these arrays in a single step, we’ll need to define the structure of the arrays in the Flatten Settings property. This is done by specifying the array name and the name of the fields within the array. For example, to unroll the “orders” array, we would set the Flatten Settings property to “orders/order_id, orders/order_date”. We would also need to define the structure of the “customers” array by setting the Flatten Settings property to “customers/customer_id, customers/customer_name”.

Conclusion
The Flatten Activity in ADF can be used to unroll multiple arrays in a single step. This can be useful when you have a dataset with multiple nested arrays that you need to expand into individual rows. To do this, you’ll need to define the structure of the nested array, which is done by setting the Flatten Settings property. This property defines the structure of the nested array by specifying the array name and the name of the fields within the array.
References:
Unroll multiple arrays in a single Flatten step in ADF
and goals.

1) Flatten multiple arrays in ADF
2) Unroll arrays