#HOW TO USE FOLDER FACTORY CODE#

In this situation a central variable controls which activity path is used at runtime. Another option and technique I’ve used in the past is to handle different environment setups internally to Data Factory via a Switch activity.

#HOW TO USE FOLDER FACTORY CODE#

Having that separation of debug and development is important to understand for that first Data Factory service and even more important to get it connected to a source code system.įor clarification, other downstream environments (test, UAT, production) do not need to be connected to source control.įinal thoughts on environment setup. Pull requests of feature branches would be peer reviewed before merging into the main delivery branch and published to the development Data Factory service. Next, I’d expect developers working within those code branches to be using the ADF debug feature to perform basic end to end testing of newly created pipelines and using break points on activities as required. Then, that development service should be used with multiple code repository branches that align to backlog features. Using Azure DevOps or GitHub doesn’t matter, although authentication against Azure DevOps is slightly simpler within the same tenant. Obvious for any solution, but when applying this to ADF, I’d expect to see the development service connected to source control as a minimum. Having a clean separation of resources for development, testing and production. Platform Setup Environment Setup & Developer Debugging Let’s start, my set of Data Factory best practices: Check it out if you prefer a detailed guide on creating a good Data Factory.Īs a side project I recently created a PowerShell script to inspect an ARM template export for a given Data Factory instance and provide a summarised check list coverings a partial set of the bullet points on the right, view the blog post here. I did the technical review of this book with Richard, it has a lot of great practical content.

Securing Activity Inputs & Outputs (new).

Getting Key Vault Secrets at Runtime (new).

Linked Service Security via Azure Key Vault.

Understanding Activity Concurrency (new).

Multiple Data Factory Instance’s (updated).

Environment Setup & Developer Debugging ….

But maybe not to everybody, especially if your new to ADF. Some of the below statements might seem obvious and fairly simple.

Hopefully together we can mature these things into a common set of best practices or industry standards when using the cloud resource. The following are my suggested answers to this and what I think makes a good Data Factory.

In either case, I would phrase the question what makes a good Data Factory implementation? We can call this technical standards or best practices if you like. However, after 6 years of working with ADF I think its time to start suggesting what I’d expect to see in any good Data Factory implementation, one that is running in production as part of a wider data platform solution. My colleagues and friends from the community keep asking me the same thing… What are the best practices from using Azure Data Factory (ADF)? With any emerging, rapidly changing technology I’m always hesitant about the answer.

How to use folder factory