Home > database >  Is there a recommended method to clean up Azure Data Factory (Datasets, linked services etc.)?
Is there a recommended method to clean up Azure Data Factory (Datasets, linked services etc.)?

Time:01-24

Elaborating the title further, the current ADF I am working on has a lot of legacy code, i.e. multiple datasets, linked services. Unfortunately no naming convention or system of creating new items were defined.

  1. I have tried to list down all the pipelines, the associated datasets (linked services as well), but this seems a lengthy approach and we have around 100 odd pipelines.

  2. I tried by exporting the complete data factory as an ARM template and tried to create a parser which would automatically create the above list, but it seems that ARM templates are more interlinked than I had thought, I dropped this plan.

Is their a better approach for solving this problem?

CodePudding user response:

You can pull the pipelines/dataflows that uses the particular dataset. this details will be available in the properties tab -> Related tab. enter image description here

Also you get the list of datasets that uses a particular linked service by going to manage tab ->Linkedservice -> Related enter image description here

CodePudding user response:

Since you haven't mentioned data factory version (v1 or v2) and mentioned it has lot of legacy code, I am making an assumption that it could be a v1 factory. In that case, you can check the ADF v1 to ADF v2 migration tool which helps with basic migration tasks (listing resources, doing simple migration steps etc.)

  •  Tags:  
  • Related