Home > Software engineering >  Copy files in subdirs to azure storage with ADF
Copy files in subdirs to azure storage with ADF

Time:01-21

I have a folder structures like this:

folder1/folder2
               /YearNumber1
                           /monthYear1
                                     /somefile.csv, tbFiles.csv
                           /monthYear2
                                     /somefile2.csv, tbFiles2.csv
                           ...(many folders as above)
               /YearNumber2
                           /montYear11
                                      /somefileXXYYZz.csv, otherFile.csv
                           /monthYear12
                                       /someFileRandom.csv. dedFile.csv
                           ...(many folders as above)

Source: Binary, linked via fileshare linked service

Destination: Binary, on azure blob storage

I don't want to retain the structure, just need to copy all csv files.

Using CopyActivity:

Wildcard Path: @concat('folder1/folder2/','*/','*/',) / '*.csv'

with recursive

But it copies nothing, 0 Bytes.

CodePudding user response:

You can use the below options in the CopyActivity Source Setting:

1. File path type

Allowed wildcards are: * (matches zero or more characters) and ? (matches zero or single character); use ^ to escape if your actual folder name has wildcard or this escape char inside.
See official MS docs for more examples in enter image description here

If there are only .csv files in your source directories you can simply specify wildcardFileName as just *

  •  Tags:  
  • Related