Problem: We need to perform a task under which we have to transfer all files ( CSV format) stored in AWS S3 bucket to a on-premise LAN folder using the Lambda functions. This will be a scheduled tasks which will be carried out after every 1 hour, and the file will again be transferred from S3 to on-premise LAN folder while replacing the existing ones. Size of these files is not large (preferably under few MBs).

I am not able to find out any AWS managed service to accomplish this task.

I am a newbie to AWS, any solution to this problem is most welcome.

Thanks,

CodePudding user response：

Check out S3 Sync, I think it will help you accomplish this task: https://awscli.amazonaws.com/v2/documentation/api/latest/reference/s3/sync.html#examples

To run any AWS CLI in your computer, you will need to setup credentials, and the setup account/roles should have permissions to do the task (e.g. access S3) Check out AWS CLI setup here: https://docs.aws.amazon.com/cli/latest/userguide/cli-chap-configure.html

CodePudding user response：

Actually, I am looking for a solution by which I can push S3 files to on-premise folder automatically

For that you need to make the on-premise network visible to the logic (lambda, whatever..) "pushing" the content. The default solution is using the AWS site-to-site VPN.

There are multiple options for setting up the VPN, you could choose based on the needs.

Then the on-premise network will look just like another subnet.

However - VPN has its complexity and cost. In most of the cases it is much easier to "pull" data from the on-premise environment.

To sync data there are multiple options. For a managed service, I could point out the S3 Gateway which based on your description sounds like an insane overkill.

Maybe you could start with a simple cron job (or a task timer if working with windows) and run a CLI command to sync the S3 content or just copy specified files.