S3 Delete Objects

This job type delete objects from the bucket with the optional prefix you supply.

This is useful for clearing a staging area (eg: after processing with other Athena job types)

“Run Time” is the Time in UTC time zone when you want the job to run.

“Number of Runs per day” is the number of times per day you want the job to run. This could be anywhere from 1 to 1440 times per day (every minute is currently the maximum)

Fill in the “Email address” to be receive job run notifications by email. This only works if you’ve signed up for SendGrid and completed the details in the SendGrid section.

“Pipeline Name” groups jobs together to run consecutively and dependently. This means subsequent job steps eg: Step 2 will only run once Step 1 has completed successfully. To change the Pipeline that the job belongs to, simply change the Pipeline Name field.

“Region” is the AWS region where you want the query to run.

“Job Name” is the unique name of the job. It can’t exist even in a different pipeline.

“Bucket”

Name of the S3 bucket containing the objects you want to delete (exlude the s3:// prefix)

“Prefix”

(Optional) The prefix of the files you want to delete from S3 (exclude the s3:// prefix). eg: path/to/my/text/

“Sub Folder Names”

(Optional) The pipe ie: | separated list of sub folders to delete eg: myfolder1|myfolder2

This is useful for only specifying specific folder to remove the contents of as opposed to all object with an optional prefix.

If you don’t specify a prefix and subfolder names, then all objects will be removed from the bucket.

Leave a Comment