Suppose you added two jobs to step 1 as follows: In order for the pipeline to run, it is necessary to only have one of the job steps have a non-zero “Runs per day” It is not necessary to have both jobs with a non-zero “Runs per day” This pipeline will run each job one …
This job type delete objects from the bucket with the optional prefix you supply. This is useful for clearing a staging area (eg: after processing with other Athena job types) “Run Time” is the Time in UTC time zone when you want the job to run. “Number of Runs per day” is the number of times per …
This job type allows you to copy S3 objects from one bucket to another. This is useful for staging purposes, especially for running other Athena job types using this location. “Run Time” is the Time in UTC time zone when you want the job to run. “Number of Runs per day” is the number of times per …
This job type allows you to build and run SQL statements dynamically. This is extremely powerful because it allows you to remove the monotony of creating many SQL statements. “Run Time” is the Time in UTC time zone when you want the job to run. “Number of Runs per day” is the number of times per day …
This job type circumvents the 100 partition insert limit. See https://docs.aws.amazon.com/athena/latest/ug/ctas-insert-into.html It does this by calculating how many partitions in total will be affected by your query. If there are more than 100 partitions to insert into, it will run the re-run the query iterating until all partitions are inserted to. Each run will only …
This job type is useful for waiting for a condition to be met. Your SQL statement must be written so that it only returns “true” or “false” This job Type runs an Athena Query and checks the result. eg: select true eg: select case when 1=1 then true else false end as result Only once …
This job type lets you run multiple Athena queries in one step: “Run Time” is the Time in UTC time zone when you want the job to run. “Number of Runs per day” is the number of times per day you want the job to run. This could be anywhere from 1 to 1440 times per day …
There are two ways to check the logs of your jobs that run within pipelines: Login to the EC2 instance: Username is “ec2-user” When you’re logged in, it should look similar to: The command to run is: sudo tail -f /var/log/tomcat/catalina.out The output will look like the following when there are no pipelines to run: …
This is a simple no frills query that you want to run using Athena: “Run Time” is the Time in UTC time zone when you want the job to run. “Number of Runs per day” is the number of times per day you want the job to run. This could be anywhere from 1 to 1440 …
When starting out you’ll have no jobs saved yet: All you need to do is select the “Add Step” button: Thereafter you’ll be taken to the page where you can add a job step: On the Job Type selector button, you’re able to select which type of job you’d like to add: There are various …