But what is orchestration? rockstarETL’s pipelines allow you to define processing steps. Multiple jobs can execute simultaneously in the same step. Jobs in subsequent steps will only begin once the previous steps have executed. rockstarETL also allows you to schedule your pipeline to run at a certain time and up to every 5 minutes: Task …
With the SQL job type, rockstarETL simply runs your query. You should use a DML or BQ ML statement. A select statement won’t display the result back to you. The reason is because with rockstarETL, you build pipelines, which run the steps at the time and frequency you specify. How do I use the SQL job type? Write your SQL …
It’s dynamic SQL!
rockstarETL is a one-stop, codeless ETL/ELT orchestration platform. Pipelines -> Steps -> Jobs rockstarETL enables you to define pipelines. A pipeline is a sequence of steps executed in order from Step 1 through to the last step specified. Each step can have multiple jobs. These will be executed in parallel. Below, in Step 1, there are …
rockstarETL often asks you to specify a path when configuring your jobs. Eg: This is a text or json file you need to create. Inside should be either the SQL query statement. You then save this file in your Cloud Storage Bucket and tell rockstarETL where to find it. rockstarETL then executes the SQL statement …
This job is a BigQuery SQL statement that returns a boolean ie: “true” or “false”. The step will only complete once the result of the statement evaluates to “true”. You would use this anytime you need the step to only complete when some condition is met. Effectively you are able to pause or delay pipelines …