This job is a BigQuery SQL statement that returns a boolean ie: “true” or “false”.
The step will only complete once the result of the statement evaluates to “true”.
You would use this anytime you need the step to only complete when some condition is met.
Effectively you are able to pause or delay pipelines by using BigQuery SQL to define a condition to be met.
An example SQL statement would be:
“SELECT EXISTS (SELECT 1 FROM your-project.yourdataset.yourtable
WHERE COMPLETED_TIME >= TIMESTAMP_ADD(CURRENT_TIMESTAMP(),INTERVAL -1 hour)”
This is often useful when you want your pipeline to only start once some other processing is done. Eg: process something then insert a row into a status or logging table. Then use the above statement to query your log table.
In the following example, Step 2 will only execute once the condition specified in Step 1 is met:
