While Cloud Storage Transfer does offer support for loading data on a schedule, there are some important limitations.
Let’s take a look:
rockstarETL does not share these limitations!
Waiting an hour before being able to use your data may be a deal-breaker!
By contrast, rockstarETL will detect your source files immediately: As soon as they are copied to your Cloud Storage Bucket, they can be loaded. (Naturally, you need to have a load job start running as part of your data pipeline 😉 )
rockstarETL does not need you to create the table in advance! rockstarETL creates the destination table as part of the load job, if it doesn’t already exist!
The most important thing to note is:
Data pipelines need to be an end to end coherent sequence of steps.
The problem with using different services to schedule different parts of your data pipelines is stitching these together. ETL/ELT requires strict enforcement of dependencies. This is what rockstarETL does for you.