rockstarETL (Rest API) vs DataFlow vs Client libraries? DataFlow I would not recommend it for batch loads. It’s just too slow. DataFlow has it’s place if you want to stream. Just bear in mind, streaming has a cost associated with it! And using rockstarETL is so performant, you have to seriously wonder if the additional …
What is the best way to load data into BigQuery? Read More »
How do I get my data into BigQuery? Let’s compare the options for loading data in BigQuery. Factors we’ll use are: Price Features Complexity and Risks If you look at Google’s Business Intelligence guide: We’ll compare the first four options above and then we’ll take a look at the Partner options. Cloud Dataflow and Cloud Dataprep …
The Load Job type let’s you load multiple files from your Cloud Storage bucket into separate BigQuery tables in one step! So you could load thirty to fifty BigQuery tables from the same number of different source folders. In addition, you could have thousands of files in each of these thirty to fifty sub-folders. The documentation explains …
While Cloud Storage Transfer does offer support for loading data on a schedule, there are some important limitations. Let’s take a look: rockstarETL does not share these limitations! Waiting an hour before being able to use your data may be a deal-breaker! By contrast, rockstarETL will detect your source files immediately: As soon as they are copied to …