rockstarETL (Rest API) vs DataFlow vs Client libraries? DataFlow I would not recommend it for batch loads. It’s just too slow. DataFlow has it’s place if you want to stream. Just bear in mind, streaming has a cost associated with it! And using rockstarETL is so performant, you have to seriously wonder if the additional …
What is the best way to load data into BigQuery? Read More »
BigQuery supports User Defined Functions written in SQL or Javascript. This is such a nice feature since you are effectively freed from SQL limitations. Or if a built in function doesn’t exist or isn’t serving your purposes, you can simply create your own using Javascript and use it inside of your SQL queries or views! …
But what is orchestration? rockstarETL’s pipelines allow you to define processing steps. Multiple jobs can execute simultaneously in the same step. Jobs in subsequent steps will only begin once the previous steps have executed. rockstarETL also allows you to schedule your pipeline to run at a certain time and up to every 5 minutes: Task …
With the SQL job type, rockstarETL simply runs your query. You should use a DML or BQ ML statement. A select statement won’t display the result back to you. The reason is because with rockstarETL, you build pipelines, which run the steps at the time and frequency you specify. How do I use the SQL job type? Write your SQL …
It’s dynamic SQL!
BigQuery is the goal. The promised land. It is a marvel of engineering and features. Performance and usability are truly impressive. Just think, being able to run cloud data warehousing, ad hoc exploratory analysis AND machine learning! All this from one platform! Incredible. And then Google goes and throws in TensorFlow as well! Mind = blown! 🙂 …
How do I get my data into BigQuery? Let’s compare the options for loading data in BigQuery. Factors we’ll use are: Price Features Complexity and Risks If you look at Google’s Business Intelligence guide: We’ll compare the first four options above and then we’ll take a look at the Partner options. Cloud Dataflow and Cloud Dataprep …
rockstarETL is a one-stop, codeless ETL/ELT orchestration platform. Pipelines -> Steps -> Jobs rockstarETL enables you to define pipelines. A pipeline is a sequence of steps executed in order from Step 1 through to the last step specified. Each step can have multiple jobs. These will be executed in parallel. Below, in Step 1, there are …
rockstarETL often asks you to specify a path when configuring your jobs. Eg: This is a text or json file you need to create. Inside should be either the SQL query statement. You then save this file in your Cloud Storage Bucket and tell rockstarETL where to find it. rockstarETL then executes the SQL statement …
The Load Job type let’s you load multiple files from your Cloud Storage bucket into separate BigQuery tables in one step! So you could load thirty to fifty BigQuery tables from the same number of different source folders. In addition, you could have thousands of files in each of these thirty to fifty sub-folders. The documentation explains …