BigQuery is the goal. The promised land.
It is a marvel of engineering and features. Performance and usability are truly impressive.
Just think, being able to run cloud data warehousing, ad hoc exploratory analysis AND machine learning! All this from one platform! Incredible.
And then Google goes and throws in TensorFlow as well! Mind = blown! 🙂
But, like anywhere worth going to, without the correct guide, gear and map, things can get a little, well, let’s just say, hectic!
To enjoy this promised land requires the small but essential step of first getting your data into BigQuery. And then continually loading in new data on a regular basis.
For any workflow, whether exploratory or mature enterprise, the key is loading data consistently and reliably.
If you attempt to use existing tools: this, quite simply, is a large and costly challenge.
There are also a myriad of options. Which do you choose? How do you identify and evaluate the pitfalls and limitations of each approach?
rockstarETL solves both the complexity and the exorbitant cost of other products.
rockstarETL is simple to use. Your data pipelines are run consistently and reliably.
rockstarETL provides both ETL and ELT paradigms. ELT is recommended.
ELT gives you all the control, because you simply load your raw data from Cloud Storage into one BigQuery dataset. Thereafter you transform this data from this dataset to a final dataset by running DML queries. These queries can be written into Views.
Thus all your logic lives in a BigQuery dataset of your choosing.