Contrary to popular belief, there is a strong case for small and medium size firms to use the cloud! The argument that the cloud is only for large corporates and multinationals is a fallacy. In fact, the nimble nature of small and mediums makes cloud adoption even more attractive.
Orchestration tools like Apache Airflow are great but are also more general purpose. You have the option of hosting yourself or using a managed service from GCP and AWS. However these tools require Python programming skills to use. rockstarETL only requires that you know SQL. There is no programming language required. rockstarETL is focused on …
In a word? Security. Sure a vm is a little more administration to setup and manage (really quite a tiny amount in fact) but what you get is a resource that you have ultimate control over. You can turn it off whenever you like and you can manage the SSL certificates and the firewall rules …
So we’ve got our raw data in s3: And we have an Athena table over this data: The Athena query statement used to create this table: CREATE EXTERNAL TABLE invoices (invoiceid int,customerid int,billtocustomerid int,orderid int,deliverymethodid int,contactpersonid int,accountspersonid int,salespersonpersonid int,packedbypersonid int,invoicedate date,customerpurchaseordernumber string,iscreditnote int,totaldryitems int,totalchilleritems int,deliveryrun string,runposition string,confirmeddeliverytime timestamp,confirmedreceivedby string,lasteditedby int,lasteditedwhen timestamp)ROW FORMAT DELIMITEDFIELDS TERMINATED BY …
Using the Athena Batch Partition Insert Job Type Read More »
First we’ll create a new database to house the final tables Then we’ll change to the newly created database: And we’ll insert from our staging database into a table in the final database: First we’ll just select everything: Notice how without the partition pruning we scan 6.21 MB: Whereas when we include the partition filters …
Check that none of your jobs which used to be set to step 1 doesn’t have a non-zero “Number of Runs per day”
Suppose you added two jobs to step 1 as follows: In order for the pipeline to run, it is necessary to only have one of the job steps have a non-zero “Runs per day” It is not necessary to have both jobs with a non-zero “Runs per day” This pipeline will run each job one …
This job type delete objects from the bucket with the optional prefix you supply. This is useful for clearing a staging area (eg: after processing with other Athena job types) “Run Time” is the Time in UTC time zone when you want the job to run. “Number of Runs per day” is the number of times per …
This job type allows you to copy S3 objects from one bucket to another. This is useful for staging purposes, especially for running other Athena job types using this location. “Run Time” is the Time in UTC time zone when you want the job to run. “Number of Runs per day” is the number of times per …
This job type allows you to build and run SQL statements dynamically. This is extremely powerful because it allows you to remove the monotony of creating many SQL statements. “Run Time” is the Time in UTC time zone when you want the job to run. “Number of Runs per day” is the number of times per day …