How to import and export your pipeline jobs between instances

If you should ever need to move your jobs between instances, eg: if there’s an update, this is what you need to do:

First SSH into your instance:

On AWS, select your instance and then “Connect”

Then SSH client:

On GCP, from the SSH button:

My preference is to view the gcloud command:

Take out the “beta” and paste it into powershell. Upon hitting enter, this fires up a new putty session.

Once SSH’d into your instance, you need to log into mysql (mariadb) as root using:

sudo mysql -u root -p

You’ll need the root password to do this 😉

Once logged in as root, you need to change the database to “maxil” using:

use maxil;

Then you need to export the Jobs from the database to a local file (on the instance) eg:

SELECT * from Jobs INTO OUTFILE '/tmp/jobs.txt' FIELDS TERMINATED BY ',' OPTIONALLY ENCLOSED BY '"' LINES TERMINATED BY '\n';

In another SSH session, you can verify the file exists:

cd /tmp/

then:

ls

Then to preview the file:

cat jobs.txt

Thereafter you need to copy this file to either your S3 bucket on AWS or your Google Cloud Storage bucket:

On AWS:

aws s3 cp jobs.txt s3://rockstaretlstaging/jobs/

On GCP:

gsutil cp jobs.txt gs://adeptsentinalstaging20191216/jobs/

Now you need to copy the file from your bucket to your new instance:

On AWS:

aws s3 cp s3://rockstaretlstaging/jobs/jobs.txt /tmp/jobs.txt

On GCP:

gsutil cp gs://adeptsentinalstaging20191216/jobs/jobs.txt /tmp/jobs.txt

Now that the jobs file is on your new instance, the final step is to import it back into the mysql database (mariadb):

While logged into mysql, issue the following command:

LOAD DATA INFILE '/tmp/jobs.txt'
INTO TABLE Jobs
FIELDS TERMINATED BY ','
ENCLOSED BY '"'
LINES TERMINATED BY '\n';

If you’ve edited the file on Windows, replace the last line with

LINES TERMINATED BY '\n';

before importing back into the database.

You can confirm the imports success by:

select * from Jobs;

Your jobs and pipelines should appear in the new instance exactly as they were in your old instance!