Google Cloud Storage for Compute Engine and Personal Backup

Using Google Cloud Storage to backup Compute Engine and Dropbox files

Google Cloud Storage (GCS) is a foundational component of Google Cloud. While GCS is a great tool for GCP users, it also acts as a storage foundation for a number of GCP services.

In this article, we’ll explore a use case for leveraging GCS to copy files from a Compute Engine (GCE) instance to a local external hard drive (and vice-versa).

Create and Prepare a GCS Bucket

First, we’ll create a GCS bucket where we’ll copy our files. Since our use case includes copying from a Compute Engine instance, we’ll also apply the appropriate permissions for the Compute Engine Service Account.

Open Cloud Shell

Start by opening a Cloud Shell instance from the Google Cloud Console — in the same project as your Compute Engine instance. To access Cloud Shell, open your web browser to https://console.cloud.google.com/. Make sure that the correct project is selected at the top left of the screen, then click the Cloud Shell icon at the top right of the screen.

With the project selected, click on the Cloud Shell icon on the right.

You should now see the Cloud Shell initialize on the bottom half of the screen. Try issuing the command:

gcloud auth list

Now that you’re in the Cloud Shell, we’ll run a few of the commands below to:

• prepare variables
• create a storage bucket
• apply permissions to the storage bucket

Prepare Variables

Next, we’ll prepare a few variables to make our job easier when we create a bucket and apply permissions.

PROJECT_ID=`gcloud config get-value project`

PROJECT_NUM=$( gcloud projects describe $PROJECT_ID --format 'value(projectNumber)' )

Create the GCS Bucket

The following command creates a GCS bucket in the us-central1 region with “autoclass” enabled.

gsutil mb -l us-central1 --autoclass gs://${PROJECT_ID}-dropboxfiles

Apply Permissions

The following command allows the compute-engine default Service Account to manage objects in the GCS bucket that we just created.

gsutil iam ch serviceAccount:${PROJECT_NUM}-compute@developer.gserviceaccount.com:objectAdmin gs://${PROJECT_ID}-dropboxfiles

That’s it for the Cloud Shell commands! Next, we’ll head on over to the Compute Engine instance where your files are stored!!

Compute Engine

In the Cloud Console, navigate to the Compute Engine dashboard — https://console.cloud.google.com/compute/

Compute Engine Scopes

First, we’ll need to update the default Compute Engine scopes to ensure that we can write to the GCS bucket that we just created.

In the list of VM Instances, click on the name of the instance where your files are currently stored.

Under Details, Scroll all the way down to the API and Identity Management section and click SHOW DETAILS.

Look for the section that says Storage. By default, this will say “Read Only”. We’ll need to change it to “Full”

In order to make this change, we’ll need to shutdown the VM. Go back to the list of instances to select the VM that we need to edit, then click “Stop”.

Once the VM has fully powered down, click the Name of the VM.

Click Edit at the top of the screen.

Scroll down to Access Scopes, select the radio button for “Set access for each API”, then scroll down to “Storage” and select “Full”

Scroll to the bottom and click the blue “SAVE” button.

Go back to the list of instances, select your VM, and click “START / RESUME”

Login to Compute Engine Instance

After the VM has completed startup, login to the Compute Engine instance by clicking the “SSH” link under “Connect” on the right of the screen.

A new tab or window will open with an SSH / terminal session on your VM.

Prepare Variables

As we did in the Cloud Shell, we’ll start by preparing variables in the VM instance. Run the following command to set a variable for the PROJECT_ID.

PROJECT_ID=`gcloud config get-value project`

Copy a test file to the GCS Bucket

Issue the following commands to create a test file and copy it to the GCS bucket to ensure that the VM has write access to the bucket

echo "Test File" > testfile.txt

gsutil cp testfile.txt gs://${PROJECT_ID}-dropboxfiles/

gsutil ls gs://${PROJECT_ID}-dropboxfiles/

You should see the output of the last command showing the file that we just copied.

To backup the files that you have stored on the Compute Engine instance, issue the following command. Let’s assume that all of your files are loacted in a sub-directory called “foo” in you home directory /home/someuser — where the full path is /home/someuse/foo. We “cd” one level below where your files are loacted (/home/someuser) and issue the following command:

gsutil cp -r foo gs://${PROJECT_ID}-dropboxfiles/

This command copies the directory foo and all of it’s contents (recursively using the -r flag) to the GCS bucket gs://${PROJECT_ID}-dropboxfiles/

Congrats! You have successfully created a backup of all of your Compute Engine files to Cloud Storage!!

Copy GCS files to Local Machine

In the case where you need to copy files that have been stored in GCS to your local machine (and a locally attached hard drive), we’ll need to installed the Google Cloud SDK on our local machine, authenticate to Google Cloud, and execute another gsutil command to copy the files.

Install the Google Cloud SDK

Follow the instructions here to install the Google Cloud SDK on your local machine. https://cloud.google.com/sdk/docs/install

Once you’ve completed installation, the final step is to run the “gcloud init” command. Be sure to login with the same account that you used earlier in the Google Cloud console when we created the GCS buckets.

Set PROJECT_ID Variable

To set the PROJECT_ID variable, we can copy it from the output of the previous steps or list all projects and copy it from here. To list your projects, issue the following command:

gcloud projects list

Copy the PROJECT_ID of the appropriate project from the list and past it into the following command:

PROJECT_ID=<<paste project name here>>

For example, I would issue the following command:
PROJECT_ID=jake-cloud-projects

You can confirm that is worked by issuing the following command:

echo $PROJECT_ID

Copy files from Cloud Storage

Now that we have the Google Cloud SDK installed on our local machine, determine the location where you’ll want to copy the files that we stored in GCS. Let’s assume that you have an external drive mounted on /mnt/external (or D:\ for Windows). We’ll issue the following command to copy the “foo” directory that we backed up earlier.

gsutil cp -r gs://${PROJECT_ID}-dropboxfiles/foo /mnt/external

OR for Windows:
gsutil cp -r gs://${PROJECT_ID}-dropboxfiles/foo D:\

Congratulations, you’ve successfully copied your files stored on GCS to your local machine.

To perform a backup of your local mahine to GCS, reverse the command above.

gsutil cp -r /mnt/external gs://${PROJECT_ID}-dropboxfiles/foo/