Writing A Python Script To Send Files To Amazon S3

Amazon Simple Storage Service or Amazon S3 is a storage service with a web API. I use Amazon S3 to store backups of my blog and other sites. I made a simple python script to handle file uploads to S3.

In order to use Amazon S3, first create a bucket using your Amazon AWS account. As the name suggests, bucket is a container. You can create buckets using the AWS management console.

The script we're going to write will take two input parameters

  1. Path to file, local
  2. Target S3 path

Amazon S3 buckets don't support hierarchical directories. To simulate such a file system, you can use '/' in the target file name.

To use AWS from Python, boto is the de facto recommended library. You can install boto from your distribution package management system or download from code.google.com repository.

Import the libraries:

#!/usr/bin/env python
import os
import sys
from boto.s3.connection import S3Connection
from boto.s3.key import Key

Read the command line arguments:

local_file_path = sys.argv[1]
s3_path = sys.argv[2]

Using AWS Identity and Access Management (IAM) service, create an account which you can use from scripts. Make a note of the key and secret of the newly created account.

I have named my bucket 'j_bucket'.

Initialize the connection:

conn = S3Connection('your account key', 'your account secret')
pb = conn.get_bucket('j_backups')

Now, we're ready to create an S3 key. Once the key object is created, wet set the contents of the key and also the name of the key.

k = Key(pb)
file_name_to_use_in_s3 = "%s/%s"%(s3_path, os.path.basename(local_file_path))
k.name = file_name_to_use_in_s3

Save the file as send-to-s3.py and set execute permissions. We can now use the script like:

./send-to-s3.py path/to/my/local/file "mystore/mybackups"

In this example. the file "path/to/my/local/file" will be transferred to S3 bucket with the name "mystore/mybackups".

The script is also available on github.

Add new comment

Filtered HTML

  • Web page addresses and e-mail addresses turn into links automatically.
  • Allowed HTML tags: <a> <em> <strong> <cite> <blockquote> <code> <ul> <ol> <li> <dl> <dt> <dd>
  • Lines and paragraphs break automatically.

Plain text

  • No HTML tags allowed.
  • Web page addresses and e-mail addresses turn into links automatically.
  • Lines and paragraphs break automatically.
By submitting this form, you accept the Mollom privacy policy.