Corey's Projects: S3 Scripts


	Professional Software Consulting

Amazon Web Services S3 Utilities

S3 (Simple Secure Storage) is Amazon's cloud computing product that enables anyone to store any amount of data for as long as you want for a very reasonable price. For more about Amazon's cloud products, click here.

Amazon provides a REST or SOAP interface to their cloud. That means that only programs can easily access their system. As a developer, your choice is to either write your own program, or use someone else's. I have done both.

I needed S3 storage because I was designing a system that might, over time, be required to house thousands of files totalling multiple terabytes in size. While my web server needed to access S3, so did my back-end programs and scripts. I wanted a solution with a simple installation process, one that easily deployable, and most importantly, one that gave my programs a uniform interface no matter where they are in the system.

Although there is a PEAR S3 class, I was looking for something that could be used without requiring PEAR on every production system for which I'm responsible. Using the AWS forums, I found someone in South Africa who had developed a rudimentary class that I could use as a starting point. It is a simple file, called S3.php.

While this worked fine for PHP, it didn't work so well for my KSH scripts and other non-PHP programs. So I wrote a set of wrapper scripts that provided a command line API. With this, I would have the versatility to access my S3 files from any point in my system, as well as on the command line in case I wanted to check or fix something manually.

After enhancing the S3.php class so that error checking conformed to my standards (and sending my changes to the script's author), I had a set of scripts that worked very well to do all the things from the command line that Amazon offerred with regards to S3. In addition to these scripts, I used a Firefox add-on called S3Fox as a check that my scripts were working properly.

View the code
S3.php	The REST API core code
s3_bucketContent.php	Lists the objects (files) within a bucket
s3_bucketCreate.php	Creates a new bucket
s3_bucketDestroy.php	Destroys a bucket you own
s3_bucketList.php	Provides a list of your buckets
s3_fileDelete.php	Deletes an object (file) from a bucket
s3_fileDownload.php	Downloads a file from S3 to your computer
s3_fileInfo.php	Provides information about an S3 file
s3_fileUpload.php	Uploads a file from your computer to S3
cli_args.php	Function to convert CLI arguments to an array

Download the code
s3_utils.tar.gz	7.97 KB	s3_utils.zip	18.74 KB
Note: Scripts are designed with a unix-like usage statement; invoke the script with no parameters, and it will tell you what parameters are available as well as which are required.
Note: While scripts are designed to return 0 (success) or 1 (failure), because of how Amazon has constructed their API, sometimes errors are not returned even though they should. For example, I would expect s3_bucketCreate.php to return an error if the bucket already exists, but it doesn't.

Something wrong with this page or this site? Let the webmaster know by clicking HERE