How to access Amazon S3 cloud storage from command line in Linux
Amazon S3 is a cloud storage provided by Amazon Web Services (AWS). Amazon S3 publishes a set of web services interfaces, upon which many third-party commercial services or client software are developed.
In this tutorial, I will describe how to access Amazon S3 cloud storage from the command line in Linux.
One of the most popular Amazon S3 command line clients is s3cmd, which is written in python. As a simple AWS S3 command line tool, s3cmd is ideal to use when you want to run scripted cron jobs such as daily backups.
Install s3cmd on Linux
To install s3cmd on Ubuntu or Debian:
$ sudo apt-get install s3cmd
To install s3cmd on Fedora:
$ sudo yum install s3cmd
To install s3cmd on CentOS or RHEL, download .rpm package from its official site, and install it manually. For 64-bit CentOS or RHEL 6:
$ sudo rpm -ivh s3cmd-1.0.0-4.1.x86_64.rpm
When you run s3cmd for the first time, you need to configure it by running the following command.
$ s3cmd --configure
It will ask you for a series of questions:
access key and secret key for AWS S3 encryption password for encrypted data transfer to and from AWS S3. path to GPG program used to encrypt data (e.g., /usr/bin/gpg) whether to use HTTPS protocol name and port of HTTP proxy if used
Configuration will then be saved as a plain text in ~/.s3cfg.
Basic Usage of s3cmd
To list all existing buckets in your AWS S3 account:
$ s3cmd ls
2011-05-28 22:30 s3://mybucket1
2011-05-29 00:14 s3://mybucket2
To create a new bucket:
$ s3cmd mb s3://dev99 Bucket 's3://dev99/' created
To upload files to an existing bucket:
$ s3cmd put 1.png 2.png 3.png s3://dev99 1.png -> s3://dev99/1.png [1 of 3] 26261 of 26261 100% in 5s 4.33 kB/s done 2.png -> s3://dev99/2.png [2 of 3] 201430 of 201430 100% in 2s 98.05 kB/s done 3.png -> s3://dev99/3.png [3 of 3] 46630 of 46630 100% in 0s 56.62 kB/s done
The default access permission for the uploaded files is "private", meaning that only you, with correct access and secret keys, will be able to access the files.
To upload files to an existing bucket with public access permission:
$ s3cmd put --acl-public 4.png s3://dev99 4.png -> s3://dev99/4.png [1 of 1] 30778 of 30778 100% in 8s 3.34 kB/s done Public URL of the object is: http://dev99.s3.amazonaws.com/4.png
With public access permission granted, the uploaded file can be accessed by anyone by going to http://dev99.s3.amazonaws.com/4.png on any web browser.
To view the content of an existing bucket:
$ s3cmd ls s3://dev99 2013-06-02 02:52 26261 s3://dev99/1.png 2013-06-02 02:52 201430 s3://dev99/2.png 2013-06-02 02:52 46630 s3://dev99/3.png 2013-06-02 02:56 30778 s3://dev99/4.png
To download files (e.g., all .png files) contained in an existing bucket:
$ s3cmd get s3://dev99/*.png s3://dev99/1.png -> ./1.png [1 of 4] 26261 of 26261 100% in 0s 39.39 kB/s done s3://dev99/2.png -> ./2.png [2 of 4] 201430 of 201430 100% in 7s 24.64 kB/s done s3://dev99/3.png -> ./3.png [3 of 4] 46630 of 46630 100% in 1s 39.34 kB/s done s3://dev99/4.png -> ./4.png [4 of 4] 30778 of 30778 100% in 0s 97.01 kB/s done
To remove files in an existing bucket:
$ s3cmd del s3://dev99/*.png File s3://dev99/1.png deleted File s3://dev99/2.png deleted File s3://dev99/3.png deleted File s3://dev99/4.png deleted
To get information about an existing bucket, including its storage location and access control lists:
$ s3cmd info s3://dev99 s3://dev99/ (bucket): Location: us-east-1 ACL: dan.nanni: READ ACL: dan.nanni: WRITE ACL: dan.nanni: READ_ACP ACL: dan.nanni: WRITE_ACP
To encrypt a file before uploading it to an existing bucket:
$ s3cmd -e put encrypt.png s3://dev99 /tmp/tmpfile-pzT1zV3kLZlxDwqA0kwy -> s3://dev99/encrypt.png [1 of 1] 196890 of 196890 100% in 1s 99.51 kB/s done
When downloading an encrypted file with s3cmd, it will automatically detect the encryption, and decrypt the file on the fly upon downloading it. Thus, to download and access an encrypted file, simply run as usual:
$ s3cmd get s3://dev99/encrypt.png s3://dev99/encrypt.png -> ./encrypt.png [1 of 1] 196890 of 196890 100% in 1s 131.29 kB/s done
To remove an existing bucket:
$ s3cmd rb s3://dev99 Bucket 's3://dev99/' removed
Note that you cannot remove a bucket if it is not empty.