s3cmdを使う
たまに使うのでs3cmdの備忘録です。
今回はAmazon Linux上で、s3cmd-1.5.0-beta1を使っています。
準備
まずはS3にアクセスできるユーザとS3のバケットを用意します。
S3にアクセスできる権限を持ったユーザを作っておくことにします。
s3cmdでバケットを作成することも可能ですが、今回は事前にバケットを作成しておきます。
ログもとるようにしておきました。
インストールと設定
インストールは簡単です。
# wget http://downloads.sourceforge.net/project/s3tools/s3cmd/1.5.0-beta1/s3cmd-1.5.0-beta1.tar.gz # tar -xvzf s3cmd-1.5.0-beta1.tar.gz # python setup.py install
次に設定ファイルを作成します。
$ s3cmd --configure Enter new values or accept defaults in brackets with Enter. Refer to user manual for detailed description of all options. Access key and Secret key are your identifiers for Amazon S3 Access Key: <Your Access Key> Secret Key: <Your Secret Key> Encryption password is used to protect your files from reading by unauthorized persons while in transfer to S3 Encryption password: <Your Encryption password> Path to GPG program [/usr/bin/gpg]: When using secure HTTPS protocol all communication with Amazon S3 servers is protected from 3rd party eavesdropping. This method is slower than plain HTTP and can't be used if you're behind a proxy Use HTTPS protocol [No]: On some networks all internet access must go through a HTTP proxy. Try setting it here if you can't conect to S3 directly HTTP Proxy server name: New settings: Access Key: <Your Access Key> Secret Key: <Your Secret Key> Encryption password: <Your Encrypption password> Path to GPG program: /usr/bin/gpg Use HTTPS protocol: False HTTP Proxy server name: HTTP Proxy server port: 0 Test access with supplied credentials? [Y/n] Y Please wait, attempting to list all buckets... Success. Your access key and secret key worked fine :-) Now verifying that encryption works... Success. Encryption and decryption worked fine :-) Save settings? [y/N] y Configuration saved to '/home/ec2-user/.s3cfg'
使ってみる(Linux版)
ファイルの操作は直感的にできます。
$ s3cmd ls s3://think-t/work/1M.img 2014-05-25 01:23 1024000 s3://think-t/work/1M.img $ s3cmd put 1M.img s3://think-t/work/ 1M.img -> s3://think-t/work/1M.img [1 of 1] 1024000 of 1024000 100% in 0s 5.01 MB/s done $ s3cmd get s3://think-t/work/1M.img 1M.img s3://think-t/work/1M.img -> 1M.img [1 of 1] 1024000 of 1024000 100% in 0s 6.05 MB/s done $ s3cmd del s3://think-t/work/1M.img File s3://think-t/work/1M.img deleted
rsyncのような使い方も。
$ s3cmd sync work s3://think-t/ work/tmp1/1M.img -> s3://think-t/work/tmp1/1M.img [1 of 9] 1024000 of 1024000 100% in 0s 6.13 MB/s done work/tmp1/2M.img -> s3://think-t/work/tmp1/2M.img [2 of 9] 2048000 of 2048000 100% in 0s 10.40 MB/s done work/tmp1/5M.img -> s3://think-t/work/tmp1/5M.img [3 of 9] 5120000 of 5120000 100% in 0s 13.51 MB/s done work/tmp2/10M.img -> s3://think-t/work/tmp2/10M.img [4 of 9] 10240000 of 10240000 100% in 0s 18.02 MB/s done work/tmp2/20M.img -> s3://think-t/work/tmp2/20M.img [part 1 of 2, 15MB] 15728640 of 15728640 100% in 0s 22.55 MB/s done work/tmp2/20M.img -> s3://think-t/work/tmp2/20M.img [part 2 of 2, 4MB] 4751360 of 4751360 100% in 0s 18.15 MB/s done work/tmp2/50M.img -> s3://think-t/work/tmp2/50M.img [part 1 of 4, 15MB] 15728640 of 15728640 100% in 0s 20.92 MB/s done (略) work/tmp2/50M.img -> s3://think-t/work/tmp2/50M.img [part 4 of 4, 3MB] 4014080 of 4014080 100% in 0s 18.28 MB/s done work/tmp3/100M.img -> s3://think-t/work/tmp3/100M.img [part 1 of 7, 15MB] 15728640 of 15728640 100% in 0s 23.07 MB/s done (略) work/tmp3/100M.img -> s3://think-t/work/tmp3/100M.img [part 7 of 7, 7MB] 8028160 of 8028160 100% in 0s 8.07 MB/s done work/tmp3/200M.img -> s3://think-t/work/tmp3/200M.img [part 1 of 14, 15MB] 15728640 of 15728640 100% in 1s 9.13 MB/s done (略) work/tmp3/200M.img -> s3://think-t/work/tmp3/200M.img [part 14 of 14, 320kB] 327680 of 327680 100% in 0s 4.67 MB/s done work/tmp3/500M.img -> s3://think-t/work/tmp3/500M.img [part 1 of 33, 15MB] 15728640 of 15728640 100% in 1s 8.82 MB/s done (略) work/tmp3/500M.img -> s3://think-t/work/tmp3/500M.img [part 33 of 33, 8MB] 8683520 of 8683520 100% in 1s 5.42 MB/s done Process files that was not remote copied Done. Uploaded 909312000 bytes in 95.0 seconds, 9.12 MB/s. Copied 0 files saving 0 bytes transfer.
デフォルトだと15MB以上のファイルは自動で分割されます。
(.s3cfgで「enable_multipart = True」「multipart_chunk_size_mb = 15」)
$ s3cmd put 6G.img s3://think-t/work/ 6G.img -> s3://think-t/work/6G.img [part 1 of 391, 15MB] 15728640 of 15728640 100% in 1s 12.16 MB/s done 6G.img -> s3://think-t/work/6G.img [part 2 of 391, 15MB] 15728640 of 15728640 100% in 1s 8.11 MB/s done 6G.img -> s3://think-t/work/6G.img [part 3 of 391, 15MB] (略) 6G.img -> s3://think-t/work/6G.img [part 391 of 391, 9MB] 9830400 of 9830400 100% in 1s 7.38 MB/s done
S3の場合はリクエストに課金が発生するので、大きいファイルを処理する場合は、
「--multipart-chunk-size-mb」オプションを付けるか、.s3cfgの設定を見直して、
分割サイズをコントロールするのが良さそうです。
$ s3cmd put --multipart-chunk-size-mb=5120 6G.img s3://think-t/work/ 6G.img -> s3://think-t/work/6G.img [part 1 of 2, 5GB] 5368709120 of 5368709120 100% in 624s 8.20 MB/s done 6G.img -> s3://think-t/work/6G.img [part 2 of 2, 739MB] 775290880 of 775290880 100% in 90s 8.14 MB/s done
infoオプションでファイルの情報をより詳細に取得できます。
$ s3cmd info s3://think-t/work/1M.img s3://think-t/work/1M.img (object): File size: 1024000 Last mod: Sun, 25 May 2014 01:29:15 GMT MIME type: application/octet-stream; charset=binary MD5 sum: 80ec129d645c70cf0de45b1a5a682235 SSE: NONE policy: none ACL: ----: FULL_CONTROL
ハッシュ値だけなら、「--list-md5」オプションを付ければOK
$ s3cmd ls --list-md5 s3://think-t/work/1M.img 2014-05-25 01:29 1024000 80ec129d645c70cf0de45b1a5a682235 s3://think-t/work/1M.img
全てのディレクトリを表示します。
$ s3cmd la --bucket-location=US s://think-t/work/
DIR s3://think-t/logs/
DIR s3://think-t/work/
バケットのサイズを表示します。
$ s3cmd du s3://think-t/ 1166628 s3://think-t/
やっぱり便利です。
今日はこんなところで。
