阿里云 OSS 迁移到 AWS S3

一、迁移工具

Amazon-s3-resumable-upload 是 AWS S3 的迁移工具,有单机版、集群版、无服务器Serverless(Lambda)版本,可以用于本地到S3上云,国内外S3互传等场景。其中如果源站是阿里云,则暂时只有单机版本支持。

部署迁移工具需要在AWS云上创建一个EC2,使用Amazon Linux 2操作系统,并安装Python3和相关依赖库。

二、环境部署

1、EC2环境选择

  • 测试阶段可选择 t2.medium 规格
  • 生产阶段建议选择 m5.large 规格 或 c5.large 规格
  • 磁盘默认8GB即可,本迁移工具不在磁盘落盘,因此不需要准备大量的EBS磁盘
  • 安全规则组选择开放TCP 22端口允许远程登录即可

2、安装基础环境

以 Amazon Linux 2 操作系统为例,执行以下命令安装。注意需要Python3版本,另外oss2是阿里云的sdk。注:部分下载过程可能比较慢,请等待一段时间。

yum update -y
yum install git
amazon-linux-extras install python3
pip3 install --upgrade pip --user
pip3 install boto3
pip3 install oss2

3、配置AWS CLI访问凭证

Amazon Linux 2 已经内置了 AWS CLI。因此执行如下命令:

aws configure

并配置正确的 Access_Key 就可。注意Region需要选择与目标S3桶一致的Region,输出格式(Output format)设置为 json即可。

4、下载脚本

执行如下命令下载脚本,将其保存在同一个目录下。

wget https://s3.cn-north-1.amazonaws.com.cn/myworkshop-lxy/oss-to-s3/s3_upload.py
wget https://s3.cn-north-1.amazonaws.com.cn/myworkshop-lxy/oss-to-s3/s3_upload_config.ini

5、设置OSS源站和S3目标桶信息。

编辑 s3_upload_config.ini 文件,找到其中如下一段,修改 DesBucket 为 S3 目标桶的名字,S3—Prefix为空,表示复制时候不筛选前缀,后边的 SrcFileIndex为*表示包含所有文件,其他字段不需要改动。

[Basic]
JobType = ALIOSS_TO_S3
DesBucket = s3bucketname
S3Prefix =
SrcFileIndex = *
DesProfileName = default

找到如下一段是OSS源站设置,修改OSS桶名、ID、Endpoint信息。

[ALIOSS_TO_S3]
ali_SrcBucket = ossbucketname
ali_access_key_id = yourkeyid
ali_access_key_secret = yourkeysecret
ali_endpoint = oss-cn-beijing.aliyuncs.com

完成后保存退出。

请注意此配置文件中其余部分,包括不是使用的Local-to-S3、S3-to-S3等模式的配置参数需要保留原状,即不要删除也不要注释掉,保持原状即可。

三、迁移数据

1、首次全量复制

首先验证 AWS CLI 工作正常。在 Root 权限下或者 User 权限下均可。执行命令时候请保持与上文配置 AWS CLI 访问凭证时候是在同一个账户下即可。执行如下命令。

aws s3 ls

确认可以显示出来目标S3桶。

执行如下命令开始复制。

python3 s3_upload.py --nogui

执行结果如下。

Reading config file: s3_upload_config.ini
Logging to file: /root/log/s3_upload-2020-08-24T04-17-55.log
Logging level: INFO
2020-08-24 04:17:55,895 INFO - Found credentials in shared credentials file: ~/.aws/credentials
2020-08-24 04:17:55,936 INFO - Checking write permission for: oss-to-s3-lxy
2020-08-24 04:17:56,004 INFO - Get source file list
2020-08-24 04:17:56,110 INFO - Get oss file list oss-to-s3-lxy
2020-08-24 04:17:56,228 INFO - Get s3 file list oss-to-s3-lxy
2020-08-24 04:17:56,253 INFO - Bucket list length:1
2020-08-24 04:17:56,253 INFO - Get unfinished multipart upload
2020-08-24 04:17:56,269 INFO - Start file: ABCDWallpaper-20200809.jpeg
--->Downloading ABCDWallpaper-20200809.jpeg - small file
2020-08-24 04:17:56,271 INFO - Start file: ABCDWallpaper-20200810.jpeg
--->Downloading ABCDWallpaper-20200810.jpeg - small file
2020-08-24 04:17:56,273 INFO - Start file: ABCDWallpaper-20200811.jpeg
--->Downloading ABCDWallpaper-20200811.jpeg - small file
2020-08-24 04:17:56,274 INFO - Start file: ABCDWallpaper-20200812.jpeg
--->Downloading ABCDWallpaper-20200812.jpeg - small file
2020-08-24 04:17:56,276 INFO - Start file: ABCDWallpaper-20200813.jpeg
--->Downloading ABCDWallpaper-20200813.jpeg - small file
    --->Uploading ABCDWallpaper-20200809.jpeg - small file
    --->Uploading ABCDWallpaper-20200811.jpeg - small file
        --->Complete ABCDWallpaper-20200809.jpeg - small file - 652.2 KB/s
2020-08-24 04:17:57,197 INFO - Start file: ABCDWallpaper-20200814.jpeg
--->Downloading ABCDWallpaper-20200814.jpeg - small file
    --->Uploading ABCDWallpaper-20200812.jpeg - small file
    --->Uploading ABCDWallpaper-20200813.jpeg - small file
        --->Complete ABCDWallpaper-20200812.jpeg - small file - 697.0 KB/s
2020-08-24 04:17:57,285 INFO - Start file: ABCDWallpaper-20200815.jpeg
--->Downloading ABCDWallpaper-20200815.jpeg - small file
        --->Complete ABCDWallpaper-20200811.jpeg - small file - 640.7 KB/s
2020-08-24 04:17:57,319 INFO - Start file: ABCDWallpaper-20200816.jpeg
--->Downloading ABCDWallpaper-20200816.jpeg - small file
        --->Complete ABCDWallpaper-20200813.jpeg - small file - 676.7 KB/s
2020-08-24 04:17:57,373 INFO - Start file: ABCDWallpaper-20200817.jpeg
--->Downloading ABCDWallpaper-20200817.jpeg - small file
    --->Uploading ABCDWallpaper-20200810.jpeg - small file
        --->Complete ABCDWallpaper-20200810.jpeg - small file - 727.8 KB/s
2020-08-24 04:17:57,977 INFO - Start file: ABCDWallpaper-20200818.jpeg
--->Downloading ABCDWallpaper-20200818.jpeg - small file
    --->Uploading ABCDWallpaper-20200814.jpeg - small file
    --->Uploading ABCDWallpaper-20200816.jpeg - small file
        --->Complete ABCDWallpaper-20200814.jpeg - small file - 752.6 KB/s
2020-08-24 04:17:58,587 INFO - Start file: ABCDWallpaper-20200819.jpeg
--->Downloading ABCDWallpaper-20200819.jpeg - small file
        --->Complete ABCDWallpaper-20200816.jpeg - small file - 729.6 KB/s
2020-08-24 04:17:58,620 INFO - Start file: ABCDWallpaper-20200820.jpeg
--->Downloading ABCDWallpaper-20200820.jpeg - small file
    --->Uploading ABCDWallpaper-20200817.jpeg - small file
        --->Complete ABCDWallpaper-20200817.jpeg - small file - 756.5 KB/s
2020-08-24 04:17:58,790 INFO - Start file: ABCDWallpaper-20200821.jpeg
--->Downloading ABCDWallpaper-20200821.jpeg - small file
    --->Uploading ABCDWallpaper-20200815.jpeg - small file
    --->Uploading ABCDWallpaper-20200821.jpeg - small file
        --->Complete ABCDWallpaper-20200815.jpeg - small file - 753.6 KB/s
        --->Complete ABCDWallpaper-20200821.jpeg - small file - 503.2 KB/s
    --->Uploading ABCDWallpaper-20200819.jpeg - small file
    --->Uploading ABCDWallpaper-20200818.jpeg - small file
        --->Complete ABCDWallpaper-20200819.jpeg - small file - 969.3 KB/s
        --->Complete ABCDWallpaper-20200818.jpeg - small file - 907.0 KB/s
    --->Uploading ABCDWallpaper-20200820.jpeg - small file
        --->Complete ABCDWallpaper-20200820.jpeg - small file - 1.3 MB/s
2020-08-24 04:17:59,832 INFO - Comparing destination and source ...
2020-08-24 04:17:59,832 INFO - Get s3 file list oss-to-s3-lxy
2020-08-24 04:17:59,853 INFO - Bucket list length:14
2020-08-24 04:17:59,853 INFO - Get oss file list oss-to-s3-lxy
2020-08-24 04:17:59,881 WARNING - All source files are in destination Bucket/Prefix. Job well done.
MISSION ACCOMPLISHED - Time: 0:00:03.956390  - FROM: oss-to-s3-lxy/ TO oss-to-s3-lxy/
Logged to file: /root/log/s3_upload-2020-08-24T04-17-55.log
PRESS ENTER TO QUIT

按回车键即可退出运行。

说明:全量复制的范围是由 s3_upload_config.ini 配置文件中的 SrcFileIndex = * 这个配置决定的。

2、增量传输

增量传输的方法与完全参数的过程一样,

python3 s3_upload.py --nogui

返回结果如下。

Reading config file: s3_upload_config.ini
Logging to file: /root/log/s3_upload-2020-08-24T04-25-22.log
Logging level: INFO
2020-08-24 04:25:22,553 INFO - Found credentials in shared credentials file: ~/.aws/credentials
2020-08-24 04:25:22,595 INFO - Checking write permission for: oss-to-s3-lxy
2020-08-24 04:25:22,665 INFO - Get source file list
2020-08-24 04:25:22,773 INFO - Get oss file list oss-to-s3-lxy

2020-08-24 04:25:22,907 INFO - Get s3 file list oss-to-s3-lxy
2020-08-24 04:25:22,936 INFO - Bucket list length:13
2020-08-24 04:25:22,936 INFO - Get unfinished multipart upload
2020-08-24 04:25:22,954 INFO - Start file: ABCDWallpaper-20200809.jpeg
2020-08-24 04:25:22,954 INFO - Duplicated. ABCDWallpaper-20200809.jpeg same size, goto next file.
2020-08-24 04:25:22,954 INFO - Start file: ABCDWallpaper-20200810.jpeg
2020-08-24 04:25:22,954 INFO - Duplicated. ABCDWallpaper-20200810.jpeg same size, goto next file.
2020-08-24 04:25:22,955 INFO - Start file: ABCDWallpaper-20200811.jpeg
2020-08-24 04:25:22,955 INFO - Duplicated. ABCDWallpaper-20200811.jpeg same size, goto next file.
2020-08-24 04:25:22,955 INFO - Start file: ABCDWallpaper-20200812.jpeg
2020-08-24 04:25:22,955 INFO - Duplicated. ABCDWallpaper-20200812.jpeg same size, goto next file.
2020-08-24 04:25:22,956 INFO - Start file: ABCDWallpaper-20200813.jpeg
2020-08-24 04:25:22,956 INFO - Duplicated. ABCDWallpaper-20200813.jpeg same size, goto next file.
2020-08-24 04:25:22,956 INFO - Start file: ABCDWallpaper-20200814.jpeg
2020-08-24 04:25:22,956 INFO - Duplicated. ABCDWallpaper-20200814.jpeg same size, goto next file.
2020-08-24 04:25:22,956 INFO - Start file: ABCDWallpaper-20200815.jpeg
2020-08-24 04:25:22,956 INFO - Duplicated. ABCDWallpaper-20200815.jpeg same size, goto next file.
2020-08-24 04:25:22,956 INFO - Start file: ABCDWallpaper-20200816.jpeg
2020-08-24 04:25:22,956 INFO - Duplicated. ABCDWallpaper-20200816.jpeg same size, goto next file.
2020-08-24 04:25:22,956 INFO - Start file: ABCDWallpaper-20200817.jpeg
2020-08-24 04:25:22,956 INFO - Duplicated. ABCDWallpaper-20200817.jpeg same size, goto next file.
2020-08-24 04:25:22,957 INFO - Start file: ABCDWallpaper-20200818.jpeg
2020-08-24 04:25:22,957 INFO - Duplicated. ABCDWallpaper-20200818.jpeg same size, goto next file.
2020-08-24 04:25:22,957 INFO - Start file: ABCDWallpaper-20200819.jpeg
2020-08-24 04:25:22,957 INFO - Duplicated. ABCDWallpaper-20200819.jpeg same size, goto next file.
2020-08-24 04:25:22,957 INFO - Start file: ABCDWallpaper-20200820.jpeg
2020-08-24 04:25:22,957 INFO - Duplicated. ABCDWallpaper-20200820.jpeg same size, goto next file.
2020-08-24 04:25:22,957 INFO - Start file: ABCDWallpaper-20200821.jpeg
--->Downloading ABCDWallpaper-20200821.jpeg - small file
    --->Uploading ABCDWallpaper-20200821.jpeg - small file
        --->Complete ABCDWallpaper-20200821.jpeg - small file - 976.7 KB/s
2020-08-24 04:25:23,230 INFO - Comparing destination and source ...
2020-08-24 04:25:23,230 INFO - Get s3 file list oss-to-s3-lxy
2020-08-24 04:25:23,277 INFO - Bucket list length:14
2020-08-24 04:25:23,277 INFO - Get oss file list oss-to-s3-lxy
2020-08-24 04:25:23,314 WARNING - All source files are in destination Bucket/Prefix. Job well done.
MISSION ACCOMPLISHED - Time: 0:00:00.696466  - FROM: oss-to-s3-lxy/ TO oss-to-s3-lxy/
Logged to file: /root/log/s3_upload-2020-08-24T04-25-22.log

复制完成。可以看到在遍历且对比后,仅复制了一个增量文件。

3、补充说明

(1)使用虚拟终端

建议使用screen虚拟终端,避免复制过程网络断线。Amazon Linux 2已经内置了screen组件。

执行 screen 启动一个新的虚拟终端。单网络断线时候,执行 screen -x 即可连接回之前的虚拟终端。

(2)分目录(前缀)进行拷贝

为了降低增量复制时候遍历整个存储桶带来的不必要负载,建议在迁移期间,对OSS存储桶采取修改前缀即新建目录的办法。例如当前数据在pic1目录下,当准备进行全量复制前,修改程序代码,将OSS存储桶内新生成的文件都保存到pic2目录中,pic1不再产生增量。这样可以避免下次增量同步产生遍历所有文件的开销。

为了复制存储桶内的目录,需要修改配置文件,需要修改 s3_upload_config.ini 配置文件,将其中的配置改为如下。

S3Prefix = pic1
SrcFileIndex = *

这样的两行组合表示只同步pic1目录下的所有文件。全量同步完成后,后续再同步pic2目录,即可有效降低开销。

(3)同步特定字符串开头的文件

假设在存储桶的根目录下有若干文件ABCDWallpaper-xxxx.jpg,都是以ABCD开头,那么修改 s3_upload_config.ini 配置文件如下。

S3Prefix = ABCD
SrcFileIndex = *

执行复制,将只同步以ABCD开头的文件。

四、参考资料

Amazon-s3-resumable-upload 项目Github网址。

https://github.com/aws-samples/amazon-s3-resumable-upload