github.com/r-medina/s3-benchmark

S3 Benchmark

Your Amazon S3 performance depends on 3 things:

1. Your distance to the S3 endpoint.
2. The size of your objects.
3. The number of parallel transfers you can make.

With this tool you can measure S3's performance from any location, using different thread counts and object sizes. Here's an example from a c5.4xlarge EC2 instance:

Usage

Download

macOS

curl -OL https://github.com/dvassallo/s3-benchmark/raw/master/build/darwin-amd64/s3-benchmark

Linux 64-bit x86

curl -OL https://github.com/dvassallo/s3-benchmark/raw/master/build/linux-amd64/s3-benchmark

Linux 64-bit ARM

curl -OL https://github.com/dvassallo/s3-benchmark/raw/master/build/linux-arm64/s3-benchmark

Credentials

This tool needs AWS credentials with full S3 permissions. If you run this on EC2, it will automatically use your EC2 instance profile. Otherwise it will try to find the credentials from the usual places.

Run

Make the file executable:

chmod +x s3-benchmark

Run a quick test (takes a few minutes):

./s3-benchmark

Or run the full test (takes a few hours):

./s3-benchmark -full

See this for all the other options.

Build

1. Install Go

   sudo apt-get install golang-go
   

   or

   sudo yum install go
   

   may work too.

2. Setup Go environment variables (Usually GOPATH and GOBIN) and test Go installation

3. Clone the repo

4. Install dep

   go get -u github.com/golang/dep/cmd/dep
   

5. Go to source directory and run dep ensure

6. Run go run main.go

S3 to EC2 Bandwidth

I ran this benchmark on all current generation EC2 instance types as of 2019-04-02. I put the data in an interactive spreadsheet that you can download here. Feel free to share it with your friends and colleagues.

Analysis

Here's the maximum download throughput I got from all 155 instance types:

EC2 Instance Type	Max S3 Throughput MB/s
c5n.18xlarge	8,003
p3dn.24xlarge	6,269
c5n.9xlarge	5,741
c5n.2xlarge	2,861
c5n.4xlarge	2,851
r5.metal	2,718
z1d.12xlarge	2,718
r5d.24xlarge	2,718
i3.metal	2,716
r5.24xlarge	2,714
m5d.24xlarge	2,714
r4.16xlarge	2,714
h1.16xlarge	2,713
m5.metal	2,713
x1.32xlarge	2,711
c5d.18xlarge	2,710
c5.18xlarge	2,710
p3.16xlarge	2,710
x1e.32xlarge	2,709
m5.24xlarge	2,709
z1d.metal	2,708
i3.16xlarge	2,707
f1.16xlarge	2,706
m4.16xlarge	2,705
g3.16xlarge	2,705
p2.16xlarge	2,702
m5d.metal	2,673
m5a.24xlarge	1,801
m5ad.24xlarge	1,706
r5ad.24xlarge	1,699
r5a.24xlarge	1,562
c5n.xlarge	1,543
x1.16xlarge	1,410
p2.8xlarge	1,409
g3.8xlarge	1,400
x1e.16xlarge	1,400
r4.8xlarge	1,400
i3.8xlarge	1,400
p3.8xlarge	1,400
h1.8xlarge	1,399
c5.9xlarge	1,387
r5.12xlarge	1,387
z1d.6xlarge	1,387
m5.12xlarge	1,387
m5d.12xlarge	1,387
c5d.9xlarge	1,387
r5d.12xlarge	1,386
g3.4xlarge	1,163
r4.4xlarge	1,163
f1.4xlarge	1,163
i3.4xlarge	1,162
x1e.8xlarge	1,162
h1.4xlarge	1,161
h1.2xlarge	1,161
x1e.4xlarge	1,160
m5.4xlarge	1,157
r5a.4xlarge	1,156
r5.4xlarge	1,156
r5d.4xlarge	1,156
m5a.12xlarge	1,156
m5d.4xlarge	1,156
m5ad.4xlarge	1,156
c5d.4xlarge	1,156
r5ad.12xlarge	1,156
c5.4xlarge	1,156
m5ad.12xlarge	1,156
r5a.12xlarge	1,156
m5a.4xlarge	1,155
r5ad.4xlarge	1,155
z1d.3xlarge	1,155
a1.4xlarge	1,153
i3.2xlarge	1,143
p3.2xlarge	1,142
x1e.2xlarge	1,138
f1.2xlarge	1,137
r5ad.2xlarge	1,136
m5d.2xlarge	1,136
r4.2xlarge	1,135
r5d.2xlarge	1,134
m5.2xlarge	1,133
z1d.2xlarge	1,132
m5ad.xlarge	1,132
c5d.2xlarge	1,132
m5a.2xlarge	1,132
m5ad.2xlarge	1,131
r5a.2xlarge	1,131
c5.2xlarge	1,131
r5ad.xlarge	1,130
r5.2xlarge	1,129
r4.xlarge	1,127
m5a.xlarge	1,125
g3s.xlarge	1,124
r5a.xlarge	1,123
i3.xlarge	1,119
z1d.xlarge	1,116
m5.xlarge	1,114
c5.xlarge	1,114
m5d.xlarge	1,114
r5.xlarge	1,114
r5d.xlarge	1,113
c5d.xlarge	1,113
i2.8xlarge	1,092
d2.8xlarge	1,066
c4.8xlarge	1,066
m4.10xlarge	1,066
z1d.large	1,002
x1e.xlarge	980
c5.large	949
a1.2xlarge	944
c5d.large	942
r5d.large	936
m5d.large	891
m5.large	873
c5n.large	851
r5.large	846
r5ad.large	783
m5ad.large	762
r5a.large	740
a1.xlarge	737
m5a.large	726
i3.large	624
t3.2xlarge	569
t3.xlarge	568
t3.medium	558
t3.large	553
d2.4xlarge	544
c4.4xlarge	544
r4.large	541
a1.large	514
t3.small	395
t3.micro	349
t3.nano	319
c4.2xlarge	272
d2.2xlarge	272
m4.4xlarge	246
i2.4xlarge	244
g2.8xlarge	237
a1.medium	169
p2.xlarge	154
t2.nano	118
m4.2xlarge	118
i2.2xlarge	118
g2.2xlarge	118
t2.2xlarge	116
t2.xlarge	113
t2.large	109
t2.medium	108
c4.xlarge	102
d2.xlarge	102
m4.xlarge	87
i2.xlarge	80
c4.large	71
m4.large	53
t2.micro	46
t2.small	39

Here's the performance of all instances with 32 MB objects (the legend is truncated, but all instances are plotted):

And here's the same chart with just the 3 outlier instances that have 50 Gigabit or more network bandwidth:

Here's a typical throughput profile showing how object size affects performance:

S3's 90th percentile time to first byte is typically around 20 ms regardless of the object size. However, small instances start to see elevated latencies with increased parallelization due to their limited resources. Here's the p90 first byte latency on a small instance:

And here's the p90 first byte latency on a larger instance:

Unlike the first byte latency, the time to last byte obviously follows the object size. S3 seems to deliver downloads at a rate of about 93 MB/s per thread, so this latency is a function of that and the first byte latency — at least until the network bandwidth gets saturated. Here's one example:

If you want to analyze the data further, you can download the spreadsheet with all the raw data and the interactive features shown in the screenshots. Here's a 45 second demo of what you can do with it.

License

This project is released under the MIT License.