0

We are moving from NFS to S3 but I am not happy with the write performance.

How can I speed it up ? Are there any configurations to use while moving large files?

Now it's around ~10MB/s vs our regular NFS filesystem 200MB/s ie 20 times slower.

I am curious about the general write performance you guys see on your system.

My observations using s3cmd version 2.1.0:

Put a 4gb tar file on s3 storage shows 8.45 MB/s

upload: '' -> 's3://yetest/backups/si106-251/si106-251.20221219202432.tgz' [part 299, 15MB] 15728640 of 15728640 100% in 1s 8.45 MB/s done

upload: '' -> 's3://yetest/backups/si106-251/si106-251.20221219202432.tgz' [part 299, 15MB] 15728640 of 15728640 100% in 1s 8.45 MB/s done <------ 8.45MB/s

user11666461
  • 761
  • 7
  • 12
Gr8DBA
  • 9
  • 2
  • Is your NFS accessed over the Internet? – Anon Coward Jan 09 '23 at 18:47
  • @AnonCoward NFS is storage mounted as a drive on UNIX host with a 10gbps pipe to datacenter – Gr8DBA Jan 09 '23 at 20:07
  • So, you appear to be comparing a connecting to a datacenter to internet speeds. Have you tried using the AWS CLI along with [updating the various S3 options](https://docs.aws.amazon.com/cli/latest/topic/s3-config.html)? – Anon Coward Jan 09 '23 at 20:10
  • @AnonCoward yes I have ... playing around with bandwidth etc ie up to 200Mb/s but I only see it peaks out at ~30mb/s - NOTE this is always a LARGE backup tar file I trying to store hence I can not split the file up to multiple uploads ... max_concurrent_requests = 20 max_queue_size = 10000 multipart_threshold = 64MB multipart_chunksize = 16MB max_bandwidth = 200MB/s use_accelerate_endpoint = true addressing_style = path TEST upload a 1gb file upload: '/root/test1.img' -> 's3: in 0s 27.57 MB/s done – Gr8DBA Jan 09 '23 at 22:28

0 Answers0