S3 PutObject performance

[mcortellino]

Hello I am measuring the performance speed of PutObject API on an S3 bucket under this scenario:

• Client app: a pod running on EKS built with .NET8 and AWSSDK.S3 Version="3.7.407"
• Bucket in the same region of EKS with a vpc endpoint

After some tuning I come up with a write average speed for a 1Kb file of 77ms. Are there any real improvement margin beyond this? The app will write thousand of files per sec so an improvement on each call will be really important to us.

Thanks

[dscpinheiro]

It's not easy to give blanket recommendations on performance... Are you planning to write thousands of small files to S3? Because that'll "force" an API call for each one which can also get expensive.

The SDK does offer a TransferUtility helper, which allows you to upload either an individual file or a directory to S3, and the SDK will use multi-part uploads and upload files in parallel (which should give better results than the individual PutObject calls):

using Amazon.S3.Transfer;

var config = new TransferUtilityConfig
{
    // Number of threads to use for concurrent uploads
    ConcurrentServiceRequests = 10,

    // Minimum size (in bytes) before using multi-part uploads instead of PutObject
    MinSizeBeforePartUpload = 16 * (long)Math.Pow(2.0, 20.0),
};

var tu = new TransferUtility(s3Client, config);
await tu.UploadDirectoryAsync(new TransferUtilityUploadDirectoryRequest
{
    BucketName = "my-bucket",
    Directory = "local-directory",
    KeyPrefix = "s3-prefix",
    UploadFilesConcurrently = true,
});

You can also enable verbose logging to see where time is spent:

using Amazon;
using Amazon.Runtime;
using Amazon.Runtime.CredentialManagement;
using Amazon.S3;

AWSConfigs.LoggingConfig.LogTo = LoggingOptions.Console;
AWSConfigs.LoggingConfig.LogMetricsFormat = LogMetricsFormatOption.JSON;
AWSConfigs.LoggingConfig.LogResponses = ResponseLoggingOption.Always;
AWSConfigs.LoggingConfig.LogMetrics = true;

using var s3Client = new AmazonS3Client();
await s3Client.PutObjectAsync(...);

The Best practices design patterns: optimizing Amazon S3 performance page on the S3 user guide also has some guidelines that do not depend on the .NET SDK.

[mcortellino]

Hi @dscpinheiro thanks for your feedback. Actually in our first implementation we used TransferUtility class and the performance was 3x worst ( like 294ms on average for a 5Kb file ). It seems that the overhead of using TransferUtility for such small file is degrading the performance. For business reason we need to persists each file separately without buffering them on file system to guarantee the consistency and avoid data loss.

Anyway I will give a try with the verbose logging, thanks

[normj]

Any chance in your performance testing you could try out S3 Express? https://aws.amazon.com/s3/storage-classes/express-one-zone/

Also not saying you should do this in production but curious what your performance numbers will look like if you disable the SDK's checksum logic using the command AWSConfigsS3.DisableDefaultChecksumValidation = true;.

[mcortellino]

Unfortunately the one zone express is not available in the region where the application is deployed, I tried with AWSConfigsS3.DisableDefaultChecksumValidation = true but I didn't noticed any significant improvement I am still in the range 70-80 ms for each request.

I have also enabled the detailed metrics as suggested 97/98% of the time is spent on HttpRequestTime as example this is one of the trace:

"timings":{"CredentialsRequestTime":0.0102,"RequestSigningTime":0.9376,"HttpRequestTime":83.3764,"ResponseUnmarshallTime":0.0849,"ResponseProcessingTime":0.2661,"ClientExecuteTime":85.5894}

[normj]

That information implies the majority of the time is being spent on the network call and on the S3 side. Outside of what the SDK can influence. This is why S3 express was built to reduce those times, it is unfortunate that is not currently available in your region. What is your hard requirement goal single file upload speed or total concurrency through put?

[mcortellino]

A good upload speed for us considering such small files would be around 15/20 ms, the throughput it 's not really a problem, we don't expects concurrency values that could represent a problem for s3 and our tests confirms it.

[mcortellino]

Just for information. I have reached another 40% performance increase just putting the HttpClientCacheSize parameter to 1 (In the original implementation it was set to an higher number). The curious thing is that changing it to 2 cause a fallback to old (slow) speed values. It seems that 1 is the only good value.

[normj]

The SDK uses the value 1 for HttpClientCacheSize. The property is a bit of a legacy property added when .NET Core was using libcurl for HTTP support. This is pre .NET Core 3.1. Back then having a value greater then 1 was useful. I'm not sure it is ever useful anymore now that .NET uses its own socket library instead of libcurl.