Motivation

See #13294
The issue above is due to using the version-specific write lock to enforce write preconditions, which doesn't work as the new version already has its own lock which isn't the same as the previous version being overwritten, so the 2 new concurrent versions aren't sharing the lock thus both execute.

Current S3 locking is pretty complex as we need to lock for multiple, very different scenarios:

• we have a per object version (in non-versioned bucket, just per object) reader-writer lock which is used to prevent writes during read, and read during writes: when we pass the file object back to the web server to be sent over the client, we do not want another client modifying the object content. We do not have a lot of control over the "file sending" process, so the data stream sent back to the web server is wrapped with a read-lock, and following convention, the web server closes the file, releasing the lock. This works pretty well and we don't have concurrency issues anymore
• when S3 released conditional writes, my first initiative was to used the already existing locking mechanism we had, as we could use the writer lock to enforce only one concurrent write (which is already the case) and verify the conditions inside the lock. This also worked very well, except for versioned buckets. We can dive a bit deeper into it.

This is how S3 versioning works: https://docs.aws.amazon.com/AmazonS3/latest/userguide/versioning-workflows.html

Basically, an object key is now a reference to a stack of object versions (themselves regular S3 object, or delete markers). The problem is that concurrent writes on a versioned object are not overwriting the previous object, but only putting a new object on the stack, so the current "per object version" locking doesn't work, we're not accessing the existing object lock when doing a conditional write, but only the new, current object.

We need a way to lock "the whole stack" instead. I've tried adding this to the current storage layer, but it doesn't work: if we share the same write lock around an "object stack", we enter a deadlock when doing a in-place CopyObject: CopyObject can read the previous object version and write a new version on top of it. If it shares the same lock, it's pretty clear this cannot work.

So we need to use a specific lock for conditional writes, per object key (stack) when using versioned buckets. For regular objects, it already works correctly by design, so I decided to not add it for them and keep the changes contained to versioned buckets.

Side-note: AWS did some changes in DeleteObject to now allow IfMatch to be used which wasn't the case before, so maybe we can use the new lock for all keys or use open() instead to get the right lock. I'm not sure IfMatch can be used with a specific version id, so using open without the lock might just work fine on the latest object of the stack.

Changes

• add locking mechanism for versioned buckets only for conditional writes, with clean ups

Testing

This is pretty hard to test in Python due to bad concurrency, but the reporter of the issue shared a small Go snippet that I could execute which would fail reliably before the fix, and is now properly working 👍