Bug #65794
openCeph Reef RGW error response fails to be parsed during awscli create-bucket
0%
Description
Using Ceph Reef v18.2.2 (with Rook v1.13.5 but it is not important here). Running tests with RGW bucket quota exceed and got the following message from aws cli:
# aws --endpoint-url https://rook-ceph-rgw-openstack-store.rook-ceph.svc:8443/ --ca-bundle /etc/rgwcerts/cacert s3api create-bucket --bucket quota argument of type 'NoneType' is not iterable
It seems this is due to RGW starts returning message tag with empty body since Ceph Reef:
2024-05-03 14:48:59,116 - MainThread - botocore.parsers - DEBUG - Response body: b'<?xml version="1.0" encoding="UTF-8"?><Error><Code>TooManyBuckets</Code><Message></Message><BucketName>quota</BucketName><RequestId>tx000003a27407b65589525-006634f95a-47cc-openstack-store</RequestId><HostId>47cc-openstack-store-openstack-store</HostId></Error>'
(see full debug log in error-message-reef.log)
When using Ceph Quincy v17.2.7 - nothing unexpected, I got TooManyBuckets error as always:
# aws --endpoint-url https://rook-ceph-rgw-openstack-store.rook-ceph.svc:8443/ --ca-bundle /etc/rgwcerts/cacert s3api create-bucket --bucket quota-bucket An error occurred (TooManyBuckets) when calling the CreateBucket operation: Unknow
And here is corresponding RGW response for over-quota bucket creation in Ceph Quincy:
2024-05-03 15:01:06,069 - MainThread - botocore.parsers - DEBUG - Response body: b'<?xml version="1.0" encoding="UTF-8"?><Error><Code>TooManyBuckets</Code><BucketName>quota-bucket</BucketName><RequestId>tx000004f549f743c0fcb7f-006634fc32-d524-openstack-store</RequestId><HostId>d524-openstack-store-openstack-store</HostId></Error>'
(full debug log could be found in error-message-quincy.log)
awscli and botocore versions are the same for Ceph Quincy and Ceph Reef tests so the version diff has no effect here.
Files
Updated by Casey Bodley 11 days ago
- Related to Bug #59433: s3 error response missing Message field added
Updated by Casey Bodley 11 days ago
- Blocks Backport #59615: quincy: s3 error response missing Message field added
Updated by Casey Bodley 11 days ago
- Tags set to s3
- Backport set to reef squid
- Regression changed from No to Yes
Peter Razumovsky wrote:
It seems this is due to RGW starts returning message tag with empty body since Ceph Reef:
can't win, i guess. before, some clients failed to decode the response because there was no Message element (https://tracker.ceph.com/issues/59433). https://github.com/ceph/ceph/pull/51052 added the empty Message element for that case
i guess boto interprets empty as NoneType? maybe we should return a space character instead
Updated by Peter Razumovsky 11 days ago
- File proving-example.py proving-example.py added
Yep, this is behaviour of boto, it parses xml response with
import xml.etree.cElementTree as ETree
and produces a dict with item.text values as a result. In that case, xml parser returns `None` for empty Message tag, you're right:
{'Code': 'TooManyBuckets', 'Message': None, 'BucketName': 'quota', 'RequestId': 'tx000003a27407b65589525-006634f95a-47cc-openstack-store', 'HostId': '47cc-openstack-store-openstack-store'}
See full proving example in attachment. The code is taken from https://github.com/boto/botocore/blob/develop/botocore/parsers.py#L500 and https://github.com/boto/botocore/blob/develop/botocore/parsers.py#L477.