3

I have a ceph test cluster with three nodes with radosgw installed on the first node. ceph version is 14.2.6 (f0aa067ac7a02ee46ea48aa26c6e298b5ea272e9) nautilus (stable)

# cat /etc/ceph/ceph.conf
[global]
fsid = a204e3b5-b0e1-4598-aadc-f46011b562f0

mon_initial_members = ceph-test-01, ceph-test-02, ceph-test-03
mon_host = 172.19.19.252,172.19.19.251,172.19.19.244

auth_cluster_required = cephx
auth_service_required = cephx
auth_client_required = cephx

public network = 172.19.19.0/24

rgw_frontends = beast port=80

And Apache Kafka broker for receiving messages.

I've created a topic according to the official CEPH docs:

(https://docs.ceph.com/docs/master/radosgw/notifications/).

POST http://ceph-test-01/?Action=CreateTopic&Name=bucketnotifications&push-endpoint=kafka://kafka-test-01:9092 with S3 credentials
Response:
<CreateTopicResponse xmlns="https://sns.amazonaws.com/doc/2010-03-31/">
    <CreateTopicResult>
        <TopicArn>arn:aws:sns:us::bucketnotifications</TopicArn>
    </CreateTopicResult>
    <ResponseMetadata>
        <RequestId>14804357-9c29-437d-9789-c127e1c13d23.44108.275</RequestId>
    </ResponseMetadata>
</CreateTopicResponse>

I've created a bucket named new-bucket and checked its configuration:

GET http://ceph-test-01/new-bucket?notification
Response:
<NotificationConfiguration>
    <TopicConfiguration>
        <Id>bucketnotifications</Id>
        <Topic>arn:aws:sns:us::bucketnotifications</Topic>
        <Event>s3:ObjectCreated:*</Event>
        <Event>s3:ObjectRemoved:*</Event>
    </TopicConfiguration>
</NotificationConfiguration>

When I add or remove files to the bucket, ceph doesn't push notifications to the Apache Kafka topic. Radosgw logs (debug mode is on) contains the following strings:

2020-03-04 10:58:00.533 7f29a8da3700 20 notification: 'bucketnotifications' on topic: 'bucketnotifications' and bucket: 'new-bucket' (unique topic: 'bucketnotifications') apply to event of type: 's3:ObjectCreated:Put'
2020-03-04 10:58:00.533 7f29a8da3700  1 ERROR: failed to create push endpoint: kafka://kafka-test-01:9092 due to: pubsub endpoint configuration error: unknown schema in: kafka://kafka-test-01:9092
2020-03-04 10:58:00.533 7f29a8da3700  5 req 188 0.248s s3:put_obj WARNING: publishing notification failed, with error: -22

But Apache Kafka has no new topics created.

Alex Sergeenko
  • 642
  • 5
  • 22
fenhir
  • 31
  • 2
  • Remove `kafka://`? – OneCricketeer Mar 05 '20 at 15:59
  • 1
    @cricket_007 I guess `kafka://` is came from the CEPH docs, what is the reason to remove this prefix? – Alex Sergeenko Mar 10 '20 at 14:48
  • @Alex read the error `unknown schema in: kafka://` – OneCricketeer Mar 10 '20 at 15:32
  • @cricket_007 Ain't work with `kafka://` removed either, considering it as an `http` endpoint. – Alex Sergeenko Mar 11 '20 at 15:18
  • @Alex do you know if there's an open JIRA or github issue for this? – OneCricketeer Mar 11 '20 at 15:20
  • @cricket_007 as far as I'm concerned, there is no any related issue being opened, at least I haven't found any. – Alex Sergeenko Mar 12 '20 at 06:28
  • There is one more point to consider. We've looked at `rgw_pubsub_push.cc` https://github.com/ceph/ceph/blob/f6bd5699315c434ab7819646836df4c5181471ea/src/rgw/rgw_pubsub_push.cc source file and found out that there is a flag `WITH_RADOSGW_KAFKA_ENDPOINT` exists. The generated exception looks similar to the one in our case: `throw configuration_error("unknown schema in: " + endpoint);`. Is there a chance that we use a CEPH build with Kafka endpoints off? – Alex Sergeenko Mar 12 '20 at 10:36
  • @cricket_007 Here is the issue on the `CEPH` tracker: https://tracker.ceph.com/issues/44436 – Alex Sergeenko Mar 12 '20 at 10:47

1 Answers1

1

It seems there is a bug in nautilus releases that kafka schema won't check. You can upgrade to version 15.2.0 that support this schema or wait for bug fix update on nautilus.

Seena Fallah
  • 560
  • 4
  • 12