2

I have sentry on my kubernetes cluster. After last upgrade to chart sentry-18.0.0, app version 23.3.1 one pod: sentry-ingest-replay-recordings-xxx-yyy is crash loop (CrashLoopBackOff).

logs below, any idea how to fix it?

06:16:55 [WARNING] sentry.utils.geo: settings.GEOIP_PATH_MMDB not configured.                                                                                                                                               │
│ 06:16:58 [INFO] sentry.plugins.github: apps-not-configured                                                                                                                                                                  │
│ Traceback (most recent call last):                                                                                                                                                                                          │
│   File "/usr/local/lib/python3.8/site-packages/arroyo/processing/processor.py", line 181, in run                                                                                                                            │
│     self._run_once()                                                                                                                                                                                                        │
│   File "/usr/local/lib/python3.8/site-packages/arroyo/processing/processor.py", line 212, in _run_once                                                                                                                      │
│     self.__message = self.__consumer.poll(timeout=1.0)                                                                                                                                                                      │
│   File "/usr/local/lib/python3.8/site-packages/arroyo/backends/kafka/consumer.py", line 409, in poll                                                                                                                        │
│     raise ConsumerError(str(error))                                                                                                                                                                                         │
│ arroyo.errors.ConsumerError: KafkaError{code=UNKNOWN_TOPIC_OR_PART,val=3,str="Subscribed topic not available: ingest-replay-recordings: Broker: Unknown topic or partition"}                                                │
│ 06:16:58 [ERROR] arroyo.processing.processor: Caught exception, shutting down...                                                                                                                                            │
│ 06:16:58 [INFO] arroyo.processing.processor: Closing <arroyo.backends.kafka.consumer.KafkaConsumer object at 0x7f17ff085160>...                                                                                             │
│ 06:16:58 [INFO] arroyo.processing.processor: Processor terminated
Traceback (most recent call last):                                                                                                                                                                                          │
│   File "/usr/local/bin/sentry", line 8, in <module>                                                                                                                                                                         │
│     sys.exit(main())                                                                                                                                                                                                        │
│   File "/usr/local/lib/python3.8/site-packages/sentry/runner/__init__.py", line 191, in main                                                                                                                                │
│     func(**kwargs)                                                                                                                                                                                                          │
│   File "/usr/local/lib/python3.8/site-packages/click/core.py", line 1128, in __call__                                                                                                                                       │
│     return self.main(*args, **kwargs)                                                                                                                                                                                       │
│   File "/usr/local/lib/python3.8/site-packages/click/core.py", line 1053, in main                                                                                                                                           │
│     rv = self.invoke(ctx)                                                                                                                                                                                                   │
│   File "/usr/local/lib/python3.8/site-packages/click/core.py", line 1659, in invoke                                                                                                                                         │
│     return _process_result(sub_ctx.command.invoke(sub_ctx))                                                                                                                                                                 │
│   File "/usr/local/lib/python3.8/site-packages/click/core.py", line 1659, in invoke                                                                                                                                         │
│     return _process_result(sub_ctx.command.invoke(sub_ctx))                                                                                                                                                                 │
│   File "/usr/local/lib/python3.8/site-packages/click/core.py", line 1395, in invoke                                                                                                                                         │
│     return ctx.invoke(self.callback, **ctx.params)                                                                                                                                                                          │
│   File "/usr/local/lib/python3.8/site-packages/click/core.py", line 754, in invoke                                                                                                                                          │
│     return __callback(*args, **kwargs)                                                                                                                                                                                      │
│   File "/usr/local/lib/python3.8/site-packages/click/decorators.py", line 26, in new_func                                                                                                                                   │
│     return f(get_current_context(), *args, **kwargs)                                                                                                                                                                        │
│   File "/usr/local/lib/python3.8/site-packages/sentry/runner/decorators.py", line 69, in inner                                                                                                                              │
│     return ctx.invoke(f, *args, **kwargs)                                                                                                                                                                                   │
│   File "/usr/local/lib/python3.8/site-packages/click/core.py", line 754, in invoke                                                                                                                                          │
│     return __callback(*args, **kwargs)                                                                                                                                                                                      │
│   File "/usr/local/lib/python3.8/site-packages/click/decorators.py", line 26, in new_func                                                                                                                                   │
│     return f(get_current_context(), *args, **kwargs)                                                                                                                                                                        │
│   File "/usr/local/lib/python3.8/site-packages/sentry/runner/decorators.py", line 29, in inner                                                                                                                              │
│     return ctx.invoke(f, *args, **kwargs)                                                                                                                                                                                   │
│   File "/usr/local/lib/python3.8/site-packages/click/core.py", line 754, in invoke                                                                                                                                          │
│     return __callback(*args, **kwargs)                                                                                                                                                                                      │
│   File "/usr/local/lib/python3.8/site-packages/sentry/runner/commands/run.py", line 696, in replays_recordings_consumer                                                                                                     │
│     run_processor_with_signals(consumer)                                                                                                                                                                                    │
│   File "/usr/local/lib/python3.8/site-packages/sentry/utils/kafka.py", line 57, in run_processor_with_signals                                                                                                               │
│     processor.run()                                                                                                                                                                                                         │
│   File "/usr/local/lib/python3.8/site-packages/arroyo/processing/processor.py", line 181, in run                                                                                                                            │
│     self._run_once()                                                                                                                                                                                                        │
│   File "/usr/local/lib/python3.8/site-packages/arroyo/processing/processor.py", line 212, in _run_once                                                                                                                      │
│     self.__message = self.__consumer.poll(timeout=1.0)                                                                                                                                                                      │
│   File "/usr/local/lib/python3.8/site-packages/arroyo/backends/kafka/consumer.py", line 409, in poll
  File "/usr/local/lib/python3.8/site-packages/arroyo/backends/kafka/consumer.py", line 409, in poll                                                                                                                        │
│     raise ConsumerError(str(error))                                                                                                                                                                                         │
│ arroyo.errors.ConsumerError: KafkaError{code=UNKNOWN_TOPIC_OR_PART,val=3,str="Subscribed topic not available: ingest-replay-recordings: Broker: Unknown topic or partition"}

i've tried to redeploy helm chart.

1 Answers1

4

If Kafka is complaining about unknown or missing topic, please connect to kafka-0 and run the below command:

/opt/bitnami/kafka/bin/kafka-topics.sh --create --topic ingest-replay-recordings --bootstrap-server localhost:9092
Adrian Mole
  • 49,934
  • 160
  • 51
  • 83