How to recover/resume kafka consumer when kafka storage is corrupted?

link管理

链接快照平台

输入网页链接，自动生成快照
标签化管理网页链接

相关文章推荐

打酱油的菠萝 · 蜘蛛侠反派大揭底（第二弹）！迷倒小蜘蛛的黑猫 ...· 4 月前 ·

忧郁的篮球 · 项目运作大事记· 5 月前 ·

温柔的红薯 · I/O 2023: Google Maps ...· 6 月前 ·

大气的小笼包 · 榆林移动“智慧养老”护航老人幸福生活 - ...· 7 月前 ·

迷茫的火龙果 · pypfopt.efficient_fron ...· 8 月前 ·

How to recover/resume kafka consumer when kafka storage is corrupted?

Kafka pods are in CreatingContainer State and Ready with 1/2 status, and the following logs are shown:

ERROR [ReplicaManager broker=0] Error processing append operation on partition __consumer_offsets-13 (kafka.server.ReplicaManager) [data-plane-kafka-request-handler-2]
org.apache.kafka.common.errors.NotEnoughReplicasException: The size of the current ISR Set(0) is insufficient to satisfy the min.isr requirement of 2 for partition __consumer_offsets-13
INFO [GroupCoordinator 0]: Preparing to rebalance group group-xxx in state PreparingRebalance with old generation 118964 (__consumer_offsets-13) (reason: Error COORDINATOR_NOT_AVAILABLE when storing group assignment during SyncGroup (member: consumer-group-xxx-x-aaa-bbb-ccc)) (kafka.coordinator.group.GroupCoordinator) [data-plane-kafka-request-handler-2]
[...]
ERROR [ReplicaFetcher replicaId=2, leaderId=0, fetcherId=0] Error for partition __consumer_offsets-13 at offset xxxxx (kafka.server.ReplicaFetcherThread) [ReplicaFetcherThread-0-0]
org.apache.kafka.common.errors.CorruptRecordException: This message has failed its CRC checksum, exceeds the valid size, has a null key for a compacted topic, or is otherwise corrupt.
ERROR [ReplicaFetcher replicaId=2, leaderId=0, fetcherId=0] Error for partition __consumer_offsets-13 at offset xxxxx (kafka.server.ReplicaFetcherThread) [ReplicaFetcherThread-0-0]
org.apache.kafka.common.errors.CorruptRecordException: This message has failed its CRC checksum, exceeds the valid size, has a null key for a compacted topic, or is otherwise corrupt.
[...]
org.apache.kafka.common.errors.KafkaStorageException: Disk error when trying to access log file on 
the disk.
ERROR [ReplicaFetcher replicaId=1, leaderId=0, fetcherId=0] Error for partition SQL_xxx-10 at offset 12345 (kafka.server.ReplicaFetcherThread) [ReplicaFetcherThread-0-0]
org.apache.kafka.common.errors.KafkaStorageException: Disk error when trying to access log file on 
the disk.
Observed below read only error on Kafka storage mount volumes:
message: 'relabel failed /var/lib/kubelet/pods/xxxxxyyyyyzzzz/volumes/kubernetes.io~csi/pvc-abcdef/mount:
      lsetxattr /var/lib/kubelet/pods/xxxxxyyyyyzzzz/volumes/kubernetes.io~csi/pvc-abcdef/mount/kafka-log0:
      read-only file system'
                  New to Red Hat?
                  Learn more about Red Hat subscriptions
                                      Using a Red Hat product through a public cloud?
                    How to access this content
          Here are the common uses of Markdown.

        Code blocks
~~~
Code surrounded in tildes is easier to read
        Links/URLs
[Red Hat Customer Portal](https://access.redhat.com)
          Learn more
          
                    Are you sure you want to request a translation?                
                We appreciate your interest in having Red Hat content localized to your language. Please note that excessive use of this feature could cause delays in getting specific content you are interested in translated.      
        
                          Request Japanese Translation      Request Chinese Translation      Request Korean Translation