Kafka从入门到放弃(一) —— 初识Kafka

[TencentCloudSDKException] code:FailedOperation.ServiceIsolate message:service is stopped due to arrears, please recharge your account in Tencent Cloud requestId:ad8c743b-3c89-49c8-bc95-f205b81fe8ff

背景与问题

[TencentCloudSDKException] code:FailedOperation.ServiceIsolate message:service is stopped due to arrears, please recharge your account in Tencent Cloud requestId:f4e79c81-c0a4-4bfc-b56a-9879a046d7fa

[TencentCloudSDKException] code:FailedOperation.ServiceIsolate message:service is stopped due to arrears, please recharge your account in Tencent Cloud requestId:7a56eaea-3334-4766-836c-567774b57a73

[TencentCloudSDKException] code:FailedOperation.ServiceIsolate message:service is stopped due to arrears, please recharge your account in Tencent Cloud requestId:7c1d8136-5ecf-44b7-851f-7526eb444730

优点

解耦

[TencentCloudSDKException] code:FailedOperation.ServiceIsolate message:service is stopped due to arrears, please recharge your account in Tencent Cloud requestId:48fb81b6-4714-4af7-9b20-8813acab7e92

[TencentCloudSDKException] code:FailedOperation.ServiceIsolate message:service is stopped due to arrears, please recharge your account in Tencent Cloud requestId:2c139dab-ca46-4584-a3ac-0fb724e9242b

[TencentCloudSDKException] code:FailedOperation.ServiceIsolate message:service is stopped due to arrears, please recharge your account in Tencent Cloud requestId:7d298584-3257-4ae1-8086-41862efc5858

[TencentCloudSDKException] code:FailedOperation.ServiceIsolate message:service is stopped due to arrears, please recharge your account in Tencent Cloud requestId:ef651b7c-59d8-463c-9cbe-16e31e931f3d

异步

[TencentCloudSDKException] code:FailedOperation.ServiceIsolate message:service is stopped due to arrears, please recharge your account in Tencent Cloud requestId:90b5683d-89a5-457e-a72c-140b28716da6

[TencentCloudSDKException] code:FailedOperation.ServiceIsolate message:service is stopped due to arrears, please recharge your account in Tencent Cloud requestId:fccef90d-5623-40d7-b32a-55aaea8bac0c

削峰

[TencentCloudSDKException] code:FailedOperation.ServiceIsolate message:service is stopped due to arrears, please recharge your account in Tencent Cloud requestId:ae76b62c-8eda-41b2-b380-9d8e0b569074

缺点

[TencentCloudSDKException] code:FailedOperation.ServiceIsolate message:service is stopped due to arrears, please recharge your account in Tencent Cloud requestId:fbbfbb68-360e-473e-b851-7d927f0decd1

[TencentCloudSDKException] code:FailedOperation.ServiceIsolate message:service is stopped due to arrears, please recharge your account in Tencent Cloud requestId:95ae2d75-da09-4759-b4f1-d8d1b465abea

[TencentCloudSDKException] code:FailedOperation.ServiceIsolate message:service is stopped due to arrears, please recharge your account in Tencent Cloud requestId:2a08d4ac-9dbd-4ef1-a5ce-b25b8eb8743d

[TencentCloudSDKException] code:FailedOperation.ServiceIsolate message:service is stopped due to arrears, please recharge your account in Tencent Cloud requestId:b4c10b23-da51-41d7-b4be-1173c8a3bb8a

消费模式

[TencentCloudSDKException] code:FailedOperation.ServiceIsolate message:service is stopped due to arrears, please recharge your account in Tencent Cloud requestId:83873b0d-9ea2-4fa0-b7fd-318d9697176d

[TencentCloudSDKException] code:FailedOperation.ServiceIsolate message:service is stopped due to arrears, please recharge your account in Tencent Cloud requestId:6b667c2e-338c-4b00-a25b-f997725ab48b

[TencentCloudSDKException] code:FailedOperation.ServiceIsolate message:service is stopped due to arrears, please recharge your account in Tencent Cloud requestId:8f13b4f0-a206-49ad-9ef8-2b5362f63cc8

但是这种方法也有缺点，因为消费者的处理速度不一样，有快有慢，容易出现问题。比如消息队列推送的速度为100M/s，消费者A处理速度为10M/s，消费者B处理速度为500M/s，这时候A就崩了，对于B来说，又造成资源浪费。

[TencentCloudSDKException] code:FailedOperation.ServiceIsolate message:service is stopped due to arrears, please recharge your account in Tencent Cloud requestId:accf140d-9d0b-4e31-923f-14be02a6cee5

本文的主角 Kafka 是基于拉取的发布订阅模式。

讲了这么多，是时候请上主角，有请 Kafka 登场！

Kafka 的基础架构

Producer：生产者，发布消息的对象，将消息推到 Kafka 集群

Consumer：消费者，消费消息的对象

Consumer Group：消费者组，Kafka 中可以将多个 Consumer 分为一个组，从整体上可以将它看作是一个Consumer

Broker：一台 Kafka 服务器就是一个 Broker，多个 Broker 组成一个 Cluster

Topic：消息的主题，可以理解为一个消息队列

Partition：分区，一个 Topic 可以分为多个 Partition，这样的好处是负载均衡；同时，一个 Partition 可以有多个副本，提高可靠性。对于 Consumer Group 来说，一个消费者组中的消费者只能订阅同一个 Topic 的不同分区，可以提高效率，又避免重复消费。

Leader & Follower：对于同一个 Partition 而言，消费者只读取 Leader 的消息，而不会读取 Follower 的消息，Follower 是 Leader 的副本，在 Leader 挂掉的时候 Follower 可成为 Leader

Zookeeper： Kafka 是基于 zk 的，用于集群管理

为什么 Kafka 要这样设计

如果消息中间件只有一台机，哪天突然宕机了，整个系统就崩了。因此需要整一个集群，搞多台服务器，所以我们搞几个 Broker。

[TencentCloudSDKException] code:FailedOperation.ServiceIsolate message:service is stopped due to arrears, please recharge your account in Tencent Cloud requestId:72f82900-e479-4be5-a2bd-f7f8e7e37d20

于是要合理分配工作，整出了 Partition，每个 Topic 对应每个生产者和消费者，同一个 Topic 又分成多个分区，分别在不同的 Broker，分担了单台节点的压力。

不过现在又有一个问题，如果一台 Broker 宕机，该节点上的分区数据也没了。为了防止单节点故障造成数据丢失，每个分区存几个副本保存在其它 Broker。

但消费者只能访问其中一个分区，不然会造成重复消费的现象，所以要区分好 Leader 和 Follower，并使消费者只能访问 Leader，而 Follower 需要在 Leader 发生故障的时候成为新的 Leader。

Original: https://www.cnblogs.com/lyuzt/p/15634244.html
Author: 大数据的奇妙冒险
Title: Kafka从入门到放弃(一) —— 初识Kafka