kafka 吞吐量测试

1.指令

kafka-producer-perf-test.sh

1.1 版本 0.10.0

• topic TOPIC
  生产数据的 topic
• num-records NUM-RECORDS
  生产 NUM-RECORDS 条数据
• record-size RECORD-SIZE
  一条数据的大小(byte)
• throughput THROUGHPUT 最大吞吐量控制(大约)
• producer-props PROP-NAME=PROP-VALUE [PROP-NAME=PROP-VALUE ...] kafka producer configuaration，kafka生产者的参数
  • bootstrap.servers 指定服务器列表
  • acks 指定 acks
  • compression.type 压缩方式
  • 参考官网 http://kafka.apache.org/documentation/#producerconfigs

1.2 版本 2.2.0

• topic TOPIC
  生产数据的 topic
• num-records NUM-RECORDS 生产 NUM-RECORDS 条数据
• payload-delimiter PAYLOAD-DELIMITER 当使用 payload-file，指定分隔符，默认为新行(default: \n)，如果没指定 payload-file 会被忽略
• throughput THROUGHPUT 最大吞吐量控制(大约)
• producer-props PROP-NAME=PROP-VALUE [PROP-NAME=PROP-VALUE ...] kafka producer configuaration，kafka生产者的参数
• producer.config CONFIG-FILE 生产者配置文件
• print-metrics
  在测试结束时打印出测试标准，默认为：false
• transactional-id TRANSACTIONAL-ID
  如果 transaction-duration-ms为> 0，则使用transactionalId。在测试并发事务的性能时非常有用。(默认值
  )
• transaction-duration-ms TRANSACTION-DURATION 每笔交易的最大age。在此时间过后，将调用 commitTransaction。只有当该值为正时才启用事务。(默认值:0) either --record-size or --payload-file 必须指定且只能指定一个
• record-size RECORD-SIZE 消息大小(以字节为单位)。注意，您必须精确地提供一个 --record-size 或者 --payload-file
• payload-file PAYLOAD-FILE 文件来读取消息的有效负载。这只适用于UTF-8编码的文本文件。将从该文件读取有效负载，并在发送消息时随机选择有效负载。注意，您必须精确地提供一个—记录大小或—有效负载文件

2.测试示例

(测试不同压缩方式的影响) lz4

[appadm@PF-temp-stress01-48-61 bin]$ ./kafka-producer-perf-test.sh --topic test_t3 --throughput 100000 --num-records 2000000 --producer-props bootstrap.servers=192.168.48.61:9091,192.168.48.62:9091,192.168.48.63:9091 acks=all compression.type=lz4 --record-size 1000
494787 records sent, 98937.6 records/sec (94.35 MB/sec), 56.4 ms avg latency, 223.0 max latency.
501551 records sent, 100270.1 records/sec (95.63 MB/sec), 49.1 ms avg latency, 95.0 max latency.
499964 records sent, 99892.9 records/sec (95.27 MB/sec), 54.9 ms avg latency, 351.0 max latency.
500970 records sent, 100133.9 records/sec (95.50 MB/sec), 49.3 ms avg latency, 91.0 max latency.
2000000 records sent, 99765.550955 records/sec (95.14 MB/sec), 52.39 ms avg latency, 351.00 ms max latency, 48 ms 50th, 77 ms 95th, 96 ms 99th, 326 ms 99.9th.

snappy

[appadm@PF-temp-stress01-48-61 bin]$ ./kafka-producer-perf-test.sh --topic test_t3 --throughput 100000 --num-records 2000000 --producer-props bootstrap.servers=192.168.48.61:9091,192.168.48.62:9091,192.168.48.63:9091 acks=all compression.type=snappy --record-size 1000
243373 records sent, 48655.1 records/sec (46.40 MB/sec), 434.1 ms avg latency, 880.0 max latency.
295596 records sent, 59071.9 records/sec (56.34 MB/sec), 463.7 ms avg latency, 1133.0 max latency.
289972 records sent, 57959.6 records/sec (55.27 MB/sec), 461.9 ms avg latency, 1564.0 max latency.
237331 records sent, 47399.8 records/sec (45.20 MB/sec), 573.6 ms avg latency, 1808.0 max latency.
238410 records sent, 47643.9 records/sec (45.44 MB/sec), 608.4 ms avg latency, 1856.0 max latency.
229594 records sent, 45918.8 records/sec (43.79 MB/sec), 614.0 ms avg latency, 2295.0 max latency.
273367 records sent, 54640.6 records/sec (52.11 MB/sec), 502.2 ms avg latency, 1589.0 max latency.
2000000 records sent, 50850.474181 records/sec (48.49 MB/sec), 529.47 ms avg latency, 2295.00 ms max latency, 179 ms 50th, 1738 ms 95th, 1863 ms 99th, 2277 ms 99.9th.

gzip

[appadm@PF-temp-stress01-48-61 bin]$ ./kafka-producer-perf-test.sh --topic test_t3 --throughput 100000 --num-records 2000000 --producer-props bootstrap.servers=192.168.48.61:9091,192.168.48.62:9091,192.168.48.63:9091 acks=all compression.type=gzip --record-size 1000
120968 records sent, 24183.9 records/sec (23.06 MB/sec), 56.0 ms avg latency, 211.0 max latency.
162449 records sent, 32360.4 records/sec (30.86 MB/sec), 61.3 ms avg latency, 214.0 max latency.
181186 records sent, 36230.0 records/sec (34.55 MB/sec), 59.0 ms avg latency, 212.0 max latency.
168412 records sent, 33682.4 records/sec (32.12 MB/sec), 45.7 ms avg latency, 102.0 max latency.
169483 records sent, 33889.8 records/sec (32.32 MB/sec), 46.7 ms avg latency, 98.0 max latency.
168733 records sent, 33746.6 records/sec (32.18 MB/sec), 49.3 ms avg latency, 111.0 max latency.
185551 records sent, 37110.2 records/sec (35.39 MB/sec), 48.7 ms avg latency, 107.0 max latency.
176316 records sent, 35235.0 records/sec (33.60 MB/sec), 45.3 ms avg latency, 101.0 max latency.
178916 records sent, 35768.9 records/sec (34.11 MB/sec), 49.3 ms avg latency, 130.0 max latency.
174084 records sent, 34795.9 records/sec (33.18 MB/sec), 46.1 ms avg latency, 101.0 max latency.
174001 records sent, 34793.2 records/sec (33.18 MB/sec), 46.8 ms avg latency, 92.0 max latency.
2000000 records sent, 34001.462063 records/sec (32.43 MB/sec), 49.94 ms avg latency, 214.00 ms max latency, 45 ms 50th, 77 ms 95th, 126 ms 99th, 184 ms 99.9th.