1 설치 및 실행

2 파이썬 예제

3 참고

d:
cd D:\kafka_2.12-2.5.0\bin\windows
zookeeper-server-start.bat D:\kafka_2.12-2.5.0\config\zookeeper.properties

d:
cd D:\kafka_2.12-2.5.0\bin\windows
kafka-server-start.bat D:\kafka_2.12-2.5.0\config/server.properties

d:
cd D:\kafka_2.12-2.5.0\bin\windows
kafka-topics.bat --create --bootstrap-server localhost:9092 --replication-factor 1 --partitions 1 --topic test
kafka-topics.bat --list --bootstrap-server localhost:9092

d:
cd D:\kafka_2.12-2.5.0\bin\windows
kafka-console-producer.bat --bootstrap-server localhost:9092 --topic test

d:
cd D:\kafka_2.12-2.5.0\bin\windows
kafka-console-consumer.bat --bootstrap-server localhost:9092 --topic test --from-beginning

• 상단 명령창은 producer
  
• 하단 명령창은 consumer
  
  

pip install kafka-python

from kafka import KafkaConsumer
import sys

bootstrap_servers = ["localhost:9092"]
topicName = 'test'
consumer = KafkaConsumer(topicName, group_id='group1', bootstrap_servers=bootstrap_servers, auto_offset_reset='earliest')

for message in consumer:
    print(message.value)


'''
try:
    for message in consumer:
        #print ("%s:%d:%d: key=%s value=%s" % (message.topic, message.partition,message.offset, message.key,message.value))
        print(message.value)
except KeyboardInterrupt:
    sys.exit()
'''   

• 카프카 auto.offset.reset 종류 및 사용방법
  
• kafka consumer의 commitSync/seek를 이용한 offset 관리 예제
  
• consumer group 이해
  
• Python Kafka Client Benchmarking
  
  • confluent-kafka가 제일 빠르다
    
  • https://docs.confluent.io/current/clients/python.html
    
• Introducing Hive-Kafka Integration for Real-Time Kafka SQL Queries