flume 配置实例.md
原创
文章标签 flume hdfs kafka hadoop 文章分类 Hadoop 大数据
©著作权归作者所有：来自51CTO博客作者大数据工匠的原创作品，请联系作者获取转载授权，否则将追究法律责任
从文件到kafka
flume-directory-kafka.conf: A single-node Flume configuration# Name the components on this agenta3.sources = r3a3.sinks = k3a3.channels = c3# Describe/configure the sourcea3.sources.r3.type = execa3.sources.r3.channels = c3a3.sources.r3.command = tail -F /usr/local/nginx/logs/access.log# Describe the sinka3.sinks.k3.type = org.apache.flume.sink.kafka.KafkaSinka3.sinks.k3.topic = countly_small_sysa3.sinks.k3.brokerList = node6.hadoop.fzzqxf.com:9092,node5.hadoop.fzzqxf.com:9092,node4.hadoop.fzzqxf.com:9092,node3.hadoop.fzzqxf.com:9092,node2.hadoop.fzzqxf.com:9092a3.sinks.k3.requiredAcks = 1a3.sinks.k3.batchSize = 20a3.sinks.k3.channel = c3# Use a channel which buffers events in memorya3.channels.c3.type = memorya3.channels.c3.capacity = 100000a3.channels.c3.transactionCapacity = 1000# Bind the source and sink to the channela3.sources.r3.channels = c3a3.sinks.k3.channel = c3
从kafka 到hdfs
list the sources, sinks and channels for the agentagent_collector.sources = kafka-src-1agent_collector.sinks = hdfs-sink-1agent_collector.channels = mem-channel-1# set channel for sourceagent_collector.sources.kafka-src-1.channels = mem-channel-1# set channel for sinkagent_collector.sinks.hdfs-sink-1.channel = mem-channel-1# properties for sourcesagent_collector.sources.kafka-src-1.type = org.apache.flume.source.kafka.KafkaSourceagent_collector.sources.kafka-src-1.channels = mem-channel-1agent_collector.sources.kafka-src-1.zookeeperConnect = master.hadoop.fzzqxf.com:2181,node1.hadoop.fzzqxf.com:2181,node4.hadoop.fzzqxf.com:2181,node5.hadoop.fzzqxf.com:2181,node6.hadoop.fzzqxf.com:2181agent_collector.sources.kafka-src-1.topic = siege-countly_log-2agent_collector.sources.kafka-src-1.groupId = flumeCountlyHdfsagent_collector.sources.kafka-src-1.batchSize=200agent_collector.sources.kafka-src-1.kafka.consumer.timeout.ms = 100# properties for channelsagent_collector.channels.mem-channel-1.type = memoryagent_collector.channels.mem-channel-1.capacity = 10000agent_collector.channels.mem-channel-1.transactionCapacity = 10000agent_collector.channels.mem-channel-1.byteCapacityBufferPercentage = 20agent_collector.channels.mem-channel-1.byteCapacity = 10485760# properties for sinksagent_collector.channels = mem-channel-1agent_collector.sinks = hdfs-sink-1agent_collector.sinks.hdfs-sink-1.type = hdfsagent_collector.sinks.hdfs-sink-1.channel = mem-channel-1agent_collector.sinks.hdfs-sink-1.hdfs.path = /data/logs/flume/realtime/countly/%Y%m%d agent_collector.sinks.hdfs-sink-1.hdfs.filePrefix = collector_1agent_collector.sinks.hdfs-sink-1.hdfs.fileType=DataStreamagent_collector.sinks.hdfs-sink-1.hdfs.minBlockReplicas=1agent_collector.sinks.hdfs-sink-1.hdfs.rollInterval=21600# unit is bytes. value is 1G*100=13421772800.agent_collector.sinks.hdfs-sink-1.hdfs.rollSize=0agent_collector.sinks.hdfs-sink-1.hdfs.rollCount=0agent_collector.sinks.hdfs-sink-1.hdfs.batchSize=1000agent_collector.sinks.hdfs-sink-1.hdfs.callTimeout=60000
作者：少帅
您的支持是对博主最大的鼓励，感谢您的认真阅读。
本文版权归作者所有，欢迎转载，但请保留该声明。
- 赞
- 收藏
- 评论
- *举报
上一篇：Inceptor命令01-表介绍
下一篇：holodesk表存储-列重复值太多，导致存储占用空间翻很多倍
Original: https://blog.51cto.com/u_15683012/5383071
Author: 大数据工匠
Title: flume 配置实例.md
原创文章受到原创版权保护。转载请注明出处：https://www.johngo689.com/517416/
转载文章受原作者版权保护。转载请注明原作者出处！