Nginx:一个高性能的HTTP和反向代理web服务器,实际生产中一般用作负载均衡;
Kafka:一种高吞吐量的分布式发布订阅消息系统,实际生产中一般用作流式处理中间件;
Nginx和Kafka都有非常优秀的性能,实际生产中,往往都是通过Nginx记录日志生成日志文件,然后再通过Flume将日志文件的内容读取到Kafka,中间会有一个写文件和读文件操作,性能势必会受到一些影响;
下面介绍一下把Nginx的日志直接写入到Kafka的实现过程:
git主要用于拉取源码文件
yum install -y git
切换到/usr/local/src目录,然后将kafka的c客户端源码clone到本地
cd /usr/local/src git clone https://github.com/edenhill/librdkafka
cd librdkafka # 安装必要的包 yum install -y gcc gcc-c++ pcre-devel zlib-devel ./configure make && make install
安装nginx整合kafka的插件,进入到/usr/local/src,clone nginx整合kafka的源码
cd /usr/local/src git clone https://github.com/brg-liuwei/ngx_kafka_module
进入到nginx的源码包目录下(编译nginx,然后将将插件同时编译)
cd /usr/local/src/nginx-1.20.2 ./configure --add-module=/usr/local/src/ngx_kafka_module/ make make install
[root@hadoop100 conf]# vim nginx.conf # '$status $body_bytes_sent "$http_referer" ' # '"$http_user_agent" "$http_x_forwarded_for"'; #access_log logs/access.log main; sendfile on; #tcp_nopush on; #keepalive_timeout 0; keepalive_timeout 65; #gzip on; # # ############### kafka config ########################### kafka; kafka_broker_list hadoop101:9092 hadoop102:9092 hadoop103:9092; # ############### kafka config ########################### server { listen 80; server_name localhost; #charset koi8-r; #access_log logs/host.access.log main; #location / { # root html; # index index.html index.htm; #} # ############### kafka config ########################### location /kafka/test { kafka_topic test; } # ############### kafka config ########################### #error_page 404 /404.html; # redirect server error pages to the static page /50x.html # error_page 500 502 503 504 /50x.html; location = /50x.html { root html; }
bin/kafka-topics.sh --zookeeper hadoop101:2181 --create --topic test --partitions 3 --replication-factor 2
error while loading shared libraries: librdkafka.so.1: cannot open shared object file: No such file or directory
echo "/usr/local/lib" >> /etc/ld.so.conf ldconfig
curl localhost/kafka/track -d "message send to kafka topic"