本文默认k8s环境以及已经部署ingress controller
公司所用ingress监控是由prometheus+grafana进行,但是监控不够全面,故使用filebeat去采集ingress日志,并自主进行可视化展示
在ingress controller中将configmap改为
kind: ConfigMap apiVersion: v1 metadata: name: ingress-nginx-controller data: access-log-path: /var/log/nginx/access.log compute-full-forwarded-for: 'true' enable-vts-status: 'true' error-log-path: /var/log/nginx/error.log forwarded-for-header: X-Forwarded-For log-format-upstream: >- {"@timestamp": "$time_iso8601","remote_addr":"$remote_addr","x-forward-for":"$http_x_forwarded_for","request_id":"$req_id","remote_user":"$remote_user","bytes_sent":$bytes_sent,"request_time":$request_time,"status":$status,"vhost":"$host","request_proto":"$server_protocol","path":"$uri","request_query":"$args","request_length":$request_length,"duration":$request_time,"method":"$request_method","http_referrer":"$http_referer","http_user_agent":"$http_user_agent","upstream-sever":"$proxy_upstream_name","proxy_alternative_upstream_name":"$proxy_alternative_upstream_name","upstream_addr":"$upstream_addr","upstream_response_length":$upstream_response_length,"upstream_response_time":$upstream_response_time,"upstream_status":$upstream_status} use-forwarded-headers: 'true'
新建目录,目录如下
dockerfile
FROM million12/centos-supervisor:4.0.2 WORKDIR /usr/local ADD filebeat-7.5.0-linux-x86_64.tar.gz . RUN ln -s filebeat-7.5.0-linux-x86_64 filebeat \ && cd filebeat \ && mkdir config \ && chmod +x filebeat \ && cp filebeat.yml config/ \ && yum -y install logrotate crontabs COPY supervisord.conf /etc/supervisord.conf RUN mkdir -p /var/log/supervisor EXPOSE 22 80 CMD ["/usr/bin/supervisord"]
因为需要使用logrotate进行日志轮转,需要安装
logrotate crontabs
supervisord.conf配置如下
[supervisord] nodaemon=true [program:cron] command=/usr/sbin/crond -i [program:filebeat] command=/usr/local/filebeat/filebeat -c /usr/local/filebeat/config/filebeat.yml
kind: ConfigMap apiVersion: v1 metadata: name: filebeat-config data: filebeat.yml: | filebeat.inputs: - type: log enabled: true paths: - /var/log/nginx/access.log json.keys_under_root: true json.overwrite_keys: true json.add_error_key: true json.ignore_decoding_error: true tags: ["access"] - type: log enabled: true paths: - /var/log/nginx/error.log json.keys_under_root: true json.overwrite_keys: true json.add_error_key: true json.ignore_decoding_error: true tags: ["error"] filebeat.config.modules: path: ${path.config}/modules.d/*.yml reload.enabled: false setup.template.settings: index.number_of_shards: 3 output.elasticsearch: hosts: ["es-local.nxgp.svc.cluster.local:9200"] index: "nginx_log-%{+yyyy.MM.dd}" indices: - index: "nginx_access-%{[beat.version]}-%{+yyyy.MM.dd}" when.contains: tags: "access" - index: "nginx_error-%{[beat.version]}-%{+yyyy.MM.dd}" when.contains: tags: "error" setup.template.name: "nginx_log" setup.template.pattern: "nginx_*" setup.template.enabled: true setup.ilm.enabled: false setup.template.overwrite: false
kind: ConfigMap apiVersion: v1 metadata: name: nginx-ingress-logrotate data: nginx: | /var/log/nginx/*.log { su root root size 50M notifempty copytruncate rotate 3 missingok compress dateext dateformat .%Y%m%d-%H }
然后进行depl更新,只展示新增部分
volumes: - name: ingress-log emptyDir: {} - name: filebeat-config configMap: name: filebeat-config defaultMode: 420 - name: logrotateconf configMap: name: nginx-ingress-logrotate items: - key: nginx path: nginx defaultMode: 420 containers: - name: controller volumeMounts: - name: ingress-log mountPath: /var/log/nginx/ - name: filebeat image: 'xxx/filebeat:7.5.0' resources: limits: cpu: '2' memory: 2Gi requests: cpu: '1' memory: 1Gi volumeMounts: - name: filebeat-config mountPath: /usr/local/filebeat/config/ - name: ingress-log mountPath: /var/log/nginx/ - name: logrotateconf mountPath: /etc/logrotate.d/nginx subPath: nginx terminationMessagePath: /dev/termination-log terminationMessagePolicy: File imagePullPolicy: Always restartPolicy: Always terminationGracePeriodSeconds: 300 dnsPolicy: ClusterFirst nodeSelector: kubernetes.io/os: linux serviceAccountName: ingress-nginx serviceAccount: ingress-nginx securityContext: {} schedulerName: default-scheduler strategy: type: RollingUpdate rollingUpdate: maxUnavailable: 25% maxSurge: 25% revisionHistoryLimit: 10 progressDeadlineSeconds: 600
logrotate是按天更新,更新时间不定
轮转效果内存占用高的是还没有进行打包的 4、es可视化展示
(1)PV
(2)UV
(3)Top10(接口访问量)
(4)Top10(客户端IP访问占比)
(5)Top10(最慢接口)
(6)后端upstream占比
(7)实时流量
(8)客户端访问占比
(9)平均并发数
(10)异常状态码统计
(11)总流量
(12)接口异常响应码
(13)接口访问耗时占比
(14)每10秒接口访问平均耗时
(15)每10秒接口访问最大耗时
(16)状态码统计
(17)访问量趋势图
(18)超过30秒以上的接口
(19)超过30秒以上的接口出现次数