前文Kubernetes笔记(七):K8s中的日志采集实践——log-pilot介绍我们对 k8s 集群中常用日志采集模式及阿里开源的 log-pilot 进行了介绍,本文介绍如何使 log-pilot 适配 ELK 7.x 及如何将 log-pilot 部署到 k8s 集群中进行日志采集。
1. 下载 log-pilot 源码
1
| git clone https://github.com/AliyunContainerService/log-pilot
|
如果直接使用作者已经调整过的基于 filebeat 7.3.1 的版本或直接使用作者已经构建好的 Docker 镜像,可直接跳到第4步。
2. 升级 filebeat 版本
修改 Dockerfile.filebeat 文件,将
1 2 3 4 5 6 7 8 9
| ENV FILEBEAT_VERSION=6.1.1-3 COPY assets/glibc/glibc-2.26-r0.apk /tmp/ RUN apk update && \ apk add python && \ apk add ca-certificates && \ apk add wget && \ update-ca-certificates && \ wget http://acs-logging.oss-cn-hangzhou.aliyuncs.com/beats/filebeat/filebeat-${FILEBEAT_VERSION}-linux-x86_64.tar.gz -P /tmp/ && \ mkdir -p /etc/filebeat /var/lib/filebeat /var/log/filebeat && \
|
修改为
1 2 3 4 5 6 7 8 9
| ENV FILEBEAT_VERSION=7.3.1 COPY assets/glibc/glibc-2.26-r0.apk /tmp/ COPY filebeat-${FILEBEAT_VERSION}-linux-x86_64.tar.gz /tmp/ RUN apk update && \ apk add python && \ apk add ca-certificates && \ apk add wget && \ update-ca-certificates && \ mkdir -p /etc/filebeat /var/lib/filebeat /var/log/filebeat && \
|
这里先将 filebeat 包下载下来放到 log-pilot 目录下,避免打镜像时下载太慢。
3. 更新 filebeat 配置
修改 assets/filebeat/config.filebeat 文件,
移除 filebeat.registry_file: /var/lib/filebeat/registry
将 filebeat.config.prospectors:
改为 filebeat.config.inputs:
调整后,配置文件为
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21
| base() { cat >> $FILEBEAT_CONFIG << EOF path.config: /etc/filebeat path.logs: /var/log/filebeat path.data: /var/lib/filebeat/data filebeat.shutdown_timeout: ${FILEBEAT_SHUTDOWN_TIMEOUT:-0} logging.level: ${FILEBEAT_LOG_LEVEL:-info} logging.metrics.enabled: ${FILEBEAT_METRICS_ENABLED:-false} logging.files.rotateeverybytes: ${FILEBEAT_LOG_MAX_SIZE:-104857600} logging.files.keepfiles: ${FILEBEAT_LOG_MAX_FILE:-10} logging.files.permissions: ${FILEBEAT_LOG_PERMISSION:-0600} ${FILEBEAT_MAX_PROCS:+max_procs: ${FILEBEAT_MAX_PROCS}} setup.template.name: "${FILEBEAT_INDEX:-filebeat}" setup.template.pattern: "${FILEBEAT_INDEX:-filebeat}-*" filebeat.config.inputs: enabled: true path: \${path.config}/prospectors.d/*.yml reload.enabled: true reload.period: 10s EOF }
|
4. 获取 Docker 镜像
1.如果是自己修改官方源码,则执行 ./build-image.sh
2.如果是下载作者源码,则
1 2 3 4
| [root@kmaster]# git clone https://github.com/ronwxy/log-pilot.git [root@kmaster]# cd log-pilot/ [root@kmaster]# git checkout filebeat-7.3.1 [root@kmaster]# ./build-image.sh
|
3.直接下载作者已经构建好的镜像
1
| [root@kmaster]# docker pull registry.cn-hangzhou.aliyuncs.com/jboost/log-pilot:filebeat-7.3.1
|
5. 在 k8s 中部署 log-pilot
我们以 DaemonSet 的方式(一个 Node 一个 Pod)将 log-pilot 部署在 k8s 中,部署配置文件如下
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75
| apiVersion: apps/v1 kind: DaemonSet metadata: name: log-pilot-filebeat namespace: kube-system spec: selector: matchLabels: app: log-pilot-filebeat template: metadata: labels: app: log-pilot-filebeat spec: containers: - name: log-pilot-filebeat image: registry.cn-hangzhou.aliyuncs.com/jboost/log-pilot:filebeat-7.3.1 env: - name: "NODE_NAME" valueFrom: fieldRef: fieldPath: spec.nodeName - name: "PILOT_LOG_PREFIX" value: "k8s" - name: "LOGGING_OUTPUT" value: "logstash" - name: "LOGSTASH_HOST" value: "{your-logstash-host}" - name: "LOGSTASH_PORT" value: "5044" volumeMounts: - name: sock mountPath: /var/run/docker.sock - name: root mountPath: /host readOnly: true - name: varlib mountPath: /var/lib/filebeat - name: varlog mountPath: /var/log/filebeat - name: localtime mountPath: /etc/localtime readOnly: true livenessProbe: failureThreshold: 3 exec: command: - /pilot/healthz initialDelaySeconds: 10 periodSeconds: 10 successThreshold: 1 timeoutSeconds: 2 securityContext: capabilities: add: - SYS_ADMIN volumes: - name: sock hostPath: path: /var/run/docker.sock - name: root hostPath: path: / - name: varlib hostPath: path: /var/lib/filebeat type: DirectoryOrCreate - name: varlog hostPath: path: /var/log/filebeat type: DirectoryOrCreate - name: localtime hostPath: path: /etc/localtime
|
6. 应用容器部署配置
在部署应用容器时,以声明式的方式在 Deployment 配置文件的容器部分添加配置即可对容器日志进行自动采集, 如下所示,只列出了与日志配置相关部分
1 2 3 4 5 6 7 8 9 10 11 12 13
| spec: containers: - env: - name: k8s_logs_frameworktest value: /mnt/logs/app*.log
volumeMounts: - mountPath: /mnt/logs name: app-log
volumes: - emptyDir: {} name: app-log
|
7. 按环境与应用建立索引
我们可以在 logstash 中根据不同的环境(这里将环境以 namespace 进行划分),及容器名称(即不同的应用)来创建不同的 elasticsearch 的索引。配置参考如下
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19
| output { if [k8s_pod_namespace] == "develop" { elasticsearch { hosts => "elasticsearch:9200" index => "dev-%{[k8s_container_name]}-%{+YYYY.MM.dd}" user => "elastic" password => "xxxxxx" }
} else { elasticsearch { hosts => "elasticsearch:9200" index => "%{[@metadata][beat]}-%{[@metadata][version]}-%{+YYYY.MM.dd}" user => "elastic" password => "xxxxxx" } } }
|
8. 相关源码与镜像
log-pilot 官方源码地址:https://github.com/AliyunContainerService/log-pilot
适配 ELK 7.x 源码地址: https://github.com/ronwxy/log-pilot/tree/filebeat-7.3.1
适配 ELK 7.x Docker 镜像地址: registry.cn-hangzhou.aliyuncs.com/jboost/log-pilot:filebeat-7.3.1