前文Kubernetes笔记（七）：K8s中的日志采集实践——log-pilot介绍我们对 k8s 集群中常用日志采集模式及阿里开源的 log-pilot 进行了介绍，本文介绍如何使 log-pilot 适配 ELK 7.x 及如何将 log-pilot 部署到 k8s 集群中进行日志采集。
1. 下载 log-pilot 源码
1
| git clone https://github.com/AliyunContainerService/log-pilot
|
如果直接使用作者已经调整过的基于 filebeat 7.3.1 的版本或直接使用作者已经构建好的 Docker 镜像，可直接跳到第4步。
2. 升级 filebeat 版本
修改 Dockerfile.filebeat 文件，将
1
2
3
4
5
6
7
8
9
| ENV FILEBEAT_VERSION=6.1.1-3
COPY assets/glibc/glibc-2.26-r0.apk /tmp/
RUN apk update && \
apk add python && \
apk add ca-certificates && \
apk add wget && \
update-ca-certificates && \
wget http://acs-logging.oss-cn-hangzhou.aliyuncs.com/beats/filebeat/filebeat-${FILEBEAT_VERSION}-linux-x86_64.tar.gz -P /tmp/ && \
mkdir -p /etc/filebeat /var/lib/filebeat /var/log/filebeat && \
|
修改为
1
2
3
4
5
6
7
8
9
| ENV FILEBEAT_VERSION=7.3.1
COPY assets/glibc/glibc-2.26-r0.apk /tmp/
COPY filebeat-${FILEBEAT_VERSION}-linux-x86_64.tar.gz /tmp/
RUN apk update && \
apk add python && \
apk add ca-certificates && \
apk add wget && \
update-ca-certificates && \
mkdir -p /etc/filebeat /var/lib/filebeat /var/log/filebeat && \
|
这里先将 filebeat 包下载下来放到 log-pilot 目录下，避免打镜像时下载太慢。
3. 更新 filebeat 配置
修改 assets/filebeat/config.filebeat 文件，
移除 filebeat.registry_file: /var/lib/filebeat/registry
将 filebeat.config.prospectors: 改为 filebeat.config.inputs:
调整后，配置文件为
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
| base() {
cat >> $FILEBEAT_CONFIG << EOF
path.config: /etc/filebeat
path.logs: /var/log/filebeat
path.data: /var/lib/filebeat/data
filebeat.shutdown_timeout: ${FILEBEAT_SHUTDOWN_TIMEOUT:-0}
logging.level: ${FILEBEAT_LOG_LEVEL:-info}
logging.metrics.enabled: ${FILEBEAT_METRICS_ENABLED:-false}
logging.files.rotateeverybytes: ${FILEBEAT_LOG_MAX_SIZE:-104857600}
logging.files.keepfiles: ${FILEBEAT_LOG_MAX_FILE:-10}
logging.files.permissions: ${FILEBEAT_LOG_PERMISSION:-0600}
${FILEBEAT_MAX_PROCS:+max_procs: ${FILEBEAT_MAX_PROCS}}
setup.template.name: "${FILEBEAT_INDEX:-filebeat}"
setup.template.pattern: "${FILEBEAT_INDEX:-filebeat}-*"
filebeat.config.inputs:
enabled: true
path: \${path.config}/prospectors.d/*.yml
reload.enabled: true
reload.period: 10s
EOF
}
|
4. 获取 Docker 镜像
1.如果是自己修改官方源码，则执行 ./build-image.sh
2.如果是下载作者源码，则
1
2
3
4
| [root@kmaster]# git clone https://github.com/ronwxy/log-pilot.git
[root@kmaster]# cd log-pilot/
[root@kmaster]# git checkout filebeat-7.3.1
[root@kmaster]# ./build-image.sh
|
3.直接下载作者已经构建好的镜像
1
| [root@kmaster]# docker pull registry.cn-hangzhou.aliyuncs.com/jboost/log-pilot:filebeat-7.3.1
|
5. 在 k8s 中部署 log-pilot
我们以 DaemonSet 的方式（一个 Node 一个 Pod）将 log-pilot 部署在 k8s 中，部署配置文件如下
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
| apiVersion: apps/v1
kind: DaemonSet
metadata:
name: log-pilot-filebeat
namespace: kube-system
spec:
selector:
matchLabels:
app: log-pilot-filebeat
template:
metadata:
labels:
app: log-pilot-filebeat
spec:
containers:
- name: log-pilot-filebeat
image: registry.cn-hangzhou.aliyuncs.com/jboost/log-pilot:filebeat-7.3.1
env:
- name: "NODE_NAME"
valueFrom:
fieldRef:
fieldPath: spec.nodeName
- name: "PILOT_LOG_PREFIX"
value: "k8s"
- name: "LOGGING_OUTPUT"
value: "logstash"
- name: "LOGSTASH_HOST"
value: "{your-logstash-host}"
- name: "LOGSTASH_PORT"
value: "5044"
volumeMounts:
- name: sock
mountPath: /var/run/docker.sock
- name: root
mountPath: /host
readOnly: true
- name: varlib
mountPath: /var/lib/filebeat
- name: varlog
mountPath: /var/log/filebeat
- name: localtime
mountPath: /etc/localtime
readOnly: true
livenessProbe:
failureThreshold: 3
exec:
command:
- /pilot/healthz
initialDelaySeconds: 10
periodSeconds: 10
successThreshold: 1
timeoutSeconds: 2
securityContext:
capabilities:
add:
- SYS_ADMIN
volumes:
- name: sock
hostPath:
path: /var/run/docker.sock
- name: root
hostPath:
path: /
- name: varlib
hostPath:
path: /var/lib/filebeat
type: DirectoryOrCreate
- name: varlog
hostPath:
path: /var/log/filebeat
type: DirectoryOrCreate
- name: localtime
hostPath:
path: /etc/localtime
|
6. 应用容器部署配置
在部署应用容器时，以声明式的方式在 Deployment 配置文件的容器部分添加配置即可对容器日志进行自动采集， 如下所示，只列出了与日志配置相关部分
1
2
3
4
5
6
7
8
9
10
11
12
13
| spec:
containers:
- env:
- name: k8s_logs_frameworktest
value: /mnt/logs/app*.log
volumeMounts:
- mountPath: /mnt/logs
name: app-log
volumes:
- emptyDir: {}
name: app-log
|
7. 按环境与应用建立索引
我们可以在 logstash 中根据不同的环境（这里将环境以 namespace 进行划分），及容器名称（即不同的应用）来创建不同的 elasticsearch 的索引。配置参考如下
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
| output {
if [k8s_pod_namespace] == "develop" {
elasticsearch {
hosts => "elasticsearch:9200"
index => "dev-%{[k8s_container_name]}-%{+YYYY.MM.dd}"
user => "elastic"
password => "xxxxxx"
}
} else {
elasticsearch {
hosts => "elasticsearch:9200"
index => "%{[@metadata][beat]}-%{[@metadata][version]}-%{+YYYY.MM.dd}"
user => "elastic"
password => "xxxxxx"
}
}
}
|
8. 相关源码与镜像
log-pilot 官方源码地址：https://github.com/AliyunContainerService/log-pilot
适配 ELK 7.x 源码地址： https://github.com/ronwxy/log-pilot/tree/filebeat-7.3.1
适配 ELK 7.x Docker 镜像地址： registry.cn-hangzhou.aliyuncs.com/jboost/log-pilot:filebeat-7.3.1
