前文Kubernetes笔记(七):K8s中的日志采集实践——log-pilot介绍我们对 k8s 集群中常用日志采集模式及阿里开源的 log-pilot 进行了介绍,本文介绍如何使 log-pilot 适配 ELK 7.x 及如何将 log-pilot 部署到 k8s 集群中进行日志采集。

1. 下载 log-pilot 源码

1
git clone https://github.com/AliyunContainerService/log-pilot

如果直接使用作者已经调整过的基于 filebeat 7.3.1 的版本或直接使用作者已经构建好的 Docker 镜像,可直接跳到第4步。

2. 升级 filebeat 版本

修改 Dockerfile.filebeat 文件,将

1
2
3
4
5
6
7
8
9
ENV FILEBEAT_VERSION=6.1.1-3
COPY assets/glibc/glibc-2.26-r0.apk /tmp/
RUN apk update && \
apk add python && \
apk add ca-certificates && \
apk add wget && \
update-ca-certificates && \
wget http://acs-logging.oss-cn-hangzhou.aliyuncs.com/beats/filebeat/filebeat-${FILEBEAT_VERSION}-linux-x86_64.tar.gz -P /tmp/ && \
mkdir -p /etc/filebeat /var/lib/filebeat /var/log/filebeat && \

修改为

1
2
3
4
5
6
7
8
9
ENV FILEBEAT_VERSION=7.3.1
COPY assets/glibc/glibc-2.26-r0.apk /tmp/
COPY filebeat-${FILEBEAT_VERSION}-linux-x86_64.tar.gz /tmp/
RUN apk update && \
apk add python && \
apk add ca-certificates && \
apk add wget && \
update-ca-certificates && \
mkdir -p /etc/filebeat /var/lib/filebeat /var/log/filebeat && \

这里先将 filebeat 包下载下来放到 log-pilot 目录下,避免打镜像时下载太慢。

3. 更新 filebeat 配置

修改 assets/filebeat/config.filebeat 文件,

移除 filebeat.registry_file: /var/lib/filebeat/registry
filebeat.config.prospectors: 改为 filebeat.config.inputs:

调整后,配置文件为

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
base() {
cat >> $FILEBEAT_CONFIG << EOF
path.config: /etc/filebeat
path.logs: /var/log/filebeat
path.data: /var/lib/filebeat/data
filebeat.shutdown_timeout: ${FILEBEAT_SHUTDOWN_TIMEOUT:-0}
logging.level: ${FILEBEAT_LOG_LEVEL:-info}
logging.metrics.enabled: ${FILEBEAT_METRICS_ENABLED:-false}
logging.files.rotateeverybytes: ${FILEBEAT_LOG_MAX_SIZE:-104857600}
logging.files.keepfiles: ${FILEBEAT_LOG_MAX_FILE:-10}
logging.files.permissions: ${FILEBEAT_LOG_PERMISSION:-0600}
${FILEBEAT_MAX_PROCS:+max_procs: ${FILEBEAT_MAX_PROCS}}
setup.template.name: "${FILEBEAT_INDEX:-filebeat}"
setup.template.pattern: "${FILEBEAT_INDEX:-filebeat}-*"
filebeat.config.inputs:
enabled: true
path: \${path.config}/prospectors.d/*.yml
reload.enabled: true
reload.period: 10s
EOF
}

4. 获取 Docker 镜像

1.如果是自己修改官方源码,则执行 ./build-image.sh

2.如果是下载作者源码,则

1
2
3
4
[root@kmaster]# git clone https://github.com/ronwxy/log-pilot.git
[root@kmaster]# cd log-pilot/
[root@kmaster]# git checkout filebeat-7.3.1
[root@kmaster]# ./build-image.sh

3.直接下载作者已经构建好的镜像

1
[root@kmaster]# docker pull registry.cn-hangzhou.aliyuncs.com/jboost/log-pilot:filebeat-7.3.1

5. 在 k8s 中部署 log-pilot

我们以 DaemonSet 的方式(一个 Node 一个 Pod)将 log-pilot 部署在 k8s 中,部署配置文件如下

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
apiVersion: apps/v1
kind: DaemonSet
metadata:
name: log-pilot-filebeat
namespace: kube-system
spec:
selector:
matchLabels:
app: log-pilot-filebeat
template:
metadata:
labels:
app: log-pilot-filebeat
spec:
containers:
- name: log-pilot-filebeat
#image: registry.cn-hangzhou.aliyuncs.com/acs/log-pilot:0.9.7-filebeat
image: registry.cn-hangzhou.aliyuncs.com/jboost/log-pilot:filebeat-7.3.1
env:
- name: "NODE_NAME"
valueFrom:
fieldRef:
fieldPath: spec.nodeName
- name: "PILOT_LOG_PREFIX"
value: "k8s"
- name: "LOGGING_OUTPUT"
value: "logstash"
- name: "LOGSTASH_HOST"
value: "{your-logstash-host}"
- name: "LOGSTASH_PORT"
value: "5044"
volumeMounts:
- name: sock
mountPath: /var/run/docker.sock
- name: root
mountPath: /host
readOnly: true
- name: varlib
mountPath: /var/lib/filebeat
- name: varlog
mountPath: /var/log/filebeat
- name: localtime
mountPath: /etc/localtime
readOnly: true
livenessProbe:
failureThreshold: 3
exec:
command:
- /pilot/healthz
initialDelaySeconds: 10
periodSeconds: 10
successThreshold: 1
timeoutSeconds: 2
securityContext:
capabilities:
add:
- SYS_ADMIN
volumes:
- name: sock
hostPath:
path: /var/run/docker.sock
- name: root
hostPath:
path: /
- name: varlib
hostPath:
path: /var/lib/filebeat
type: DirectoryOrCreate
- name: varlog
hostPath:
path: /var/log/filebeat
type: DirectoryOrCreate
- name: localtime
hostPath:
path: /etc/localtime

6. 应用容器部署配置

在部署应用容器时,以声明式的方式在 Deployment 配置文件的容器部分添加配置即可对容器日志进行自动采集, 如下所示,只列出了与日志配置相关部分

1
2
3
4
5
6
7
8
9
10
11
12
13
spec:
containers:
- env:
- name: k8s_logs_frameworktest
value: /mnt/logs/app*.log

volumeMounts:
- mountPath: /mnt/logs
name: app-log

volumes:
- emptyDir: {}
name: app-log

7. 按环境与应用建立索引

我们可以在 logstash 中根据不同的环境(这里将环境以 namespace 进行划分),及容器名称(即不同的应用)来创建不同的 elasticsearch 的索引。配置参考如下

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
output {
if [k8s_pod_namespace] == "develop" {
elasticsearch {
hosts => "elasticsearch:9200"
index => "dev-%{[k8s_container_name]}-%{+YYYY.MM.dd}"
user => "elastic"
password => "xxxxxx"
}


} else {
elasticsearch {
hosts => "elasticsearch:9200"
index => "%{[@metadata][beat]}-%{[@metadata][version]}-%{+YYYY.MM.dd}"
user => "elastic"
password => "xxxxxx"
}
}
}

8. 相关源码与镜像

log-pilot 官方源码地址:https://github.com/AliyunContainerService/log-pilot
适配 ELK 7.x 源码地址: https://github.com/ronwxy/log-pilot/tree/filebeat-7.3.1
适配 ELK 7.x Docker 镜像地址: registry.cn-hangzhou.aliyuncs.com/jboost/log-pilot:filebeat-7.3.1

评论