Promtail收集Docker容器日志

通过promtail直接获取docker容器日志

方案

promtail版本需要在 promtail:2.8.3 以上

因为 promtail:2.6.1 存在问题，无法通过配置 docker_sd_config 收集日志，存在以下报错

level=warn ts=2024-10-28T06:26:18.581600589Z caller=target.go:120 target=docker/5be06c2b393cbad92902f0c978f25221a662f68cea170522a8753b27936d36b6 msg="could not transfer logs" written=0 container=5be06c2b393cbad92902f0c978f25221a662f68cea170522a8753b27936d36b6 err="Unrecognized input header: 50"
level=warn ts=2024-10-28T06:26:18.581736775Z caller=target.go:120 target=docker/9751a71b12db7c5c6677c89a7035a71184e6f7c7a9a55a391c1ece92d47a275f msg="could not transfer logs" written=0 container=9751a71b12db7c5c6677c89a7035a71184e6f7c7a9a55a391c1ece92d47a275f err="Unrecognized input header: 50"
level=warn ts=2024-10-28T06:26:18.582212205Z caller=target.go:120 target=docker/682d6b5bef49dc5ca753185e5a69f715d1ffc6da5431701749ceb2e7f9549c1b msg="could not transfer logs" written=0 container=682d6b5bef49dc5ca753185e5a69f715d1ffc6da5431701749ceb2e7f9549c1b err="Unrecognized input header: 50"

原因:

参考：

https://github.com/grafana/loki/issues/5950

已修复

https://github.com/grafana/loki/pull/7829

查看影响版本

https://github.com/grafana/loki/commit/ce09aac46d6d70b75f203144ceb17c579fc81c58

好像从2.8.3 -> v3.2.1之间的都是好的

经确认，2.8.3 版本正常，再高未尝试，应该也可以

配置文件示例

/opt/promtail/promtail-config.yaml

server:
  http_listen_port: 9080
  grpc_listen_port: 0

positions:
  filename: /mnt/config/positions.yaml

clients:
  - url: http://local-loki:3100/loki/api/v1/push
    # 添加扩展标签 hostname 值为环境变量中的 HOSTNAME
    external_labels:
      hostname: ${HOSTNAME}

limits_config:
#  限制每秒日志收集行数
#  readline_rate_enabled: true
#  readline_rate: 4000
#  readline_burst: 5000
  readline_rate_enabled: false
  readline_rate: 10000
  readline_burst: 10000
  readline_rate_drop: false

target_config:
  sync_period: "10s"

scrape_configs:
  - job_name: service_logs
    docker_sd_configs:
      - host: unix:///var/run/docker.sock
        refresh_interval: 5s
    relabel_configs:
      # 获取容器名称作为 service 标签
      - source_labels: ['__meta_docker_container_name']
        regex: '/(.*)'
        target_label: 'service'
      # 获取容器id前八位作为 __container_id__ 标签
      - source_labels: ['__meta_docker_container_id']
        regex: '([0-9a-z]{8}).*'
        target_label: '__container_id__'
      # 添加job标签，值为service_logs
      - target_label: job
        replacement: service_logs
      # 添加agent标签，值为promtail
      - target_label: agent
        replacement: promtail

启动命令

docker run --name promtail \
-h $(hostname) \
-itd \
--cpus 1 -m 2G \
--restart=unless-stopped \
--log-opt max-size=512m \
--log-opt max-file=3 \
-v /etc/hosts:/etc/hosts \
-v /etc/localtime:/etc/localtime \
-v /opt/promtail:/mnt/config \
-v /var/run/docker.sock:/var/run/docker.sock \
grafana/promtail:2.8.3 -config.file=/mnt/config/promtail-config.yaml -config.expand-env=true

说明:

-v /opt/promtail:/mnt/config 挂载配置文件

-v /var/run/docker.sock:/var/run/docker.sock 挂载宿主机docker 的sock文件，为了docker_sd_configs 能够正常使用

-config.file=/mnt/config/promtail-config.yaml 指定配置文件

-config.expand-env=true 启用配置文件使用环境变量