OpsDaiLou - Loki
type: channel
link: https://blog.opsdailou.top/index.php/tag/Loki/
items: Loki context deadline exceeded 错误排查与解决
items: Loki Per stream rate limit exceeded 错误排查与解决
items: Loki Ingestion rate limit exceeded 错误排查与解决
items: Loki日志收集系统架构

Loki context deadline exceeded 错误排查与解决
type: item
date: 2025-05-08T07:40:01+00:00
description: Promtail报错level=warn ts=2025-05-07T06:44:22.715765487Z caller=client.go:369 component=client host=local-loki:3100 msg="error sending batch, will retry" status=500 error="server returned HTTP status 500 Internal Server Error (500): rpc error: code = DeadlineExceeded desc = context deadline exceeded"查看loki 日志level=info ts=2025-05-07T06:44:11.830518711Z caller=checkpoint.go:615 msg="starting checkpoint" level=info ts=2025-05-07T06:44:11.830788697Z caller=checkpoint.go:340 msg="attempting checkpoint for" dir=/data/wal/checkpoint.1237659 level=warn ts=2025-05-07T06:44:20.643216928Z caller=logging.go:86 traceID=1f665e71538b4138 orgID=fake msg="POST /loki/api/v1/push (500) 5.012615543s Response: \"rpc error: code = DeadlineExceeded desc = context deadline exceeded\\n\" ws: false; Connection: close; Content-Length: 266310; Content-Type: application/x-protobuf; User-Agent: promtail/2.6.1; " level=warn ts=2025-05-07T06:44:21.530545119Z caller=logging.go:86 traceID=1e71d2fb0b8fa807 orgID=fake msg="POST /loki/api/v1/push (500) 5.289903404s Response: \"rpc error: code = DeadlineExceeded desc = context deadline exceeded\\n\" ws: false; Connection: close; Content-Length: 217308; Content-Type: application/x-protobuf; User-Agent: promtail/2.6.1; " level=warn ts=2025-05-07T06:44:22.441075158Z caller=logging.go:86 traceID=646a00c6b84f9eff orgID=fake msg="POST /loki/api/v1/push (500) 5.100367208s Response: \"rpc error: code = DeadlineExceeded desc = context deadline exceeded\\n\" ws: false; Connection: close; Content-Length: 265673; Content-Type: application/x-protobuf; User-Agent: promtail/2.6.1; " level=warn ts=2025-05-07T06:44:22.448904508Z caller=logging.go:86 traceID=7c39e6b8a70cd408 orgID=fake msg="POST /loki/api/v1/push (500) 5.31823812s Response: \"rpc error: code = DeadlineExceeded desc = context deadline exceeded\\n\" ws: false; Connection: close; Content-Length: 267119; Content-Type: application/x-protobuf; User-Agent: promtail/2.6.1; " level=info ts=2025-05-07T06:45:04.0186…

Loki Per stream rate limit exceeded 错误排查与解决
type: item
date: 2025-05-08T07:25:57+00:00
description: 查看Promtail日志，发现存在Per stream rate limit exceeded (limit: 3MB/sec)报错，触发了Loki每个流每秒最大字节速率限制，可能导致数据丢失，查询异常问题出现。Promtail异常日志level=warn ts=2025-05-06T03:09:37.961274973Z caller=client.go:369 component=client host=local-loki:3100 msg="error sending batch, will retry" status=429 error="server returned HTTP status 429 Too Many Requests (429): entry with timestamp 2025-05-06 03:09:37.873760873 +0000 UTC ignored, reason: 'Per stream rate limit exceeded (limit: 3MB/sec) while attempting to ingest for stream '{agent=\"promtail\", filename=\"/opt/test_service/logs/info.log\", hostname=\"test-001\", job=\"service_logs\", service=\"test_service\"}' totaling 494791B, consider splitting a stream via additional labels or contact your Loki administrator to see if the limit can be increased' for stream: {agent=\"promtail\", filename=\"/opt/test_service/logs/info.log\", hostname=\"test-001\", job=\"service_logs\", service=\"test_service\"}," level=warn ts=2025-05-06T03:09:39.610424292Z caller=client.go:369 component=client host=local-loki:3100 msg="error sending batch, will retry" status=429 error="server returned HTTP status 429 Too Many Requests (429): entry with timestamp 2025-05-06 03:09:39.552467567 +0000 UTC ignored, reason: 'Per stream rate limit exceeded (limit: 3MB/sec) while attempting to ingest for stream '{agent=\"promtail\", filename=\"/opt/test_service/logs/info.log\", hostname=\"test-001\", job=\"service_logs\", service=\"test_service\"}' totaling 2404B, consider splitting a stream via additional labels or contact your Loki administrator to see if the limit can be increased' for stream: {agent=\"promtail\", filename=\"/opt/test_service/logs/info.log\", hostname=\"test-001\", job=\"service_logs\", service=\"test_service\"},"[...]

Loki Ingestion rate limit exceeded 错误排查与解决
type: item
date: 2025-05-08T07:25:31+00:00
description: 问题描述通过Grafana查询Loki数据源中的日志，发现存在指定时间段内，查询数据与实际文件中的日志不符的现象。排查发现Promtail存在报错日志server returned HTTP status 429 Too Many Requests (429): Ingestion rate limit exceeded for user fake (limit: 41943040 bytes/sec)，超过了Loki配置的速率上限，被限流了。遂进行调整Loki配置，进行处理。[...]

Loki日志收集系统架构
type: item
date: 2025-04-22T04:02:17+00:00
description: 通过Loki+Promtail+Grafana，搭建日志收集系统，可以用于收集Docker日志，nginx日志，kubernetes日志通过Loki存储日志数据通过promtail收集日志，上传到Loki数据源通过Grafana连接Loki数据源，查询展示数据官方文档地址LokiPromtailGrafana[...]