Loki Per stream rate limit exceeded 错误排查与解决

查看Promtail日志，发现存在Per stream rate limit exceeded (limit: 3MB/sec)报错，触发了Loki每个流每秒最大字节速率限制，可能导致数据丢失，查询异常问题出现。

Promtail异常日志

level=warn ts=2025-05-06T03:09:37.961274973Z caller=client.go:369 component=client host=local-loki:3100 msg="error sending batch, will retry" status=429 error="server returned HTTP status 429 Too Many Requests (429): entry with timestamp 2025-05-06 03:09:37.873760873 +0000 UTC ignored, reason: 'Per stream rate limit exceeded (limit: 3MB/sec) while attempting to ingest for stream '{agent=\"promtail\", filename=\"/opt/test_service/logs/info.log\", hostname=\"test-001\", job=\"service_logs\", service=\"test_service\"}' totaling 494791B, consider splitting a stream via additional labels or contact your Loki administrator to see if the limit can be increased' for stream: {agent=\"promtail\", filename=\"/opt/test_service/logs/info.log\", hostname=\"test-001\", job=\"service_logs\", service=\"test_service\"},"
level=warn ts=2025-05-06T03:09:39.610424292Z caller=client.go:369 component=client host=local-loki:3100 msg="error sending batch, will retry" status=429 error="server returned HTTP status 429 Too Many Requests (429): entry with timestamp 2025-05-06 03:09:39.552467567 +0000 UTC ignored, reason: 'Per stream rate limit exceeded (limit: 3MB/sec) while attempting to ingest for stream '{agent=\"promtail\", filename=\"/opt/test_service/logs/info.log\", hostname=\"test-001\", job=\"service_logs\", service=\"test_service\"}' totaling 2404B, consider splitting a stream via additional labels or contact your Loki administrator to see if the limit can be increased' for stream: {agent=\"promtail\", filename=\"/opt/test_service/logs/info.log\", hostname=\"test-001\", job=\"service_logs\", service=\"test_service\"},"

查看Loki中流相关配置

limits_config:
  per_stream_rate_limit: 3MB # 每个流每秒的最大字节速率

per_stream_rate_limit_burst 默认值为 15MB

该日志流 {agent="promtail", filename=..., service=...} 的 实时写入速率 超过了你设置的 3MB/s

虽然 burst=15MB，但只是允许瞬间突发，不是持续的高流量

优化

调整Loki中关于每个数据流限制的配置

limits_config:
  per_stream_rate_limit: 30MB # 每个数据流每秒的最大字节速率
  per_stream_rate_limit_burst: 90MB # 每个数据流的最大突发字节数，可以用人类可读的形式表示(1MB, 256KB, 等)

调整后，观察sync-001上的Promtail日志，没有限流报错

官方配置说明

# Maximum byte rate per second per stream, also expressible in human readable
# forms (1MB, 256KB, etc).
# 每秒每个数据流的最大字节速率，也可以用人类可读的形式表示（1MB、256KB 等）。
# CLI flag: -ingester.per-stream-rate-limit
[per_stream_rate_limit: <int> | default = 3MB]

# Maximum burst bytes per stream, also expressible in human readable forms (1MB,
# 256KB, etc). This is how far above the rate limit a stream can 'burst' before
# the stream is limited.
# 每个数据流的最大突发字节数，也可以用人类可读的形式表示（1MB、256KB 等）。这是在数据流受到限制之前，数据流可 “突发 ”的字节数。
# CLI flag: -ingester.per-stream-rate-limit-burst
[per_stream_rate_limit_burst: <int> | default = 15MB]