标签 问题记录 下的文章

摘要

有台 Ubuntu 14.04 的机器,想要查看sshd登录日志,发现没有 /var/log/auth.log 日志文件,同时连 /var/log/syslog 日志文件都不存在,遂排查并处理该问题。

基本情况:

1.rsyslog服务正常启动

2.rsyslog配置了记录日志文件

3./var/log目录中,没有相关日志文件

原因:

/var/log 目录权限是root:root 0755,rsyslog服务配置的用户和组是 syslog:adm,导致无权限在/var/log目录创建相关日志文件。

解决方法:

更改目录所有者和组

sudo chown syslog.adm /var/log/

- 阅读剩余部分 -

问题

错误修改/etc/sudoers文件,导致出现sudo命令无法正常使用的问题出现。

>>> /etc/sudoers: syntax error near line 21 <<<
sudo: parse error in /etc/sudoers near line 21
sudo: no valid sudoers sources found, quitting
sudo: unable to initialize policy plugin

注意:

这里的问题,是在Ubuntu系统环境下出现的,是基于该系统进行的整理。

- 阅读剩余部分 -

通过ssh连接服务器失败

$ ssh root@172.16.0.247
Unable to negotiate with 172.16.0.247 port 22: no matching host key type found. Their offer: ssh-rsa,ssh-dss

SSH 客户端和远程服务器无法协商使用相同的 host key 类型。

解决方法:

添加参数

# -oHostKeyAlgorithms=+ssh-rsa           允许接收远程主机提供的 ssh-rsa 类型的 主机密钥(连接握手阶段)
# -oPubkeyAcceptedAlgorithms=+ssh-rsa    允许使用 ssh-rsa 类型的 用户身份验证公钥(登录认证阶段)
ssh root@172.16.0.247 -oHostKeyAlgorithms=+ssh-rsa -oPubkeyAcceptedAlgorithms=+ssh-rsa

- 阅读剩余部分 -

Promtail报错

level=warn ts=2025-05-07T06:44:22.715765487Z caller=client.go:369 component=client host=local-loki:3100 msg="error sending batch, will retry" status=500 error="server returned HTTP status 500 Internal Server Error (500): rpc error: code = DeadlineExceeded desc = context deadline exceeded"

查看loki 日志

level=info ts=2025-05-07T06:44:11.830518711Z caller=checkpoint.go:615 msg="starting checkpoint"
level=info ts=2025-05-07T06:44:11.830788697Z caller=checkpoint.go:340 msg="attempting checkpoint for" dir=/data/wal/checkpoint.1237659
level=warn ts=2025-05-07T06:44:20.643216928Z caller=logging.go:86 traceID=1f665e71538b4138 orgID=fake msg="POST /loki/api/v1/push (500) 5.012615543s Response: \"rpc error: code = DeadlineExceeded desc = context deadline exceeded\\n\" ws: false; Connection: close; Content-Length: 266310; Content-Type: application/x-protobuf; User-Agent: promtail/2.6.1; "
level=warn ts=2025-05-07T06:44:21.530545119Z caller=logging.go:86 traceID=1e71d2fb0b8fa807 orgID=fake msg="POST /loki/api/v1/push (500) 5.289903404s Response: \"rpc error: code = DeadlineExceeded desc = context deadline exceeded\\n\" ws: false; Connection: close; Content-Length: 217308; Content-Type: application/x-protobuf; User-Agent: promtail/2.6.1; "
level=warn ts=2025-05-07T06:44:22.441075158Z caller=logging.go:86 traceID=646a00c6b84f9eff orgID=fake msg="POST /loki/api/v1/push (500) 5.100367208s Response: \"rpc error: code = DeadlineExceeded desc = context deadline exceeded\\n\" ws: false; Connection: close; Content-Length: 265673; Content-Type: application/x-protobuf; User-Agent: promtail/2.6.1; "
level=warn ts=2025-05-07T06:44:22.448904508Z caller=logging.go:86 traceID=7c39e6b8a70cd408 orgID=fake msg="POST /loki/api/v1/push (500) 5.31823812s Response: \"rpc error: code = DeadlineExceeded desc = context deadline exceeded\\n\" ws: false; Connection: close; Content-Length: 267119; Content-Type: application/x-protobuf; User-Agent: promtail/2.6.1; "
level=info ts=2025-05-07T06:45:04.018620273Z caller=table_manager.go:134 msg="uploading tables"
level=info ts=2025-05-07T06:45:04.018708423Z caller=index_set.go:86 msg="uploading table index_20215"

context deadline exceeded 超出上下文截至时间

这几个报错的时间都在5s,应该是有配置限制

处理方法

参考:https://github.com/grafana/loki/issues/6182#issuecomment-1695993787

# Ingester_client块配置了分配器distributor将如何连接到摄入器ingesters。仅在运行所有组件,分销商distributor或查询querier时都适当。
ingester_client:
  remote_timeout: 10s # 客户端的远程请求超时时间

默认配置

# The remote request timeout on the client side.
# 客户端的远程请求超时。
# CLI flag: -ingester.client.timeout
[remote_timeout: <duration> | default = 5s]

查看Promtail日志,发现存在Per stream rate limit exceeded (limit: 3MB/sec)报错,触发了Loki每个流每秒最大字节速率限制,可能导致数据丢失,查询异常问题出现。

Promtail异常日志

level=warn ts=2025-05-06T03:09:37.961274973Z caller=client.go:369 component=client host=local-loki:3100 msg="error sending batch, will retry" status=429 error="server returned HTTP status 429 Too Many Requests (429): entry with timestamp 2025-05-06 03:09:37.873760873 +0000 UTC ignored, reason: 'Per stream rate limit exceeded (limit: 3MB/sec) while attempting to ingest for stream '{agent=\"promtail\", filename=\"/opt/test_service/logs/info.log\", hostname=\"test-001\", job=\"service_logs\", service=\"test_service\"}' totaling 494791B, consider splitting a stream via additional labels or contact your Loki administrator to see if the limit can be increased' for stream: {agent=\"promtail\", filename=\"/opt/test_service/logs/info.log\", hostname=\"test-001\", job=\"service_logs\", service=\"test_service\"},"
level=warn ts=2025-05-06T03:09:39.610424292Z caller=client.go:369 component=client host=local-loki:3100 msg="error sending batch, will retry" status=429 error="server returned HTTP status 429 Too Many Requests (429): entry with timestamp 2025-05-06 03:09:39.552467567 +0000 UTC ignored, reason: 'Per stream rate limit exceeded (limit: 3MB/sec) while attempting to ingest for stream '{agent=\"promtail\", filename=\"/opt/test_service/logs/info.log\", hostname=\"test-001\", job=\"service_logs\", service=\"test_service\"}' totaling 2404B, consider splitting a stream via additional labels or contact your Loki administrator to see if the limit can be increased' for stream: {agent=\"promtail\", filename=\"/opt/test_service/logs/info.log\", hostname=\"test-001\", job=\"service_logs\", service=\"test_service\"},"

- 阅读剩余部分 -