为什么ulimit配置对service服务不生效?
本地通过ulimit已经配置了最大文件打开数,但是发现对systemd管理的nginx服务不生效。通过调查分析发现,使用命令行启动nginx和使用systemctl启动nginx,limit限制是不同的。
以下是分析过程:
1.查看ulimit配置
查看已配置的ulimit限制,最大文件打开数配置的是 65535
swift@server:~$ ulimit -a
core file size (blocks, -c) 0
data seg size (kbytes, -d) unlimited
scheduling priority (-e) 0
file size (blocks, -f) unlimited
pending signals (-i) 15576
max locked memory (kbytes, -l) 64
max memory size (kbytes, -m) unlimited
open files (-n) 65535
pipe size (512 bytes, -p) 8
POSIX message queues (bytes, -q) 819200
real-time priority (-r) 0
stack size (kbytes, -s) 8192
cpu time (seconds, -t) unlimited
max user processes (-u) 15576
virtual memory (kbytes, -v) unlimited
file locks (-x) unlimited
swift@server:~$2.对比命令行启动和systemd启动服务后的limit限制
分别以两种方式启动nginx服务,查看进程号PID和limit值限制(Max open files)
2.1 命令行启动nginx
可以发现Max open files 限制是 65535,和本地ulimti -n查看到的配置65535一致
swift@server:~$ sudo nginx
swift@server:~$ ps -ef |grep nginx
root 3946 1 0 18:17 ? 00:00:00 nginx: master process nginx
www-data 3947 3946 0 18:17 ? 00:00:00 nginx: worker process
www-data 3948 3946 0 18:17 ? 00:00:00 nginx: worker process
swift 3950 1614 0 18:17 pts/0 00:00:00 grep --color=auto nginx
swift@server:~$ cat /proc/3946/limits
Limit Soft Limit Hard Limit Units
Max cpu time unlimited unlimited seconds
Max file size unlimited unlimited bytes
Max data size unlimited unlimited bytes
Max stack size 8388608 unlimited bytes
Max core file size 0 unlimited bytes
Max resident set unlimited unlimited bytes
Max processes 15576 15576 processes
Max open files 65535 65535 files
Max locked memory 65536 65536 bytes
Max address space unlimited unlimited bytes
Max file locks unlimited unlimited locks
Max pending signals 15576 15576 signals
Max msgqueue size 819200 819200 bytes
Max nice priority 0 0
Max realtime priority 0 0
Max realtime timeout unlimited unlimited us2.2 使用systemctl start nginx.server启动nginx
可以发现Max open files 是 1024,而不是本地ulimti -n查看到的配置65535
swift@server:~$ sudo systemctl start nginx.service
swift@server:~$ ps -ef |grep nginx
root 4001 1 0 18:25 ? 00:00:00 nginx: master process /usr/sbin/nginx -g daemon on; master_process on;
www-data 4003 4001 0 18:25 ? 00:00:00 nginx: worker process
www-data 4004 4001 0 18:25 ? 00:00:00 nginx: worker process
swift 4008 1614 0 18:25 pts/0 00:00:00 grep --color=auto nginx
swift@server:~$ cat /proc/4001/limits
Limit Soft Limit Hard Limit Units
Max cpu time unlimited unlimited seconds
Max file size unlimited unlimited bytes
Max data size unlimited unlimited bytes
Max stack size 8388608 unlimited bytes
Max core file size 0 unlimited bytes
Max resident set unlimited unlimited bytes
Max processes 15576 15576 processes
Max open files 1024 4096 files
Max locked memory 65536 65536 bytes
Max address space unlimited unlimited bytes
Max file locks unlimited unlimited locks
Max pending signals 15576 15576 signals
Max msgqueue size 819200 819200 bytes
Max nice priority 0 0
Max realtime priority 0 0
Max realtime timeout unlimited unlimited us
swift@server:~$
从以上两种启动nginx的方式可以看到通过systemctl启动的nginx对本地配置的ulimit的最大文件打开数设置不生效。
3.原因分析
ulimit配置对service服务不生效的原因,是因为服务不是从shell启动的,而是systemd启动的,它有自己独立的资源限制配置。
核心原因就是:
ulimit只影响当前shell及其子进程,systemd service不属于这个进程树。
3.1 ulimit的作用范围:
ulimit -n 65535 只会影响
当前shell
└── 该 shell 启动的进程比如:
bash
├─ nginx
├─ python
└─ java这些程序会继承:
Max Open files = 65535如果nginx是通过systemctl start nginx启动的,进程关系是
systemd
└─ nginx而不是
bash
└─ nginx所以,不会继承ulimit配置
3.2 systemd自身的限制
systemd service 使用LimitNOFILE控制最大文件描述符。
1.通过直接修改服务的service配置文件进行调整
例如:调整nginx这个service服务的最大文件打开数限制,可以在其nginx.service文件中添加
LimitNOFILE=65535修改service配置文件后,需要运行systemctl daemon-reload重新加载配置后,再进行重启服务操作systemctl restart nginx
2.通过修改全局配置文件进行调整
/etc/systemd/system.conf
/etc/systemd/user.conf
system.conf是系统实例使用的,user.conf是用户实例使用的。
可以通过修改/etc/systemd/system.conf配置文件调整最大文件描述符限制
# 修改进程的Max open files限制
DefaultLimitNOFILE=65535注意:service自身的配置文件中的LimitNOFILE会覆盖全局配置中的DefaultLimitNOFILE
4.修改nginx的service限制方式
1.通过修改全局配置文件来实现对systemd service的限制
修改全局配置文件
sudo vim /etc/systemd/system.conf
# 修改进程的Max open files限制
DefaultLimitNOFILE=65535加载配置,重启服务,查看limit限制,已经生效,Max open files 限制是 65535
# 重新加载配置文件
swift@server:~$ sudo systemctl daemon-reload
swift@server:~$ sudo systemctl restart nginx.service
swift@server:~$ ps -ef |grep nginx
root 4228 1 0 18:34 ? 00:00:00 nginx: master process /usr/sbin/nginx -g daemon on; master_process on;
www-data 4229 4228 0 18:34 ? 00:00:00 nginx: worker process
www-data 4230 4228 0 18:34 ? 00:00:00 nginx: worker process
swift 4233 1614 0 18:34 pts/0 00:00:00 grep --color=auto nginx
swift@server:~$ cat /proc/4228/limits
Limit Soft Limit Hard Limit Units
Max cpu time unlimited unlimited seconds
Max file size unlimited unlimited bytes
Max data size unlimited unlimited bytes
Max stack size 8388608 unlimited bytes
Max core file size 0 unlimited bytes
Max resident set unlimited unlimited bytes
Max processes 15576 15576 processes
Max open files 65535 65535 files
Max locked memory 65536 65536 bytes
Max address space unlimited unlimited bytes
Max file locks unlimited unlimited locks
Max pending signals 15576 15576 signals
Max msgqueue size 819200 819200 bytes
Max nice priority 0 0
Max realtime priority 0 0
Max realtime timeout unlimited unlimited us不过有个问题,修改全局配置文件,会对其它service服务也有影响,如:查看sshd服务的limit限制
swift@server:~$ ps -ef |grep sshd
root 1231 1 0 17:29 ? 00:00:00 /usr/sbin/sshd -D
root 1536 1231 0 17:29 ? 00:00:00 sshd: swift [priv]
swift 1613 1536 0 17:29 ? 00:00:10 sshd: swift@pts/0
swift@server:~$ cat /proc/1231/limits
Limit Soft Limit Hard Limit Units
Max cpu time unlimited unlimited seconds
Max file size unlimited unlimited bytes
Max data size unlimited unlimited bytes
Max stack size 8388608 unlimited bytes
Max core file size 0 unlimited bytes
Max resident set unlimited unlimited bytes
Max processes 15576 15576 processes
Max open files 1024 4096 files
Max locked memory 65536 65536 bytes
Max address space unlimited unlimited bytes
Max file locks unlimited unlimited locks
Max pending signals 15576 15576 signals
Max msgqueue size 819200 819200 bytes
Max nice priority 0 0
Max realtime priority 0 0
Max realtime timeout unlimited unlimited us重启sshd服务,这是在修改过全局配置文件后的重启,可以发现Max open files 限制也变成了 65535
swift@server:~$ sudo systemctl restart sshd.service
swift@server:~$ ps -ef |grep sshd
root 1536 1 0 17:29 ? 00:00:00 sshd: swift [priv]
swift 1613 1536 0 17:29 ? 00:00:11 sshd: swift@pts/0
root 4274 1 0 18:37 ? 00:00:00 /usr/sbin/sshd -D
swift 4277 1614 0 18:38 pts/0 00:00:00 grep --color=auto sshd
swift@server:~$
swift@server:~$ cat /proc/1536/limits
Limit Soft Limit Hard Limit Units
Max cpu time unlimited unlimited seconds
Max file size unlimited unlimited bytes
Max data size unlimited unlimited bytes
Max stack size 8388608 unlimited bytes
Max core file size 0 unlimited bytes
Max resident set unlimited unlimited bytes
Max processes 15576 15576 processes
Max open files 65535 65535 files
Max locked memory 65536 65536 bytes
Max address space unlimited unlimited bytes
Max file locks unlimited unlimited locks
Max pending signals 15576 15576 signals
Max msgqueue size 819200 819200 bytes
Max nice priority 0 0
Max realtime priority 0 0
Max realtime timeout unlimited unlimited us
swift@server:~$2.只修改对应的service配置文件,在其中添加限制(LimitNOFILE=65535)
swift@server:~$ systemctl status nginx.service
● nginx.service - A high performance web server and a reverse proxy server
Loaded: loaded (/lib/systemd/system/nginx.service; enabled; vendor preset: enabled)
Active: active (running) since Wed 2022-08-10 18:34:31 CST; 4min 53s ago
Process: 4218 ExecStop=/sbin/start-stop-daemon --quiet --stop --retry QUIT/5 --pidfile /run/nginx.pid (code=exited, status=0/SUCCESS)
Process: 4226 ExecStart=/usr/sbin/nginx -g daemon on; master_process on; (code=exited, status=0/SUCCESS)
Process: 4223 ExecStartPre=/usr/sbin/nginx -t -q -g daemon on; master_process on; (code=exited, status=0/SUCCESS)
Main PID: 4228 (nginx)
Tasks: 3
Memory: 10.6M
CPU: 10ms
CGroup: /system.slice/nginx.service
├─4228 nginx: master process /usr/sbin/nginx -g daemon on; master_process on
├─4229 nginx: worker process
└─4230 nginx: worker process
swift@server:~$ sudo vim /lib/systemd/system/nginx.service
# Stop dance for nginx
# =======================
#
# ExecStop sends SIGSTOP (graceful stop) to the nginx process.
# If, after 5s (--retry QUIT/5) nginx is still running, systemd takes control
# and sends SIGTERM (fast shutdown) to the main process.
# After another 5s (TimeoutStopSec=5), and if nginx is alive, systemd sends
# SIGKILL to all the remaining processes in the process group (KillMode=mixed).
#
# nginx signals reference doc:
# http://nginx.org/en/docs/control.html
#
[Unit]
Description=A high performance web server and a reverse proxy server
After=network.target
[Service]
Type=forking
PIDFile=/run/nginx.pid
ExecStartPre=/usr/sbin/nginx -t -q -g 'daemon on; master_process on;'
ExecStart=/usr/sbin/nginx -g 'daemon on; master_process on;'
ExecReload=/usr/sbin/nginx -g 'daemon on; master_process on;' -s reload
ExecStop=-/sbin/start-stop-daemon --quiet --stop --retry QUIT/5 --pidfile /run/nginx.pid
TimeoutStopSec=5
KillMode=mixed
LimitNOFILE=65535
[Install]
WantedBy=multi-user.target
加载配置,重启服务
swift@server:~$ sudo vim /lib/systemd/system/nginx.service
swift@server:~$ sudo systemctl daemon-reload
swift@server:~$ sudo systemctl restart nginx.service
swift@server:~$ ps -ef |grep nginx
root 4343 1 0 18:41 ? 00:00:00 nginx: master process /usr/sbin/nginx -g daemon on; master_process on;
www-data 4344 4343 0 18:41 ? 00:00:00 nginx: worker process
www-data 4345 4343 0 18:41 ? 00:00:00 nginx: worker process
swift 4348 1614 0 18:41 pts/0 00:00:00 grep --color=auto nginx
swift@server:~$ cat /proc/4343/limits
Limit Soft Limit Hard Limit Units
Max cpu time unlimited unlimited seconds
Max file size unlimited unlimited bytes
Max data size unlimited unlimited bytes
Max stack size 8388608 unlimited bytes
Max core file size 0 unlimited bytes
Max resident set unlimited unlimited bytes
Max processes 15576 15576 processes
Max open files 65535 65535 files
Max locked memory 65536 65536 bytes
Max address space unlimited unlimited bytes
Max file locks unlimited unlimited locks
Max pending signals 15576 15576 signals
Max msgqueue size 819200 819200 bytes
Max nice priority 0 0
Max realtime priority 0 0
Max realtime timeout unlimited unlimited us
swift@server:~$注意:
nginx自身的service中的LimitNOFILE限制,会覆盖全局配置中的DefaultLimitNOFILE
如果在/lib/systemd/system/nginx.service文件添加的限制是LimitNOFILE=60000,全局配置文件/etc/systemd/system.conf中修改的默认进程的Max open files限制是DefaultLimitNOFILE=65535,那么最终进程的Max open files 的限制为在其service配置文件中添加的(LimitNOFILE=60000)