本地通过ulimit已经配置了最大文件打开数,但是发现对systemd管理的nginx服务不生效。通过调查分析发现,使用命令行启动nginx使用systemctl启动nginxlimit限制是不同的。

以下是分析过程:

1.查看ulimit配置

查看已配置的ulimit限制,最大文件打开数配置的是 65535

swift@server:~$ ulimit -a
core file size          (blocks, -c) 0
data seg size           (kbytes, -d) unlimited
scheduling priority             (-e) 0
file size               (blocks, -f) unlimited
pending signals                 (-i) 15576
max locked memory       (kbytes, -l) 64
max memory size         (kbytes, -m) unlimited
open files                      (-n) 65535
pipe size            (512 bytes, -p) 8
POSIX message queues     (bytes, -q) 819200
real-time priority              (-r) 0
stack size              (kbytes, -s) 8192
cpu time               (seconds, -t) unlimited
max user processes              (-u) 15576
virtual memory          (kbytes, -v) unlimited
file locks                      (-x) unlimited
swift@server:~$

2.对比命令行启动和systemd启动服务后的limit限制

分别以两种方式启动nginx服务,查看进程号PIDlimit值限制(Max open files

2.1 命令行启动nginx

可以发现Max open files 限制是 65535,和本地ulimti -n查看到的配置65535一致

swift@server:~$ sudo nginx
swift@server:~$ ps -ef |grep nginx
root       3946      1  0 18:17 ?        00:00:00 nginx: master process nginx
www-data   3947   3946  0 18:17 ?        00:00:00 nginx: worker process
www-data   3948   3946  0 18:17 ?        00:00:00 nginx: worker process
swift      3950   1614  0 18:17 pts/0    00:00:00 grep --color=auto nginx
swift@server:~$ cat /proc/3946/limits
Limit                     Soft Limit           Hard Limit           Units
Max cpu time              unlimited            unlimited            seconds
Max file size             unlimited            unlimited            bytes
Max data size             unlimited            unlimited            bytes
Max stack size            8388608              unlimited            bytes
Max core file size        0                    unlimited            bytes
Max resident set          unlimited            unlimited            bytes
Max processes             15576                15576                processes
Max open files            65535                65535                files
Max locked memory         65536                65536                bytes
Max address space         unlimited            unlimited            bytes
Max file locks            unlimited            unlimited            locks
Max pending signals       15576                15576                signals
Max msgqueue size         819200               819200               bytes
Max nice priority         0                    0
Max realtime priority     0                    0
Max realtime timeout      unlimited            unlimited            us

2.2 使用systemctl start nginx.server启动nginx

可以发现Max open files1024,而不是本地ulimti -n查看到的配置65535

swift@server:~$ sudo systemctl start nginx.service
swift@server:~$ ps -ef |grep nginx
root       4001      1  0 18:25 ?        00:00:00 nginx: master process /usr/sbin/nginx -g daemon on; master_process on;
www-data   4003   4001  0 18:25 ?        00:00:00 nginx: worker process
www-data   4004   4001  0 18:25 ?        00:00:00 nginx: worker process
swift      4008   1614  0 18:25 pts/0    00:00:00 grep --color=auto nginx
swift@server:~$ cat /proc/4001/limits
Limit                     Soft Limit           Hard Limit           Units
Max cpu time              unlimited            unlimited            seconds
Max file size             unlimited            unlimited            bytes
Max data size             unlimited            unlimited            bytes
Max stack size            8388608              unlimited            bytes
Max core file size        0                    unlimited            bytes
Max resident set          unlimited            unlimited            bytes
Max processes             15576                15576                processes
Max open files            1024                 4096                 files
Max locked memory         65536                65536                bytes
Max address space         unlimited            unlimited            bytes
Max file locks            unlimited            unlimited            locks
Max pending signals       15576                15576                signals
Max msgqueue size         819200               819200               bytes
Max nice priority         0                    0
Max realtime priority     0                    0
Max realtime timeout      unlimited            unlimited            us
swift@server:~$

从以上两种启动nginx的方式可以看到通过systemctl启动的nginx对本地配置的ulimit的最大文件打开数设置不生效。

3.原因分析

ulimit配置对service服务不生效的原因,是因为服务不是从shell启动的,而是systemd启动的,它有自己独立的资源限制配置。

核心原因就是:

ulimit只影响当前shell及其子进程,systemd service不属于这个进程树。

3.1 ulimit的作用范围:

ulimit -n 65535 只会影响

当前shell
└── 该 shell 启动的进程

比如:

bash
 ├─ nginx
 ├─ python
 └─ java

这些程序会继承:

Max Open files = 65535

如果nginx是通过systemctl start nginx启动的,进程关系是

systemd
 └─ nginx

而不是

bash
 └─ nginx

所以,不会继承ulimit配置

3.2 systemd自身的限制

systemd service 使用LimitNOFILE控制最大文件描述符。

1.通过直接修改服务的service配置文件进行调整

例如:调整nginx这个service服务的最大文件打开数限制,可以在其nginx.service文件中添加

LimitNOFILE=65535

修改service配置文件后,需要运行systemctl daemon-reload重新加载配置后,再进行重启服务操作systemctl restart nginx

2.通过修改全局配置文件进行调整

/etc/systemd/system.conf

/etc/systemd/user.conf

system.conf是系统实例使用的,user.conf是用户实例使用的。

可以通过修改/etc/systemd/system.conf配置文件调整最大文件描述符限制

# 修改进程的Max open files限制
DefaultLimitNOFILE=65535

注意:service自身的配置文件中的LimitNOFILE会覆盖全局配置中的DefaultLimitNOFILE

4.修改nginxservice限制方式

1.通过修改全局配置文件来实现对systemd service的限制

修改全局配置文件

sudo vim /etc/systemd/system.conf

# 修改进程的Max open files限制
DefaultLimitNOFILE=65535

加载配置,重启服务,查看limit限制,已经生效,Max open files 限制是 65535

# 重新加载配置文件
swift@server:~$ sudo systemctl daemon-reload
swift@server:~$ sudo systemctl restart nginx.service
swift@server:~$ ps -ef |grep nginx
root       4228      1  0 18:34 ?        00:00:00 nginx: master process /usr/sbin/nginx -g daemon on; master_process on;
www-data   4229   4228  0 18:34 ?        00:00:00 nginx: worker process
www-data   4230   4228  0 18:34 ?        00:00:00 nginx: worker process
swift      4233   1614  0 18:34 pts/0    00:00:00 grep --color=auto nginx
swift@server:~$ cat /proc/4228/limits
Limit                     Soft Limit           Hard Limit           Units
Max cpu time              unlimited            unlimited            seconds
Max file size             unlimited            unlimited            bytes
Max data size             unlimited            unlimited            bytes
Max stack size            8388608              unlimited            bytes
Max core file size        0                    unlimited            bytes
Max resident set          unlimited            unlimited            bytes
Max processes             15576                15576                processes
Max open files            65535                65535                files
Max locked memory         65536                65536                bytes
Max address space         unlimited            unlimited            bytes
Max file locks            unlimited            unlimited            locks
Max pending signals       15576                15576                signals
Max msgqueue size         819200               819200               bytes
Max nice priority         0                    0
Max realtime priority     0                    0
Max realtime timeout      unlimited            unlimited            us

不过有个问题,修改全局配置文件,会对其它service服务也有影响,如:查看sshd服务的limit限制

swift@server:~$ ps -ef |grep sshd
root       1231      1  0 17:29 ?        00:00:00 /usr/sbin/sshd -D
root       1536   1231  0 17:29 ?        00:00:00 sshd: swift [priv]
swift      1613   1536  0 17:29 ?        00:00:10 sshd: swift@pts/0
swift@server:~$ cat /proc/1231/limits
Limit                     Soft Limit           Hard Limit           Units
Max cpu time              unlimited            unlimited            seconds
Max file size             unlimited            unlimited            bytes
Max data size             unlimited            unlimited            bytes
Max stack size            8388608              unlimited            bytes
Max core file size        0                    unlimited            bytes
Max resident set          unlimited            unlimited            bytes
Max processes             15576                15576                processes
Max open files            1024                 4096                 files
Max locked memory         65536                65536                bytes
Max address space         unlimited            unlimited            bytes
Max file locks            unlimited            unlimited            locks
Max pending signals       15576                15576                signals
Max msgqueue size         819200               819200               bytes
Max nice priority         0                    0
Max realtime priority     0                    0
Max realtime timeout      unlimited            unlimited            us

重启sshd服务,这是在修改过全局配置文件后的重启,可以发现Max open files 限制也变成了 65535

swift@server:~$ sudo systemctl restart sshd.service
swift@server:~$ ps -ef |grep sshd
root       1536      1  0 17:29 ?        00:00:00 sshd: swift [priv]
swift      1613   1536  0 17:29 ?        00:00:11 sshd: swift@pts/0
root       4274      1  0 18:37 ?        00:00:00 /usr/sbin/sshd -D
swift      4277   1614  0 18:38 pts/0    00:00:00 grep --color=auto sshd
swift@server:~$
swift@server:~$ cat /proc/1536/limits
Limit                     Soft Limit           Hard Limit           Units
Max cpu time              unlimited            unlimited            seconds
Max file size             unlimited            unlimited            bytes
Max data size             unlimited            unlimited            bytes
Max stack size            8388608              unlimited            bytes
Max core file size        0                    unlimited            bytes
Max resident set          unlimited            unlimited            bytes
Max processes             15576                15576                processes
Max open files            65535                65535                files
Max locked memory         65536                65536                bytes
Max address space         unlimited            unlimited            bytes
Max file locks            unlimited            unlimited            locks
Max pending signals       15576                15576                signals
Max msgqueue size         819200               819200               bytes
Max nice priority         0                    0
Max realtime priority     0                    0
Max realtime timeout      unlimited            unlimited            us
swift@server:~$

2.只修改对应的service配置文件,在其中添加限制(LimitNOFILE=65535)

swift@server:~$ systemctl status nginx.service
● nginx.service - A high performance web server and a reverse proxy server
   Loaded: loaded (/lib/systemd/system/nginx.service; enabled; vendor preset: enabled)
   Active: active (running) since Wed 2022-08-10 18:34:31 CST; 4min 53s ago
  Process: 4218 ExecStop=/sbin/start-stop-daemon --quiet --stop --retry QUIT/5 --pidfile /run/nginx.pid (code=exited, status=0/SUCCESS)
  Process: 4226 ExecStart=/usr/sbin/nginx -g daemon on; master_process on; (code=exited, status=0/SUCCESS)
  Process: 4223 ExecStartPre=/usr/sbin/nginx -t -q -g daemon on; master_process on; (code=exited, status=0/SUCCESS)
 Main PID: 4228 (nginx)
    Tasks: 3
   Memory: 10.6M
      CPU: 10ms
   CGroup: /system.slice/nginx.service
           ├─4228 nginx: master process /usr/sbin/nginx -g daemon on; master_process on
           ├─4229 nginx: worker process
           └─4230 nginx: worker process

swift@server:~$ sudo vim /lib/systemd/system/nginx.service
# Stop dance for nginx
# =======================
#
# ExecStop sends SIGSTOP (graceful stop) to the nginx process.
# If, after 5s (--retry QUIT/5) nginx is still running, systemd takes control
# and sends SIGTERM (fast shutdown) to the main process.
# After another 5s (TimeoutStopSec=5), and if nginx is alive, systemd sends
# SIGKILL to all the remaining processes in the process group (KillMode=mixed).
#
# nginx signals reference doc:
# http://nginx.org/en/docs/control.html
#
[Unit]
Description=A high performance web server and a reverse proxy server
After=network.target

[Service]
Type=forking
PIDFile=/run/nginx.pid
ExecStartPre=/usr/sbin/nginx -t -q -g 'daemon on; master_process on;'
ExecStart=/usr/sbin/nginx -g 'daemon on; master_process on;'
ExecReload=/usr/sbin/nginx -g 'daemon on; master_process on;' -s reload
ExecStop=-/sbin/start-stop-daemon --quiet --stop --retry QUIT/5 --pidfile /run/nginx.pid
TimeoutStopSec=5
KillMode=mixed
LimitNOFILE=65535

[Install]
WantedBy=multi-user.target

加载配置,重启服务

swift@server:~$ sudo vim /lib/systemd/system/nginx.service
swift@server:~$ sudo systemctl daemon-reload
swift@server:~$ sudo systemctl restart nginx.service
swift@server:~$ ps -ef |grep nginx
root       4343      1  0 18:41 ?        00:00:00 nginx: master process /usr/sbin/nginx -g daemon on; master_process on;
www-data   4344   4343  0 18:41 ?        00:00:00 nginx: worker process
www-data   4345   4343  0 18:41 ?        00:00:00 nginx: worker process
swift      4348   1614  0 18:41 pts/0    00:00:00 grep --color=auto nginx
swift@server:~$ cat /proc/4343/limits
Limit                     Soft Limit           Hard Limit           Units
Max cpu time              unlimited            unlimited            seconds
Max file size             unlimited            unlimited            bytes
Max data size             unlimited            unlimited            bytes
Max stack size            8388608              unlimited            bytes
Max core file size        0                    unlimited            bytes
Max resident set          unlimited            unlimited            bytes
Max processes             15576                15576                processes
Max open files            65535                65535                files
Max locked memory         65536                65536                bytes
Max address space         unlimited            unlimited            bytes
Max file locks            unlimited            unlimited            locks
Max pending signals       15576                15576                signals
Max msgqueue size         819200               819200               bytes
Max nice priority         0                    0
Max realtime priority     0                    0
Max realtime timeout      unlimited            unlimited            us
swift@server:~$

注意:

nginx自身的service中的LimitNOFILE限制,会覆盖全局配置中的DefaultLimitNOFILE

如果在/lib/systemd/system/nginx.service文件添加的限制是LimitNOFILE=60000,全局配置文件/etc/systemd/system.conf中修改的默认进程的Max open files限制是DefaultLimitNOFILE=65535,那么最终进程的Max open files 的限制为在其service配置文件中添加的(LimitNOFILE=60000

标签: Linux, 问题记录

添加新评论