前情提要: 最近几天使用trojan-go总是遇到不能上网的情况,经过一系列的排查发现并不是域名或服务器被墙了。然后我开始查trojan的日志,发现有很多read: connection timed out
的报错。初步猜测应该是连接上限了,于是我调整了内核参数fs.file-max=1000000
。本以为就不会出现这种情况了,但事与愿违,一天之后又出现同样的问题!!!我有点懵了,我开始怀疑trojan没有继承刚才的open files配置,于是又开始了新一轮的排查:
先找出trojan的PID
root@ip-172-26-12-41:~# ps -ef|grep trojan root 2080 1 0 May26 ? 00:00:33 /usr/bin/trojan/trojan -config /usr/local/etc/trojan/config.json root 35254 1 0 03:00 ? 00:00:03 /usr/local/bin/trojan web -p 1080 root 40802 39795 0 11:05 pts/0 00:00:00 grep --color=auto trojan
然后查询2080的Soft Limit
root@ip-172-26-12-41:~# cat /proc/2080/limits Limit Soft Limit Hard Limit Units Max cpu time unlimited unlimited seconds Max file size unlimited unlimited bytes Max data size unlimited unlimited bytes Max stack size 8388608 unlimited bytes Max core file size 0 unlimited bytes Max resident set unlimited unlimited bytes Max processes 3882 3882 processes Max open files 1024 524288 files Max locked memory 65536 65536 bytes Max address space unlimited unlimited bytes Max file locks unlimited unlimited locks Max pending signals 3882 3882 signals Max msgqueue size 819200 819200 bytes Max nice priority 0 0 Max realtime priority 0 0 Max realtime timeout unlimited unlimited us
发现Max open files的Soft Limit依旧是1024,并且Hard Limit是524288 这并不是我刚才设置的1000000。
经过在对trojan服务的定位,发现该服务是以系统服务的方式运行,并没有去遵循用户服务的limits配置。
解决方案
针对用户服务的limits配置
root@ip-172-26-12-41:~# sysctl -w fs.file-max=1000001 fs.file-max = 1000001 root@ip-172-26-12-41:~# sysctl -p
或者直接编辑 /etc/sysctl.conf
文件追加 fs.file-max = 1000000
然后保存重启即可。
针对系统服务
root@ip-172-26-12-41:~# vi /etc/systemd/system.conf
找到 DefaultLimitNOFILE
然后修改为 1000000 或者是 102400:1000000 ,前者软硬均为1000000、后者软硬分开设置。【<Soft Limit>:<Hard Limit>】
root@ip-172-26-12-41:~# vi /etc/systemd/system/trojan.service [Unit] Description=trojan-go After=network.target network-online.target nss-lookup.target mysql.service mariadb.service mysqld.service [Service] Type=simple LimitNOFILE=65535 // 这里加入LimitNOFILE的配置即可 StandardError=journal ExecStart=/usr/bin/trojan/trojan -config /usr/local/etc/trojan/config.json ExecReload=/bin/kill -HUP $MAINPID Restart=on-failure RestartSec=3s [Install] WantedBy=multi-user.target // 保存文件后需要重载服务,然后再重启服务即可生效 root@ip-172-26-12-41:~# systemctl daemon-reload root@ip-172-26-12-41:~# systemctl restart trojan.service root@ip-172-26-12-41:~# ps -ef|grep trojan root 35254 1 0 03:00 ? 00:00:03 /usr/local/bin/trojan web -p 1080 root 41521 1 0 11:41 ? 00:00:00 /usr/bin/trojan/trojan -config /usr/local/etc/trojan/config.json root 41544 39795 0 11:41 pts/0 00:00:00 grep --color=auto trojan root@ip-172-26-12-41:~# cat /proc/41521/limits Limit Soft Limit Hard Limit Units Max cpu time unlimited unlimited seconds Max file size unlimited unlimited bytes Max data size unlimited unlimited bytes Max stack size 8388608 unlimited bytes Max core file size 0 unlimited bytes Max resident set unlimited unlimited bytes Max processes 3882 3882 processes Max open files 65535 65535 files Max locked memory 65536 65536 bytes Max address space unlimited unlimited bytes Max file locks unlimited unlimited locks Max pending signals 3882 3882 signals Max msgqueue size 819200 819200 bytes Max nice priority 0 0 Max realtime priority 0 0 Max realtime timeout unlimited unlimited us