前情提要:

最近几天使用trojan-go总是遇到不能上网的情况,经过一系列的排查发现并不是域名或服务器被墙了。然后我开始查trojan的日志,发现有很多read: connection timed out的报错。初步猜测应该是连接上限了,于是我调整了内核参数fs.file-max=1000000。本以为就不会出现这种情况了,但事与愿违,一天之后又出现同样的问题!!!我有点懵了,我开始怀疑trojan没有继承刚才的open files配置,于是又开始了新一轮的排查:

  1. 先找出trojan的PID
root@ip-172-26-12-41:~# ps -ef|grep trojan
root 2080 1 0 May26 ? 00:00:33 /usr/bin/trojan/trojan -config /usr/local/etc/trojan/config.json
root 35254 1 0 03:00 ? 00:00:03 /usr/local/bin/trojan web -p 1080
root 40802 39795 0 11:05 pts/0 00:00:00 grep --color=auto trojan
  1. 然后查询2080的Soft Limit
root@ip-172-26-12-41:~# cat /proc/2080/limits 
Limit Soft Limit Hard Limit Units
Max cpu time unlimited unlimited seconds
Max file size unlimited unlimited bytes
Max data size unlimited unlimited bytes
Max stack size 8388608 unlimited bytes
Max core file size 0 unlimited bytes
Max resident set unlimited unlimited bytes
Max processes 3882 3882 processes
Max open files 1024 524288 files
Max locked memory 65536 65536 bytes
Max address space unlimited unlimited bytes
Max file locks unlimited unlimited locks
Max pending signals 3882 3882 signals
Max msgqueue size 819200 819200 bytes
Max nice priority 0 0
Max realtime priority 0 0
Max realtime timeout unlimited unlimited us

发现Max open files的Soft Limit依旧是1024,并且Hard Limit是524288 这并不是我刚才设置的1000000。

经过在对trojan服务的定位,发现该服务是以系统服务的方式运行,并没有去遵循用户服务的limits配置。

解决方案

  1. 针对用户服务的limits配置
root@ip-172-26-12-41:~# sysctl -w fs.file-max=1000001
fs.file-max = 1000001
root@ip-172-26-12-41:~# sysctl -p

或者直接编辑 /etc/sysctl.conf 文件追加 fs.file-max = 1000000 然后保存重启即可。

  1. 针对系统服务
  • 全局设置
root@ip-172-26-12-41:~# vi /etc/systemd/system.conf 

找到 DefaultLimitNOFILE 然后修改为 1000000 或者是 102400:1000000 ,前者软硬均为1000000、后者软硬分开设置。【<Soft Limit>:<Hard Limit>】

  • 单服务的限制
root@ip-172-26-12-41:~# vi /etc/systemd/system/trojan.service
[Unit]
Description=trojan-go
After=network.target network-online.target nss-lookup.target mysql.service mariadb.service mysqld.service

[Service]
Type=simple
LimitNOFILE=65535 // 这里加入LimitNOFILE的配置即可
StandardError=journal
ExecStart=/usr/bin/trojan/trojan -config /usr/local/etc/trojan/config.json
ExecReload=/bin/kill -HUP $MAINPID
Restart=on-failure
RestartSec=3s

[Install]
WantedBy=multi-user.target

// 保存文件后需要重载服务,然后再重启服务即可生效
root@ip-172-26-12-41:~# systemctl daemon-reload
root@ip-172-26-12-41:~# systemctl restart trojan.service
root@ip-172-26-12-41:~# ps -ef|grep trojan
root 35254 1 0 03:00 ? 00:00:03 /usr/local/bin/trojan web -p 1080
root 41521 1 0 11:41 ? 00:00:00 /usr/bin/trojan/trojan -config /usr/local/etc/trojan/config.json
root 41544 39795 0 11:41 pts/0 00:00:00 grep --color=auto trojan
root@ip-172-26-12-41:~# cat /proc/41521/limits
Limit Soft Limit Hard Limit Units
Max cpu time unlimited unlimited seconds
Max file size unlimited unlimited bytes
Max data size unlimited unlimited bytes
Max stack size 8388608 unlimited bytes
Max core file size 0 unlimited bytes
Max resident set unlimited unlimited bytes
Max processes 3882 3882 processes
Max open files 65535 65535 files
Max locked memory 65536 65536 bytes
Max address space unlimited unlimited bytes
Max file locks unlimited unlimited locks
Max pending signals 3882 3882 signals
Max msgqueue size 819200 819200 bytes
Max nice priority 0 0
Max realtime priority 0 0
Max realtime timeout unlimited unlimited us