114 lines
4.0 KiB
Markdown
Executable File
114 lines
4.0 KiB
Markdown
Executable File
[ 来源:imroc](https://github.com/imroc/kubernetes-practice-guide/ "内容来源") 浏览 _5239_ [扫码](#) _2019-11-16 08:54:09_
|
||
|
||
每个 linux 进程可以持有多个 fd，每个 inotify 类型的 fd 可以 watch 多个目录，每个用户下所有进程 inotify 类型的 fd 可以 watch 的总目录数有个最大限制，这个限制可以通过内核参数配置: `fs.inotify.max_user_watches`
|
||
|
||
查看最大 inotify watch 数:
|
||
|
||
```
|
||
|
||
|
||
1. `$ cat /proc/sys/fs/inotify/max_user_watches`
|
||
2. `8192`
|
||
|
||
|
||
```
|
||
|
||
使用下面的脚本查看当前有 inotify watch 类型 fd 的进程以及每个 fd watch 的目录数量，降序输出，带总数统计:
|
||
|
||
```
|
||
|
||
|
||
1. `#!/usr/bin/env bash`
|
||
|
||
6. `total=0`
|
||
7. `result="EXE PID FD-INFO INOTIFY-WATCHES\n"`
|
||
8. `while read pid fd; do \`
|
||
9. `exe="$(readlink -f /proc/$pid/exe || echo n/a)"; \`
|
||
10. `fdinfo="/proc/$pid/fdinfo/$fd" ; \`
|
||
11. `count="$(grep -c inotify "$fdinfo" || true)"; \`
|
||
12. `if [ $((count)) != 0 ]; then`
|
||
13. `total=$((total+count)); \`
|
||
14. `result+="$exe $pid $fdinfo $count\n"; \`
|
||
15. `fi`
|
||
16. `done <<< "$(lsof +c 0 -n -P -u root|awk '/inotify$/ { gsub(/[urw]$/,"",$4); print $2" "$4 }')" && echo "total $total inotify watches" && result="$(echo -e $result|column -t)\n" && echo -e "$result" | head -1 && echo -e "$result" | sed "1d" | sort -k 4rn;`
|
||
|
||
|
||
```
|
||
|
||
示例输出:
|
||
|
||
```
|
||
|
||
|
||
1. `total 7882 inotify watches`
|
||
2. `EXE PID FD-INFO INOTIFY-WATCHES`
|
||
3. `/usr/local/qcloud/YunJing/YDEyes/YDService 25813 /proc/25813/fdinfo/8 7077`
|
||
4. `/usr/bin/kubelet 1173 /proc/1173/fdinfo/22 665`
|
||
5. `/usr/bin/ruby2.3 13381 /proc/13381/fdinfo/14 54`
|
||
6. `/usr/lib/policykit-1/polkitd 1458 /proc/1458/fdinfo/9 14`
|
||
7. `/lib/systemd/systemd-udevd 450 /proc/450/fdinfo/9 13`
|
||
8. `/usr/sbin/nscd 7935 /proc/7935/fdinfo/3 6`
|
||
9. `/usr/bin/kubelet 1173 /proc/1173/fdinfo/28 5`
|
||
10. `/lib/systemd/systemd 1 /proc/1/fdinfo/17 4`
|
||
11. `/lib/systemd/systemd 1 /proc/1/fdinfo/18 4`
|
||
12. `/lib/systemd/systemd 1 /proc/1/fdinfo/26 4`
|
||
13. `/lib/systemd/systemd 1 /proc/1/fdinfo/28 4`
|
||
14. `/usr/lib/policykit-1/polkitd 1458 /proc/1458/fdinfo/8 4`
|
||
15. `/usr/local/bin/sidecar-injector 4751 /proc/4751/fdinfo/3 3`
|
||
16. `/usr/lib/accountsservice/accounts-daemon 1178 /proc/1178/fdinfo/7 2`
|
||
17. `/usr/local/bin/galley 8228 /proc/8228/fdinfo/10 2`
|
||
18. `/usr/local/bin/galley 8228 /proc/8228/fdinfo/9 2`
|
||
19. `/lib/systemd/systemd 1 /proc/1/fdinfo/11 1`
|
||
20. `/sbin/agetty 1437 /proc/1437/fdinfo/4 1`
|
||
21. `/sbin/agetty 1440 /proc/1440/fdinfo/4 1`
|
||
22. `/usr/bin/kubelet 1173 /proc/1173/fdinfo/10 1`
|
||
23. `/usr/local/bin/envoy 4859 /proc/4859/fdinfo/5 1`
|
||
24. `/usr/local/bin/envoy 5427 /proc/5427/fdinfo/5 1`
|
||
25. `/usr/local/bin/envoy 6058 /proc/6058/fdinfo/3 1`
|
||
26. `/usr/local/bin/envoy 6893 /proc/6893/fdinfo/3 1`
|
||
27. `/usr/local/bin/envoy 6950 /proc/6950/fdinfo/3 1`
|
||
28. `/usr/local/bin/galley 8228 /proc/8228/fdinfo/3 1`
|
||
29. `/usr/local/bin/pilot-agent 3819 /proc/3819/fdinfo/5 1`
|
||
30. `/usr/local/bin/pilot-agent 4244 /proc/4244/fdinfo/5 1`
|
||
31. `/usr/local/bin/pilot-agent 5901 /proc/5901/fdinfo/3 1`
|
||
32. `/usr/local/bin/pilot-agent 6789 /proc/6789/fdinfo/3 1`
|
||
33. `/usr/local/bin/pilot-agent 6808 /proc/6808/fdinfo/3 1`
|
||
34. `/usr/local/bin/pilot-discovery 6231 /proc/6231/fdinfo/3 1`
|
||
35. `/usr/local/bin/sidecar-injector 4751 /proc/4751/fdinfo/5 1`
|
||
36. `/usr/sbin/acpid 1166 /proc/1166/fdinfo/6 1`
|
||
37. `/usr/sbin/dnsmasq 7572 /proc/7572/fdinfo/8 1`
|
||
|
||
|
||
```
|
||
|
||
如果看到总 watch 数比较大，接近最大限制，可以修改内核参数调高下这个限制。
|
||
|
||
临时调整:
|
||
|
||
```
|
||
|
||
|
||
1. `sudo sysctl fs.inotify.max_user_watches=524288`
|
||
|
||
|
||
```
|
||
|
||
永久生效:
|
||
|
||
```
|
||
|
||
|
||
1. `echo "fs.inotify.max_user_watches=524288" >> /etc/sysctl.conf && sysctl -p`
|
||
|
||
|
||
```
|
||
|
||
打开 inotify\_add\_watch 跟踪，进一步 debug inotify watch 耗尽的原因:
|
||
|
||
```
|
||
|
||
|
||
1. `echo 1 >> /sys/kernel/debug/tracing/events/syscalls/sys_exit_inotify_add_watch/enable`
|
||
|
||
|
||
``` |