After encountering a rare instance where processing a FreeBSD commit caused FreshPorts to run low on resources, I want to investigate how limiting a jail might help this situation.
I searched for ‘jail limit’ – found sentencing guidelines.
I searched for ‘jail limit freebsd’ and found a FreeBSD Forums referring me to rctl(8), which I recognize from my $DAYJOB.
I’ve just rebooted my host to add this to /boot/loader.conf:
kern.racct.enable="1"
Then I started referring to my copy of FreeBSD Mastery: Jails (IT Mastery Book 15) (As an Amazon Associate I earn from qualifying purchases).
In this post
This post was written using:
- FreeBSD 13.1-RELEASE-p3
- On the host known as knew
- Using the jail named empty
The test script being used is:
[knew dan ~] % cat /jails/empty/configure-plist.sh #!/bin/sh cd /usr/ports/editors/ghostwriter echo starting make generate-plist echo done
That invokes the loop shown in the blog post also linked above.
First attempt – limit the number of processes
I tried:
% sudo rctl jail:empty:maxproc:deny=100
That did not limit the jail to 100 processes. I was watching top running in the jail and watched it pass by 160 processes before I control C’d the followin:
[knew dan ~] % sudo jexec empty /configure-plist.sh 18:25:06 starting make: "/usr/ports/Mk/bsd.port.mk" line 3406: warning: duplicate script for target "/var/ports/usr/ports/editors/ghostwriter/work-qt5" ignored make: "/usr/ports/Mk/bsd.licenses.mk" line 769: warning: using previous script for "/var/ports/usr/ports/editors/ghostwriter/work-qt5" defined here make: "/usr/ports/Mk/bsd.port.mk" line 4479: warning: duplicate script for target "/var/ports/usr/ports/editors/ghostwriter/work-qt5" ignored make: "/usr/ports/Mk/bsd.licenses.mk" line 769: warning: using previous script for "/var/ports/usr/ports/editors/ghostwriter/work-qt5" defined here make: "/usr/ports/Mk/bsd.port.mk" line 5193: warning: duplicate script for target "/var/ports/usr/ports/editors/ghostwriter/work-qt5" ignored ... make[208]: "/usr/ports/Mk/bsd.port.mk" line 5365: warning: duplicate script for target "/var/ports/usr/ports/editors/ghostwriter/work-qt5" ignored make[208]: "/usr/ports/Mk/bsd.licenses.mk" line 769: warning: using previous script for "/var/ports/usr/ports/editors/ghostwriter/work-qt5" defined here make[208]: Graph cycles through /var/ports/usr/ports/editors/ghostwriter/work-qt5 ^C*** Signal 2 *** Signal 2 *** Signal 2 *** Signal 2 *** Signal 2 ... *** Signal 2 *** Signal 2 *** Signal 2 *** Signal 2 [knew dan ~] %
Going back, I see there are no limits in place:
[knew dan ~] % sudo rctl [knew dan ~] % sudo rctl jail:empty:maxproc:deny=100 [knew dan ~] % sudo rctl [knew dan ~] %
Let’s try an example straight from the book:
[knew dan ~] % sudo rctl jail:empty:readiops:throttle=10 [knew dan ~] % sudo rctl [knew dan ~] %
WTF?
Reading the manual, I’m not using -a. Let’s try that:
[knew dan ~] % sudo rctl -a jail:empty:readiops:throttle=10 [knew dan ~] % sudo rctl jail:empty:readiops:throttle=10 [knew dan ~] %
Success! Let’s try that again.
This time with limits
This time I get:
[knew dan ~] % sudo jexec empty /configure-plist.sh starting make: "/usr/ports/Mk/bsd.port.mk" line 3406: warning: duplicate script for target "/var/ports/usr/ports/editors/ghostwriter/work-qt5" ignored make: "/usr/ports/Mk/bsd.licenses.mk" line 769: warning: using previous script for "/var/ports/usr/ports/editors/ghostwriter/work-qt5" defined here make: "/usr/ports/Mk/bsd.port.mk" line 4479: warning: duplicate script for target "/var/ports/usr/ports/editors/ghostwriter/work-qt5" ignored ... make[85]: "/usr/ports/Mk/bsd.licenses.mk" line 769: warning: using previous script for "/var/ports/usr/ports/editors/ghostwriter/work-qt5" defined here make[85]: Graph cycles through /var/ports/usr/ports/editors/ghostwriter/work-qt5 sh: Cannot fork: Resource temporarily unavailable make[86]: "/usr/ports/Mk/Uses/compiler.mk" line 150: warning: "if cc -std=c89 -c -x c /dev/null -o /dev/null 2>&1; then echo yes; fi; echo" returned non-zero status ... make[87]: Graph cycles through /var/ports/usr/ports/editors/ghostwriter/work-qt5 sh: Cannot fork: Resource temporarily unavailable make[88]: "/usr/ports/Mk/Uses/compiler.mk" line 150: warning: "if cc -std=c89 -c -x c /dev/null -o /dev/null 2>&1; then echo yes; fi; echo" returned non-zero status sh: Cannot fork: Resource temporarily unavailable make[88]: "/usr/ports/Mk/Uses/compiler.mk" line 150: warning: "if cc -std=c99 -c -x c /dev/null -o /dev/null 2>&1; then echo yes; fi; echo" returned non-zero status sh: Cannot fork: Resource temporarily unavailable make[88]: "/usr/ports/Mk/Uses/compiler.mk" line 150: warning: "if cc -std=c11 -c -x c /dev/null -o /dev/null 2>&1; then echo yes; fi; echo" returned non-zero status sh: Cannot fork: Resource temporarily unavailable make[88]: "/usr/ports/Mk/Uses/compiler.mk" line 150: warning: "if cc -std=gnu89 -c -x c /dev/null -o /dev/null 2>&1; then echo yes; fi; echo" returned non-zero status sh: Cannot fork: Resource temporarily unavailable make[88]: "/usr/ports/Mk/Uses/compiler.mk" line 150: warning: "if cc -std=gnu99 -c -x c /dev/null -o /dev/null 2>&1; then echo yes; fi; echo" returned non-zero status sh: Cannot fork: Resource temporarily unavailable make[88]: "/usr/ports/Mk/Uses/compiler.mk" line 150: warning: "if cc -std=gnu11 -c -x c /dev/null -o /dev/null 2>&1; then echo yes; fi; echo" returned non-zero status sh: Cannot fork: Resource temporarily unavailable make[88]: "/usr/ports/Mk/Uses/compiler.mk" line 150: warning: "if cc -std=c++98 -c -x c++ /dev/null -o /dev/null 2>&1; then echo yes; fi; echo" returned non-zero status ^C*** Signal 2 *** Signal 2
Not good enough. It will still loop.
This time with SIGKILL
I added a SIGKILL rule and changed the number of processes to 30. Let’s fail faster.
[knew dan ~] % sudo rctl -a jail:empty:maxproc:SIGKILL=30 [knew dan ~] % sudo rctl jail:empty:maxproc:sigkill=30 jail:empty:maxproc:deny=100 jail:empty:readiops:throttle=10 [knew dan ~] % sudo rctl -r jail:empty:maxproc:deny=100 [knew dan ~] % sudo rctl jail:empty:maxproc:sigkill=30 jail:empty:readiops:throttle=10 [knew dan ~] % sudo rctl -r jail:empty:readiops:throttle=10
I also cleaned up the rules so I had only the one I wanted to test.
[knew dan ~] % sudo jexec empty /configure-plist.sh starting make: "/usr/ports/Mk/bsd.port.mk" line 3406: warning: duplicate script for target "/var/ports/usr/ports/editors/ghostwriter/work-qt5" ignored make: "/usr/ports/Mk/bsd.licenses.mk" line 769: warning: using previous script for "/var/ports/usr/ports/editors/ghostwriter/work-qt5" defined here make: "/usr/ports/Mk/bsd.port.mk" line 4479: warning: duplicate script for target "/var/ports/usr/ports/editors/ghostwriter/work-qt5" ignored make: "/usr/ports/Mk/bsd.licenses.mk" line 769: warning: using previous script for "/var/ports/usr/ports/editors/ghostwriter/work-qt5" defined here ... make[17]: "/usr/ports/Mk/bsd.licenses.mk" line 769: warning: using previous script for "/var/ports/usr/ports/editors/ghostwriter/work-qt5" defined here make[17]: Graph cycles through /var/ports/usr/ports/editors/ghostwriter/work-qt5 Killed Killed Killed Killed Killed Killed Killed Killed Killed Killed Killed Killed Killed Killed Killed Killed Killed Killed Killed make[18]: "/usr/ports/Mk/bsd.port.mk" line 3406: warning: duplicate script for target "/var/ports/usr/ports/editors/ghostwriter/work-qt5" ignored make[18]: "/usr/ports/Mk/bsd.licenses.mk" line 769: warning: using previous script for "/var/ports/usr/ports/editors/ghostwriter/work-qt5" defined here make[18]: "/usr/ports/Mk/bsd.port.mk" line 4479: warning: duplicate script for target "/var/ports/usr/ports/editors/ghostwriter/work-qt5" ignored make[18]: "/usr/ports/Mk/bsd.licenses.mk" line 769: warning: using previous script for "/var/ports/usr/ports/editors/ghostwriter/work-qt5" defined here make[18]: "/usr/ports/Mk/bsd.port.mk" line 5193: warning: duplicate script for target "/var/ports/usr/ports/editors/ghostwriter/work-qt5" ignored ... make[18]: "/usr/ports/Mk/bsd.licenses.mk" line 769: warning: using previous script for "/var/ports/usr/ports/editors/ghostwriter/work-qt5" defined here make[18]: Graph cycles through /var/ports/usr/ports/editors/ghostwriter/work-qt5 make[19]: "/usr/ports/Mk/Uses/compiler.mk" line 80: warning: "cc --version" exited on a signal make[19]: "/usr/ports/Mk/Uses/compiler.mk" line 99: warning: "/usr/bin/clang --version" exited on a signal make[19]: "/usr/ports/Mk/Uses/compiler.mk" line 128: warning: "c++ -### /dev/null 2>&1" exited on a signal make[19]: "/usr/ports/Mk/Uses/compiler.mk" line 150: warning: "if cc -std=c89 -c -x c /dev/null -o /dev/null 2>&1; then echo yes; fi; echo" exited on a signal make[19]: "/usr/ports/Mk/Uses/compiler.mk" line 150: warning: "if cc -std=c99 -c -x c /dev/null -o /dev/null 2>&1; then echo yes; fi; echo" exited on a signal make[19]: "/usr/ports/Mk/Uses/compiler.mk" line 150: warning: "if cc -std=c11 -c -x c /dev/null -o /dev/null 2>&1; then echo yes; fi; echo" exited on a signal ... make[19]: "/usr/ports/Mk/bsd.licenses.mk" line 769: warning: using previous script for "/var/ports/usr/ports/editors/ghostwriter/work-qt5" defined here make[19]: Graph cycles through /var/ports/usr/ports/editors/ghostwriter/work-qt5 *** Signal 9 Stop. make[19]: stopped in /usr/ports/editors/ghostwriter *** Error code 1 ... Stop. make[1]: stopped in /usr/ports/editors/ghostwriter *** Error code 1 Stop. make: stopped in /usr/ports/editors/ghostwriter done [knew dan ~] %
I call that a success. Now I just have to figure out my tolerance for how many processes to allow.
Logging max processes
My next goal: how many processes does my jail use when processing a commit? This could vary, depending on how large the commit.
How do I log this information? Searching for monitoring number of processes in a jail freebsd, I found another FreeBSD Forums post which prompted me to try this:
[knew dan ~] % sudo rctl jail:empty:maxproc:log=25 [knew dan ~] % sudo rctl jail:empty:maxproc:sigkill=30
It’s really annoying that not specifying -a and specifying it results in the same output. Let me try again:
[knew dan ~] % sudo rctl -a jail:empty:maxproc:log=25 [knew dan ~] % sudo rctl jail:empty:maxproc:log=25 jail:empty:maxproc:sigkill=30
Now when I try this:
[knew dan ~] % sudo jexec empty /configure-plist.sh
The command is terminated because of the SIGKILL rule.
I went to look at the log file:
[empty dan ~] % tail -F /var/log/messages Jan 16 18:08:08 empty mosquitto[8964]: 1673892488: Starting in local only mode. Connections will only be possible from clients running on this machine. Jan 16 18:08:08 empty mosquitto[8964]: 1673892488: Create a configuration file which defines a listener to allow remote access. Jan 16 18:08:08 empty mosquitto[8964]: 1673892488: For more details see https://mosquitto.org/documentation/authentication-methods/ Jan 16 18:08:08 empty mosquitto[8964]: 1673892488: Warning: Protocol not supported Jan 16 18:08:08 empty mosquitto[8972]: 1673892488: Starting in local only mode. Connections will only be possible from clients running on this machine. Jan 16 18:08:08 empty mosquitto[8972]: 1673892488: Create a configuration file which defines a listener to allow remote access. Jan 16 18:08:08 empty mosquitto[8972]: 1673892488: For more details see https://mosquitto.org/documentation/authentication-methods/ Jan 16 18:08:08 empty mosquitto[8972]: 1673892488: Error: Address already in use Jan 16 18:08:08 empty mosquitto[8972]: 1673892488: Warning: Protocol not supported Jan 16 18:08:08 empty nrpe[8978]: Starting up daemon
Nothing. What’s wrong?
The messages are logged to the host silly, not the jail.
This is what I found in /var/log/messages on the jail host:
Jan 16 19:05:47 knew kernel: rctl: rule "jail:empty:maxproc:log=25" matched by pid 79337 (sh), uid 0, jail empty Jan 16 19:05:47 knew kernel: rctl: rule "jail:empty:maxproc:log=25" matched by pid 79339 (sh), uid 0, jail empty Jan 16 19:05:47 knew kernel: rctl: rule "jail:empty:maxproc:log=25" matched by pid 79341 (sh), uid 0, jail empty Jan 16 19:05:47 knew kernel: rctl: rule "jail:empty:maxproc:log=25" matched by pid 79343 (sh), uid 0, jail empty Jan 16 19:05:47 knew kernel: rctl: rule "jail:empty:maxproc:log=25" matched by pid 79345 (sh), uid 0, jail empty Jan 16 19:05:47 knew kernel: rctl: rule "jail:empty:maxproc:log=25" matched by pid 79347 (sh), uid 0, jail empty Jan 16 19:05:47 knew kernel: rctl: rule "jail:empty:maxproc:log=25" matched by pid 79349 (sh), uid 0, jail empty Jan 16 19:05:47 knew kernel: rctl: rule "jail:empty:maxproc:log=25" matched by pid 79351 (sh), uid 0, jail empty Jan 16 19:05:47 knew kernel: rctl: rule "jail:empty:maxproc:log=25" matched by pid 79353 (sh), uid 0, jail empty Jan 16 19:05:47 knew kernel: rctl: rule "jail:empty:maxproc:log=25" matched by pid 79355 (sh), uid 0, jail empty
I will implement something similar on the FreshPorts jail nodes. However, that is a different host, which now needs to be rebooted for slocum, another server in the basement which hosts dev, test, and stage for FreshPorts.
Initial configuration of slocum
After configuring slocum like I did knew near the top of this post, these are the commands I issued:
[slocum dan ~] % sudo rctl -a jail:dev-ingress01:maxproc:log=50 [slocum dan ~] % sudo rctl -a jail:test-ingress01:maxproc:log=100 [slocum dan ~] % sudo rctl -a jail:stage-ingress01:maxproc:log=200 [slocum dan ~] % sudo rctl jail:stage-ingress01:maxproc:log=200 jail:test-ingress01:maxproc:log=100 jail:dev-ingress01:maxproc:log=50
I’m not sure how many processes is appropriate, but let me take a guess and set each one different. If I start hitting 200 right away, I’ll bump to 300, 500, and 1000.
I had an idea while typing this. We already have these known limits checked by nrpe:
[slocum dan /usr/local/etc] % grep check_total_procs nrpe.cfg command[check_total_procs]=/usr/local/libexec/nagios/check_procs -w 1300 -c 1600
Looking in top right now, I see: 465 processes.
I think I’d be safe at setting the rctl rules to limit things to 1300 processes. Going back through service alerts in Nagios, I think that’s pretty safe.
Now it’s time to sit back and monitor.
This just in:
That’s the system periodic scripts kicking off at 0301 UTC:
NOTE: that’s the empty jail I used for testing. None of the three FreshPorts jails on the other host have reached this level.