(Published via time machine. Just kidding, my blog publication tooling was broken and it took me a long time to get around to fixing it.)
Today I was trying to make a large number of socket connections to a local Phoenix app (on MacOS 12.3 Monterey) and was running into limits on the maximum mumber of file descriptors (sockets in this case).
Specifically, I would see [warning] Ranch acceptor reducing accept rate: out of file descriptors and lots of [error] [label: {:erl_prim_loader, :file_error}, report: 'File operation error: emfile. Target: /path/to/Elixir.Phoenix.Endpoint.Cowboy2Handler.beam. Function: get_file. Process: code_server.'].
(emfile means out of file descriptors according to man 2 intro.)
After much reading and asking and experimenting, this seems to be the deal.
There are two limits on the number of file descriptors on MacOS: shell limits and system limits. I think that if a process hits either of those limits (the lower of the two), it will have errors, although I'm less sure about the system limits.
First, the shell limits.
ulimit -a shows them all, like:
-t: cpu time (seconds) unlimited
-f: file size (blocks) unlimited
-d: data seg size (kbytes) unlimited
-s: stack size (kbytes) 8176
-c: core file size (blocks) 0
-v: address space (kbytes) unlimited
-l: locked-in-memory size (kbytes) unlimited
-u: processes 5333
-n: file descriptors 200000
...or one can use ulimit -n to just see the file descriptor limit.
Hitting the file descriptor limit caused this warning:
Ranch acceptor reducing accept rate: out of file descriptors
and a bunch of associated errors.
According to this thread:
This is not a bug in Cowboy (or Ranch), Ranch just shows a message when the OS refuses to give more file descriptors.
ulimit -n 524288 will set the limit for a given shell session; adding this to .zshrc (or .bashrc or whatever the shell uses) will set it for all new sessions.
Second, the are system-wide limits.
I'm less confident about the effect of hitting a system-wide limit because the behavior was less consistent for me.
I'm pretty sure I saw an error about the system limit on maxfiles at one point, but I couldn't reliably reproduce it; maybe that has something to do with the soft vs hard limits.
I did see weird behavior during this experiment, like [error] Postgrex.Protocol (#PID<0.420.0>) failed to connect: ** (DBConnection.ConnectionError) tcp connect (localhost:5432): can't assign requested address - :eaddrnotavail or [error] Postgrex.Protocol (#PID<0.419.0>) disconnected: ** (DBConnection.ConnectionError) tcp recv (idle): timeout or being unable to launch a web request in the browser or via curl 😱.
All of which makes me think that setting the system limits higher is a good idea.
According to man launchctl, launchctl limit shows all the system-wide limits as found via getrlimit(2).
Output looks like:
cpu unlimited unlimited
filesize unlimited unlimited
data unlimited unlimited
stack 8372224 67092480
core 0 unlimited
rss unlimited unlimited
memlock unlimited unlimited
maxproc 5333 8000
maxfiles 256 unlimited
(Aisde 1: One can also use launchctl limit maxfiles to see just that one.)
(Aside 2:
Similar outp\ut can be seen for the "kernel systectl parameters":
~: sysctl -A | grep maxfiles
kern.maxfiles: 245760
kern.maxfilesperproc: 122880)
The first number in the launchctl limit output is the "soft" limit and the second the "hard" limit.
What's the difference?
According to man 2 setrlimit:
A resource limit is specified as a soft limit and a hard limit. When a soft limit is exceeded a process may receive a signal (for example, if the cpu time or file size is exceeded), but it will be allowed to continue execution until it reaches the hard limit (or modifies its resource limit).
These limits can be modified temporarily with a command like sudo launchctl limit maxfiles 524288 524288 (if you pass unlimited as the second argument, it ignores the command.)
Be warned ⚠️ that setting this value too low or too high can cause system-wide problems, and that it is not possible to specify unlimited as the hard limit, even though that's the default.
According to this post, unlimited means INT_MAX, which is equal to 2147483647 (a 32-bit SIGNED INT), and setting it any higher will cause the number to be interpreted as negative and crash MacOS.
One can make those limits persist restarts by creating /Library/LaunchDaemons/limit.maxfiles.plist to look like this:
# /Library/LaunchDaemons/limit.maxfiles.plist
Label
limit.maxfiles
ProgramArguments
launchctl
limit
maxfiles
524288
524288
RunAtLoad
ServiceIPC
Be sure to chown root:wheel limit.maxfiles.plist and set file permissions -rw-r–r– with chmod 644 limit.maxfiles.plist.
I got some of this information from this article.