pve 4.2 - CTs throw: "No space left on device"

When starting/stopping/restarting/status is run for a service in a CT located on a pve machine (version 4.2 is where I encountered this), the error: "No space left on device" is returned.

Question/Answer taken from: http://unix.stackexchange.com/questions/13751/kernel-inotify-watch-limit-reached

To avoid linkrot, here's that summarized:

Question:

I'm currently facing a problem on a linux box where as root I have commands returning error because inotify watch limit has been reached.

I googled a bit and every solution I found is to increase the limit with:

sudo sysctl fs.inotify.max_user_watches=<some random high number>

Answer:

• Each used inotify watch takes up 540 bytes (32-bit system), or 1 kB (double - on 64-bit) [sources: 1, 2]
• This comes out of kernel memory, which is unswappable.
• Assuming you set the max at 524288 and all were used (improbable), you'd be using approximately 256MB/512MB of 32-bit/64-bit kernel memory.
  • Note that your application will also use additional memory to keep track of the inotify handles, file/directory paths, etc. -- how much depends on its design.

To check the max number of inotify watches:

cat /proc/sys/fs/inotify/max_user_watches

To set max number of inotify watches

Temporarily:

• Run sudo sysctl fs.inotify.max_user_watches= with your preferred value at the end.

Permanently:

• Replace the value within the /proc/sys/fs/inotify/max_user_watches file with your own, i.e echo 524288 | sudo tee -a /proc/sys/fs/inotify/max_user_watches.

Check to see if the max number of inotify watches have been reached:

Use tail with the -f (follow) option on any old file, e.g. tail -f /var/log/dmesg: - If all is well, it will show the last 10 lines and pause; abort with Ctrl-C - If you are out of watches, it will fail with thissomewhat cryptic error:

tail: cannot watch '/var/log/dmsg': No space left on device

To see what's using up inotify watches

for foo in /proc/*/fd/*; do readlink -f $foo; done |grep inotify |cut -d/ -f3 |xargs -I '{}' -- ps --no-headers -o '%p %U %c' -p '{}' |uniq -c |sort -nr

The first column indicates used amount and the second shows the PID of that process [sources: 1, 2].