How to get the number of inotify watches used by a process

Recently, PulseAudio often refused to start because the system ran out of inotify watches. I tried to Google this issue and found that most posts simply told me to increase the allowance of inotify watches. I was not satisfied with this. I wanted to know the underlying cause of this issue. That is, why the default allowance (which is 8192) was not enough and, also, what program was responsible for running out of inotify watches. Knowing this information, I may be able to find a better solution, instead of blindly increasing the allowance without knowing who is using it.

After googling, I found some posts talking about how to know what processes/programs are using inotify. But I could hardly find posts about how to know HOW MANY INOTIFY WATCHES ARE USED BY EACH OF THOSE PROCESSES/PROGRAMS.

I finally found one in a reply in this thread. The author of that reply spent a lot of words explaining the idea behind his/her solution. I re-organized the information, eliminated those explanations, and kept only the method.

By this method, I finally found out who is responsible for running out of inotify watches: Insync. I got over 50K files in the sync folder of Insync, and so Insync used over 12.5K watches, which exceeds the default allowance, 8192.

Step 1: Find what processes/programs are using inotify

$ sudo find /proc/*/fd -lname anon_inode:inotify | cut -d/ -f3 | xargs -I '{}' -- ps --no-headers -o '%p %U %c' -p '{}' | uniq -c | sort -nr

In the output, each row represents a process/program using inotify. The first column is the number of inotify “instances” created by that program. The second column is PID. And the third and the fourth are the user and the name of that program, respectively.

Note it is the number of “instances”, not the number of “watches”.

Step 2: Find the file descriptor of inotify instances

Now we pick a program we found, and then we want to know how many “watches” the program uses. But first we need to know the file descriptor of the “instances” created by this program:

$ sudo ls -athlr /proc/<PID>/fd | grep anon_inode:inotify

The number of rows in the output should match the number of “instances” created by that program.

The file descriptor is the number before -> anon_inode:inotify in each line. For example, in lr-x------ 1 <use> <group> 64 Mar 21 23:24 19 -> anon_inode:inotify, the file descriptor for this inotify instance is 19.

Step 3: Get the number of watches in each instance

After we have the file descriptor of each instance created by our target program, we can calculate the number of watches in that instance by:

$ sudo wc -l /proc/<PID>/fdinfo/<file descriptor>

The first column in the output is the number we are looking for.

This is just the number of an instance of a program. To know the total watches of a program, we have also to check the number of other instances.

A fake example

Here’s a fake example to give a sense.

Step 1:

$ sudo find /proc/*/fd -lname anon_inode:inotify | cut -d/ -f3 | xargs -I '{}' -- ps --no-headers -o '%p %U %c' -p '{}' | uniq -c | sort -nr
--------------------------------------------------------------------
5        6679     root     systemd
3        7878     user     chrome
1        12345    user     termite

Step 2: Let’s say we want to know the number of watches of Chrome.

$ sudo ls -athlr /proc/7878/fd | grep anon_inode:inotify
---------------------------------------------------
lr-x------ 1 user user 64 Mar 21 23:24 5 -> anon_inode:inotify
lr-x------ 1 user user 64 Mar 21 23:24 18 -> anon_inode:inotify
lr-x------ 1 user user 64 Mar 21 23:24 19 -> anon_inode:inotify

Step3:

$ sudo wc -l /proc/7878/fdinfo/5
---------------------------------------------
2        /proc/565/fdinfo/5

$ sudo wc -l /proc/7878/fdinfo/18
---------------------------------------------
5        /proc/565/fdinfo/18

$ sudo wc -l /proc/7878/fdinfo/19
---------------------------------------------
1        /proc/565/fdinfo/19

Then, the total number of watches of Chrome is 2+3+1=6.