Semaphores in Linux: how to configure them

I’ve been contributing to a distributed search engine project (Majestic-12) for around 2 years and a half. They made the choice to develop their crawler in C#, which means I need to use Mono to run it on my Ubuntu boxes (mainly, the server where this very site is hosted). While Mono is a great .NET framework, it’s not exactly perfect, and in the latest version of Ubuntu Server (10.10) 64 bits, using Mono 2.4.4x, I ran into an issue with semaphores. Something that just looked like a semaphore leak.

What are semaphores?

I’m not really sure I understood it fully. To make it very short and simple, it’s a bit like a token to access system resources. So when there is no more semaphores available, a program has to wait until one is freed to run.
For more numerous and accurate details, see Semaphores in Linux and Semaphores on Wikipedia.

My issue with the Majestic-12 crawler on Mono/Linux was that it would progressively eat up all semaphores (so, like a memory leak but with semaphores = a semaphore leak?), resulting in error messages such as:

** ERROR **: shm_semaphores_init: semget error: No space left on device. Try deleting some semaphores with ipcs and ipcrm
or increase the maximum number of semaphore in the system.
aborting…

** (process:10841): WARNING (recursed) **: Thread (nil) may have been prematurely finalized

It also resulted in non-specific error messages (program failing without mentioning semaphores even though their were the root of the problem).

How to list semaphores and remove them one by one

To list used semaphore, use the command ipcs -a. This will output shared memory segments, and a bit below the used semaphores, like:

------ Semaphore Arrays --------
key        semid      owner      perms      nsems
0xffffffff 0          mj1        600        8
0x00000000 32769      mj2        600        8
0x00000000 65538      mj3        600        8
0x00000000 43319299   mj2        600        8
0x00000000 131076     mj2        600        8

Normally, when you exit a program, it will remove linked semaphores. But with my MJ12 nodes, it wasn’t the case. You can then remove one specific semaphore using ipcrm -s [semaphore ID], for instance to remove the second one in the list above you’d use ipcrm -s 32769. Of course, in my case, since I have tons of semaphores used up, a system restart is faster…

How to increase the maximum amount of semaphores

First you’ll want to check out the current maximum semaphores value. By default it’s usually around 200-250. To get the current value, use:
/sbin/sysctl -a | grep sem
This will output something like:

error: "Invalid argument" reading key "fs.binfmt_misc.register"
error: permission denied on key 'net.ipv4.route.flush'
kernel.sem = 5000       32000   32      128
error: permission denied on key 'net.ipv6.route.flush'

So here we have 5000 semaphores (that was my “fix” to the issue: add many semaphore, so the semaphore leak continues but since the stock is higher, the system freezes less often).

To set the new value, use:
/sbin/sysctl -w kernel.sem=300
This will only change the value for the current run. To make it persistent (keep the change after reboots), you need to edit /etc/sysctl.conf (nano /etc/sysctl.conf): add (or edit) the line:
kernel.sem = 300 (where 300 is the number of semaphores you want).

Sources

• Increase semaphores count in a Linux machine
• Linux Tutorial – The Operating System – The Kernel – Kernel Mechanisms – Semaphores