Recently get a frustrating error on one of the server. I introduced a new gem amatch in my rails application. I do not see any issue on local, dev or UAT server instance in installing the gem through the bundler, But on QC the bundler failed with below error:

Installing amatch 0.3.0 with native extensions
Gem::Ext::BuildError: ERROR: Failed to build gem native extension.

creating Makefile

make "DESTDIR=" clean

make "DESTDIR="
compiling amatch_ext.c
amatch_ext.c:1661:1: fatal error: error writing to /tmp/ccAtSusl.s: 
No space left on device
}
^
compilation terminated.
make: *** [amatch_ext.o] Error 1

make failed, exit code 2

Since the error clearing saying that No space left on device, I checked the space on server.

$df -h
Filesystem      Size  Used Avail Use% Mounted on
/dev/xvda1       35G  6.1G   27G  19% /
none            4.0K     0  4.0K   0% /sys/fs/cgroup
udev            1.9G   12K  1.9G   1% /dev
tmpfs           375M  304K  375M   1% /run
none            5.0M     0  5.0M   0% /run/lock
none            1.9G     0  1.9G   0% /run/shm
none            100M  4.0K  100M   1% /run/user
overflow        1.0M     0  1.0M   0% /tmp

But I see there is a lot of space on the disk

Since, the error say problem while writing to tmp folder, I checked permission of temp folder, thinking may we it causing problem.

$ ls -l / | grep tmp
drwxrwxrwt 3 root root 4096 Nov 19 13:51 tmp

But it also showing all the permissions of read and write.

Then I tried to remove all the content of tmp folder thinking , may be something is there which taking all the space

$ rm -fr /tmp/*

But again got the same error.

Now googling for some time on space distribution and partitioning on linux, came to know about overflow partition .

And I got the problem, you can see that the in the df -h command output above, the tmp is mounted on overflow . This will happen if your server continuously keep writing to tmp folder, and if you have not given a separate partition to tmp .  When system deduct some partition or memory issue it automatically mount the tmp folder to overflow partition with size of 1 MB, so that things keep working. But there is no automatic reversal i,e bringing back the tmp to root partition.

Now this could be happened on our server over the time. We have consumed our disk space, system moved tmp folder to overflow, later on we free the space by deleting unused file or increasing space, thus having free space as shown above in df -h command. But tmp folder didn’t moved back from overflow partition and having size of 1 MB. Since amatch need more then 1 MB, it is failing with space issue.

So solution is simple, unmount the tmp folder and mount it again.

# umount /tmp
umount: /tmp: device is busy.
        (In some cases useful info about processes that use
         the device is found by lsof(8) or fuser(1))

You need to kill the process using tmp folder, get the list of those process, with below command:

# lsof /tmp
COMMAND    PID USER   FD   TYPE DEVICE SIZE/OFF     NODE NAME
Passenger 6730 root   13rW  REG   0,23        0 14247683 /tmp/passenger

Kill the running process 6730 and try to unmount again.

# kill -9 6730
# umount /tmp

Mount tmp folder again and see the disk space again.

# mount -a
# df -h
Filesystem      Size  Used Avail Use% Mounted on
/dev/xvda1       35G  6.1G   27G  19% /
none            4.0K     0  4.0K   0% /sys/fs/cgroup
udev            1.9G   12K  1.9G   1% /dev
tmpfs           375M  308K  375M   1% /run
none            5.0M     0  5.0M   0% /run/lock
none            1.9G     0  1.9G   0% /run/shm
none            100M  4.0K  100M   1% /run/user

You can see that, there is no separate overflow partition for tmp folder as it now mounted on the root partition, thus it can use all the available memory in the main partition.

This time amatch didn’t reported any issue and installed successfully.

CHECKLIST IF YOU FACE THIS PROBLEM:

1. Check permission of folder
2. Check available space.Free space and see if solve your problem
3. If on overflow partition unmount and mount it again.

Resources:
http://unix.stackexchange.com/questions/15024/umount-device-is-busy-why
http://www.tldp.org/LDP/intro-linux/html/sect_03_01.html