Life on the LAN: October 2014

I had InfluxDB (a great time-series datastore) running on Ubuntu, and kept running into these errors:

IO error: /opt/influxdb/shared/data/db/shard_db_v2/00031/356947.ldb: Too many open files

The issue was that the Ubuntu default for open files is 1024:

root@server:~# ulimit -Sn
1024

There were two steps required to fix this issue with InfluxDB.

Set the open file limit system wide

Edit /etc/security/limits.conf and add the following:
* soft nofile 100000
* hard nofile 100000

This raised the limits, but when the InfluxDB process started, it was still getting the old limits:

iclark@gar-network-datastore:~$ cat /proc/992/limits

Limit Soft Limit Hard Limit Units

Max cpu time unlimited unlimited seconds

Max file size unlimited unlimited bytes

Max data size unlimited unlimited bytes

Max stack size 8388608 unlimited bytes

Max core file size 0 unlimited bytes

Max resident set unlimited unlimited bytes

Max processes 47573 47573 processes

Max open files 1024 4096 files

Max locked memory 65536 65536 bytes

Max address space unlimited unlimited bytes

Max file locks unlimited unlimited locks

Max pending signals 47573 47573 signals

Max msgqueue size 819200 819200 bytes

Max nice priority 0 0

Max realtime priority 0 0

Max realtime timeout unlimited unlimited us

To fix this, I had to set the ulimit value in the influxdb init script

Edit /etc/init.d/influxdb and add the following to the beginning of the script:
ulimit -n 100000

After this value was added, I just stopped and started the InfluxDB process, and voila:

root@gar-network-datastore:~# cat /proc/1279/limits