InfluxDB on Ubuntu: Too many open files

I had InfluxDB (a great time-series datastore) running on Ubuntu, and kept running into these errors:

IO error: /opt/influxdb/shared/data/db/shard_db_v2/00031/356947.ldb: Too many open files

The issue was that the Ubuntu default for open files is 1024:

root@server:~# ulimit -Sn
1024

There were two steps required to fix this issue with InfluxDB.

• Set the open file limit system wide
• Edit /etc/security/limits.conf and add the following:
• * soft nofile 100000
• * hard nofile 100000

This raised the limits, but when the InfluxDB process started, it was still getting the old limits:

iclark@gar-network-datastore:~$ cat /proc/992/limits

Limit                     Soft Limit           Hard Limit           Units

Max cpu time              unlimited            unlimited            seconds

Max file size             unlimited            unlimited            bytes

Max data size             unlimited            unlimited            bytes

Max stack size            8388608              unlimited            bytes

Max core file size        0                    unlimited            bytes

Max resident set          unlimited            unlimited            bytes

Max processes             47573                47573                processes

Max open files            1024                 4096                 files

Max locked memory         65536                65536                bytes

Max address space         unlimited            unlimited            bytes

Max file locks            unlimited            unlimited            locks

Max pending signals       47573                47573                signals

Max msgqueue size         819200               819200               bytes

Max nice priority         0                    0

Max realtime priority     0                    0

Max realtime timeout      unlimited            unlimited            us

• To fix this, I had to set the ulimit value in the influxdb init script
• Edit /etc/init.d/influxdb and add the following to the beginning of the script:
• ulimit -n 100000

After this value was added, I just stopped and started the InfluxDB process, and voila:

root@gar-network-datastore:~# cat /proc/1279/limits

Limit                     Soft Limit           Hard Limit           Units

Max cpu time              unlimited            unlimited            seconds

Max file size             unlimited            unlimited            bytes

Max data size             unlimited            unlimited            bytes

Max stack size            8388608              unlimited            bytes

Max core file size        0                    unlimited            bytes

Max resident set          unlimited            unlimited            bytes

Max processes             47573                47573                processes

Max open files            100000               100000               files

Max locked memory         65536                65536                bytes

Max address space         unlimited            unlimited            bytes

Max file locks            unlimited            unlimited            locks

Max pending signals       47573                47573                signals

Max msgqueue size         819200               819200               bytes

Max nice priority         0                    0

Max realtime priority     0                    0

Max realtime timeout      unlimited            unlimited            us

Twitter typeahead.js with BootStrap 3, AJAX, and CoffeeScript

I ran into a few problems today while deploying Twitter's typeahead.js.  I had dynamic typeahead data that I needed to pull from a URL (like an AJAX request), and I also needed to integrate typeahead with Bootstrap 3 (which breaks almost all of TypeAhead's CSS, and screws up bootstrap's form layout).  I also wanted to use BloodHound but have it search any part of the word (not just the beginning of the word), and have typeahead.js submit the form when an item was selected by clicking.  This project's JavaScript is actually written as CoffeeScript, so the last thing I had to do was convert working JavaScript to CoffeeScript.

The first thing I sorted out was the CSS.  Adding the following corrected all of my CSS issues (note: not all of this is CSS I wrote; much of it was pulled from various other BootStrap/typeahead solutions online, but it is tweaked as none of those worked 100% for me):

/* Twitter typeahead compatibility fixes */
.twitter-typeahead {
  float: left;
  margin-right: 3px;
}

.tt-suggestion {
  display: block;
  padding: 3px 20px;
}

.twitter-typeahead .tt-hint {
  color:#a1a1a1;
  padding: 6px 12px;
  border:1px solid transparent;
}

.twitter-typeahead .tt-query {
  border-radius: 4px!important;
  border-top-right-radius: 0!important;
  border-bottom-right-radius: 0!important;
}

.tt-dropdown-menu {
  min-width: 160px;
  margin-top: 2px;
  padding: 5px 0;
  background-color: #fff;
  border: 1px solid #ccc;
  border: 1px solid rgba(0,0,0,.2);
  *border-right-width: 2px;
  *border-bottom-width: 2px;
  -webkit-border-radius: 6px;
  -moz-border-radius: 6px;
  border-radius: 6px;
  -webkit-box-shadow: 0 5px 10px rgba(0,0,0,.2);
  -moz-box-shadow: 0 5px 10px rgba(0,0,0,.2);
  box-shadow: 0 5px 10px rgba(0,0,0,.2);
  -webkit-background-clip: padding-box;
  -moz-background-clip: padding;
  background-clip: padding-box;
}

.tt-cursor {
  cursor: pointer;
  color: #fff;
  background-color: #0081c2;
  background-image: -moz-linear-gradient(top, #0088cc, #0077b3);
  background-image: -webkit-gradient(linear, 0 0, 0 100%, from(#0088cc), to(#0077b3));
  background-image: -webkit-linear-gradient(top, #0088cc, #0077b3);
  background-image: -o-linear-gradient(top, #0088cc, #0077b3);
  background-image: linear-gradient(to bottom, #0088cc, #0077b3);
  background-repeat: repeat-x;
  filter: progid:DXImageTransform.Microsoft.gradient(startColorstr='#ff0088cc', endColorstr='#ff0077b3', GradientType=0)
}

.tt-suggestion.tt-is-under-cursor a {
  color: #fff;
}

.tt-suggestion p {
  margin: 0;
}

After my form was looking normal again, I had to get typeahead working. After a lot of trial and error and converting it to CoffeeScript, here is what I had (my API URL was at /v1/devices/list). Another note, not all of this is my original code, it is bits and pieces of javascript from around the web which I cobbled together and converted to CoffeeScript:

typeahead = ->
  devices = new Bloodhound({
    datumTokenizer: (d) ->
      test = Bloodhound.tokenizers.whitespace(d.value)
      $.each(test, (k,v) ->
        i = 0
        while( (i+1) < v.length )
          test.push(v.substr(i,v.length))
          i++
      )
      return test
    ,
    queryTokenizer: Bloodhound.tokenizers.whitespace,
    limit: 10,
    prefetch: {
      url: '/v1/devices/list',
      filter: (list) ->
        $.map(list, (device) -> { value: device })
    }
  })

  # kicks off the loading/processing of `local` and `prefetch`
  devices.clearPrefetchCache()
  devices.initialize()


  # passing in `null` for the `options` arguments will result in the default
  # options being used
  $('.typeahead').typeahead(
    { 
      highlight: true,
    },
    { 
      name: 'devices',
      # `ttAdapter` wraps the suggestion engine in an adapter that
      # is compatible with the typeahead jQuery plugin
      source: devices.ttAdapter()
    },
  )
  $('input.typeahead').bind("typeahead:selected", -> $("form").submit() )

I hope this helps someone else out there!

Ruby on Rails gem: Uninitialized Constant (class name)

I was working on a Ruby on Rails project earlier, and ran into a really frustrating issue regarding gems.

I added the influxdb gem to my Gemfile, did a bundle install, saw it installed successfully, then tried to use it in an ApplicationController.  No luck!  Over and over it was acting like the gem wasn't installed (Uninitialized Constant error ApplicationController::InfluxDB when trying to reference the class defined in the gem).

When I ran the same code using the rails console, it worked fine, which lead me to believe it was some problem with bundler or the environments.

It turns out the solution was simply to restart the apache/ngnix/whatever server! "service apache2 restart" in my case. Whew!