Opening lots and lots of files under Linux

Suppose you want a program, or a process to be precise, to open 10,000 files simultaneously. For some reason, I thought that, by default, this would be possible, but it seems that Linux sets the the default limit to 1024 files on most distributions we checked.

First of all, check how many files your system allows you to open simulateneously:

# cat /proc/sys/fs/file-max
101066

On my system, as you can see, a process should be able to open 100,000 files simultaneously without a problem. If your number is much lower, you may need to do some extra work.

Unfortunately, there are security settings above and beyond this number. To get around them, add the following line to “/etc/security/limits.conf”:

* - nofile 100000

Then, you need to log in again (fresh, not within X). To make sure it worked, type

#ulimit -n
100000

As you can see, it worked for me.

To make double sure it works, you might try to run the following C++ program:

#include <fstream>
#include <string>
#include <iostream>
#include <fstream>
#include <sstream>
#include <vector>
using namespace std;
int main() {
fstream lfs[10000];
for (int k = 0 ; k < 10000; ++k) {
stringstream strs;
strs << "stupidtest" << k;
lfs[k].open(strs.str().c_str(),ios::out);
assert(lfs[k].good());
if(lfs[k].good()) {
cout << "file created "<<strs.str()<< " ok!" << endl;
}
}
for (int k = 0 ; k < 10000; ++k) {
stringstream strs;
strs << "stupidtest" << k;
if(lfs[k].good()) {
cout << "file "<<strs.str()<< " still ok!" << endl;
}
lfs[k] << k;
if(lfs[k].good()) {
cout << "file "<<strs.str()<< " still ok
after write!" << endl;
}
}
for (int k = 0 ; k < 10000; ++k) {
lfs[k].close();
}
}

One thought on “Opening lots and lots of files under Linux”

  1. Fust as an FYI for people reading this, that setting also effects the number of open socket connections the OS will allow. If you are running something with a large number of concurrent connections (say, a chat server) you can into the same problem. The error message reported will generally be something like “too many open files”, a bit of a misnomer for sockets. Have fun!

Leave a Reply

Your email address will not be published. Required fields are marked *

To create code blocks or other preformatted text, indent by four spaces:

    This will be displayed in a monospaced font. The first four 
    spaces will be stripped off, but all other whitespace
    will be preserved.
    
    Markdown is turned off in code blocks:
     [This is not a link](http://example.com)

To create not a block, but an inline code span, use backticks:

Here is some inline `code`.

For more help see http://daringfireball.net/projects/markdown/syntax