FileUtil::exists() fails on large files

[Peter]

Calling FileUtil::exists() on a large (37Gb) file results in

terminate called after throwing an instance of 'theplu::yat::utility::IO_error'
  what():  IO_error: stat(2) call failed with errno: 75

errno = 75 means EOVERFLOW which indicates that the files size is to large to fit into the stat struct. My guess is that only 32 bits are used and since 2³² is ~4G < 37G, it fails.

But I'm not interested in the file size. I just wanna know if the file exists or not.

[Peter]

It turned out that my file was actually only 3.7Gb, so I'm not sure about the upper limit. What is an off_t defined as?

[Jari Häkkinen]

2³² is approximately 4G if unsigned ints are used, but for signed ints you get 2³¹ (2G). There is 64 bit version of stat on my mac ... stat64, maybe sing this resolves the issue? (If stat64 works, we should check that this is also available on 32-bit Unixes.) The stat64 structure also uses off_t ... I haven't tried to look for off_t.

[Peter]

You are right off_t seems to be a signed 32-bit illustrated by this cute snippet of code:

cout << "off_t min: " << numeric_limits<off_t>::max() << endl;
cout << "off_t max: " << numeric_limits<off_t>::min() << endl;
cout << "off64_t min: " << numeric_limits<off64_t>::max() << endl;
cout << "off64_t max: " << numeric_limits<off64_t>::min() << endl;

which gave this output

off_t min: 2147483647
off_t max: -2147483648
off64_t min: 9223372036854775807
off64_t max: -9223372036854775808

If stat64 is not available on 32-bits machines, it could be a good task for configure to figure out which one to use.

[Peter]

Using stat64 on my monster file worked, so I guess stat64 is preferable.

Also I tested compiling code containing stat64 on my old kafka machine, and it worked. Can we conclude to change to stat64?

[Jari Häkkinen]

Lets change to stat64. It will work for our machines. If someone complains we can always resolve the issue then and hopefully remember this discussion. Since you already tested with stat64 I suppose you can simply check in your changes and close this ticket.

[Peter]

in [1332]