Opened 13 years ago

Closed 13 years ago

#378 closed defect (fixed)

FileUtil::exists() fails on large files

Reported by: Peter Owned by: Jari Häkkinen
Priority: major Milestone: yat 0.4.2
Component: utility Version: trunk
Keywords: Cc:

Description (last modified by Peter)

Calling FileUtil::exists() on a large (37Gb) file results in

terminate called after throwing an instance of 'theplu::yat::utility::IO_error'
  what():  IO_error: stat(2) call failed with errno: 75

errno = 75 means EOVERFLOW which indicates that the files size is to large to fit into the stat struct. My guess is that only 32 bits are used and since 232 is ~4G < 37G, it fails.

But I'm not interested in the file size. I just wanna know if the file exists or not.

Change History (7)

comment:1 Changed 13 years ago by Peter

Description: modified (diff)

comment:2 Changed 13 years ago by Peter

It turned out that my file was actually only 3.7Gb, so I'm not sure about the upper limit. What is an off_t defined as?

comment:3 Changed 13 years ago by Jari Häkkinen

232 is approximately 4G if unsigned ints are used, but for signed ints you get 231 (2G). There is 64 bit version of stat on my mac ... stat64, maybe sing this resolves the issue? (If stat64 works, we should check that this is also available on 32-bit Unixes.) The stat64 structure also uses off_t ... I haven't tried to look for off_t.

comment:4 Changed 13 years ago by Peter

You are right off_t seems to be a signed 32-bit illustrated by this cute snippet of code:

cout << "off_t min: " << numeric_limits<off_t>::max() << endl;
cout << "off_t max: " << numeric_limits<off_t>::min() << endl;
cout << "off64_t min: " << numeric_limits<off64_t>::max() << endl;
cout << "off64_t max: " << numeric_limits<off64_t>::min() << endl;

which gave this output

off_t min: 2147483647
off_t max: -2147483648
off64_t min: 9223372036854775807
off64_t max: -9223372036854775808

If stat64 is not available on 32-bits machines, it could be a good task for configure to figure out which one to use.

comment:5 Changed 13 years ago by Peter

Using stat64 on my monster file worked, so I guess stat64 is preferable.

Also I tested compiling code containing stat64 on my old kafka machine, and it worked. Can we conclude to change to stat64?

comment:6 Changed 13 years ago by Jari Häkkinen

Lets change to stat64. It will work for our machines. If someone complains we can always resolve the issue then and hopefully remember this discussion. Since you already tested with stat64 I suppose you can simply check in your changes and close this ticket.

comment:7 Changed 13 years ago by Peter

Resolution: fixed
Status: newclosed

in [1332]

Note: See TracTickets for help on using tickets.