CStdioFile cannot work with files larger than 2GB?

Question

I am using Visual C++ 2008. In VC++ 2008, CFile supports 2^64 huge files. So I think CStdioFile should also support.

However, when using CStdioFile::GetLength() on a file larger than 2GB, I get a CFileException, below is the code snippet:

void CTestCStdioFileDlg::OnBnClickedButton1()
{
    // TODO: Add your control notification handler code here
    CStdioFile MyFile;
    CString strLine;
    ULONGLONG uLength;

    strLine = _T("This is a line.");

    if (MyFile.Open(_T("C:\\Temp\\MyTest.dat"), CFile::modeCreate | CFile::modeWrite | CFile::shareExclusive | CFile::typeBinary))
    {
        for (UINT uIndex = 0; uIndex = 200000000; uIndex ++)
        { 
            MyFile.WriteString(strLine);
            uLength = MyFile.GetLength();
        }

        MyFile.Close();
    }
}

After tracing into the CStdio::GetLength(), I find the following code snippet will raise exception, as below:

   nCurrent = ftell(m_pStream);            -> This will return -1
   if (nCurrent == -1)
      AfxThrowFileException(CFileException::invalidFile, _doserrno,
         m_strFileName);

It is amazing that CStdioFile still use ftell instead of _ftelli64 to deal with the stream.

I then search the document for CStdioFile, I cannot find any document on CStdioFile::GetLength, the only related is https://learn.microsoft.com/en-us/cpp/mfc/reference/cstdiofile-class?view=vs-2019#seek, and it asks me to see fseek document. But in fseek document, I still not find any related to file size limit.

Finally I find a third-party site that indicates CStdioFile::GetLength contains error: http://www.flounder.com/msdn_documentation_errors_and_omissions.htm#CStdioFile::GetLength, but it does not provide a solution.

Other than these, there are hardly any questions or posts on the 2GB limit of CStdioFile online. That is really strange.

I try to check the source code of CStdioFile iN VC++ 2017 and it is same as that for 2008.

So whether there is a simple solution for the problem without rewriting the whole CStdioFile class?

It might be a limitation of [`ftell`](https://learn.microsoft.com/en-us/cpp/c-runtime-library/reference/ftell-ftelli64?view=vs-2019) which returns a `long`. Visual Studio `long` type is a 32-bit type, even on 64-bit systems. — Some programmer dude, May 30 '20 at 03:06
@Someprogrammerdude, yes, that is the case. I just wonder if there is a simple solution for this problem? — alancc, May 30 '20 at 03:12
Don't use `CStdioFile`? Is there a reason it's used in your program? If it's a legacy program that you need to maintain, have you thought of refactoring it to use e.g. `CFile` instead? Or inherit from `CStdioFile` and override [`GetLength`](https://learn.microsoft.com/en-us/cpp/mfc/reference/cfile-class?view=vs-2019#getlength) (it's `virtual`)? — Some programmer dude, May 30 '20 at 03:17

score 5 · Accepted Answer · edited Jun 20 '20 at 09:12

CStdioFile exists primarily for a single reason: It's constructor taking a FILE* argument. The purpose of this class is to wrap a C Runtime file and expose it through a CFile-compatible interface. The implementation inherits all of the C Runtime limitations, in particular the file size limit of 2GB.

To address this, there are several options:

If possible, drop the use of CStdioFile altogether. Unless you are interfacing with (legacy) C code, there's no striking reason to ever use it.
If that is not an option, derive a custom implementation from CStdioFile and override all class members that surface file-relative offsets (GetPosition(), GetLength(), Seek()). All other class members are unaffected, and can simply be inherited. (Note: Make sure to restore the current file pointer when implementing GetLength().)

Microsoft provides extensions to its C Runtime that offer 64-bit wide offsets (_ftelli64 and _fseeki64).

If you need to go with option 2, the following extension to CStdioFile can be used as a drop-in replacement with support for files larger than 2GB.

CStdioFileExt.h:

#pragma once

#include <afx.h>

class CStdioFileExt : public CStdioFile
{
    DECLARE_DYNAMIC(CStdioFileExt)
public:
    ULONGLONG GetPosition() const override;
    ULONGLONG GetLength() const override;
    ULONGLONG Seek(LONGLONG lOff, UINT nFrom) override;
};

CStdioFileExt.cpp:

#include "CStdioFileExt.h"

ULONGLONG CStdioFileExt::GetPosition() const
{
    ASSERT_VALID(this);
    ASSERT(m_pStream != NULL);

    auto const pos = _ftelli64(m_pStream);
    if (pos == -1L)
        AfxThrowFileException(CFileException::invalidFile, _doserrno, m_strFileName);
    return static_cast<ULONGLONG>(pos);
}

ULONGLONG CStdioFileExt::GetLength() const
{
    ASSERT_VALID(this);

    auto const nCurrent = _ftelli64(m_pStream);
    if (nCurrent == -1L)
        AfxThrowFileException(CFileException::invalidFile, _doserrno, m_strFileName);

    auto nResult = _fseeki64(m_pStream, 0, SEEK_END);
    if (nResult != 0)
        AfxThrowFileException(CFileException::badSeek, _doserrno, m_strFileName);

    auto const nLength = _ftelli64(m_pStream);
    if (nLength == -1L)
        AfxThrowFileException(CFileException::invalidFile, _doserrno, m_strFileName);
    nResult = _fseeki64(m_pStream, nCurrent, SEEK_SET);
    if (nResult != 0)
        AfxThrowFileException(CFileException::badSeek, _doserrno, m_strFileName);

    return static_cast<ULONGLONG>(nLength);
}

ULONGLONG CStdioFileExt::Seek(LONGLONG lOff, UINT nFrom)
{
    ASSERT_VALID(this);
    ASSERT(nFrom == begin || nFrom == end || nFrom == current);
    ASSERT(m_pStream != NULL);

    if (_fseeki64(m_pStream, lOff, nFrom) != 0)
        AfxThrowFileException(CFileException::badSeek, _doserrno, m_strFileName);

    auto const pos = _ftelli64(m_pStream);
    return static_cast<ULONGLONG>(pos);
}

IMPLEMENT_DYNAMIC(CStdioFileExt, CStdioFile)

Thank you very much. I choose option 2 as I need a buffered file class to improve the performance. — alancc, Jun 23 '20 at 04:35

CStdioFile cannot work with files larger than 2GB?

1 Answers1