0

The following code use writeCharacters with small buffer size to writing the content of big file, but it seems it only works when writing the content to a xml file, but writeCharacters() will crash when writting the content of a big file to QBuffer, any solution? Thanks.

env: qt opensource 4.8.7; Visual Studio 2010; Windows 10;

big file: https://1drv.ms/u/s!Ap_EAuwC9QkXijbujBQcQk4Hat_O?e=KemgUY

#include <QtCore/QCoreApplication>
#include <QFile>
#include <QXmlStreamWriter>
#include <QBuffer>

int main(int argc, char *argv[])
{
   
    QCoreApplication a(argc, argv);

    QByteArray mContentBuffer;
    QByteArray arr;
    int pos, filesize;

    QFile file("C:\\Work\\bigfile.xar"); //Szie: 300Mb

    if(!file.open(QFile::ReadOnly))
    {
        return -1;
    }

    mContentBuffer = file.readAll();
    file.close();

    //QFile profile("C:\\Work\\profile.xml");

    //if(!profile.open(QFile::WriteOnly|QFile::Truncate))
    //{
    //    return -1;
    //}
    QBuffer buffer;
    buffer.open(QBuffer::ReadWrite);

    QXmlStreamWriter stream(&buffer);    

    stream.setAutoFormatting(true);
    stream.writeStartDocument();
    stream.writeStartElement("Profile");

    //stream.writeTextElement("Content", mContentBuffer.toBase64());
    stream.writeStartElement("Content");
    pos = 0; 
    filesize = mContentBuffer.size();         
    while(pos<filesize){
       arr = mContentBuffer.mid(pos, 2000000);
       stream.writeCharacters(arr.toBase64());
       pos+=arr.size();
    }
    stream.writeEndElement();
    stream.writeEndElement(); // Profile
    stream.writeEndDocument();

    return 0;
}

Here is the goal for writing a big file into a QBuffer.

bool profileModified()
{
    bool result = true;

    QFile profile("C:\\Work\\profile.xml");
    if(profile.exists())
    {
        QBuffer buffer;
        buffer.open(QBuffer::ReadWrite);
        QXmlStreamWriter stream(&buffer);
        
        exportProfile(stream);

        profile.open(QFile::ReadOnly);
        QByteArray profileArr = profile.readAll();
    
        buffer.seek(0);
      
        QByteArray bufferArr = buffer.buffer();
         
        result = (array_compare(profileArr, bufferArr) != 0);
        
        profile.close();
    }

    return result;
}
Ming
  • 379
  • 1
  • 6
  • 20
  • Referring to [your previous question](https://stackoverflow.com/questions/75317978/writetextelement-of-qxmlstreamwriter-crashed-when-writting-the-content-of-a-bi/75362260?noredirect=1#comment133052143_75362260), I see you read the file all at once. Secondly, you seem to encode the data in chunks whose size is not a multiple of 3; that produces incorrect results. Last, I am questionning your goal; what do you need so much data in a buffer for? This does not look like what a buffer should be used for. – Atmo Feb 10 '23 at 04:43
  • I tried the 90000 1800000 as truck size and it will not fix the crash. I've updated the goal for writing a big file into a QBuffer, see the code example. – Ming Feb 10 '23 at 06:52
  • 1
    The [multiple of 3] constraint is only to get a correct result, not to avoid the crash. To avoid the crash, you have to answer why you read a 300MB file at once and why you want your `QBuffer` to grow so large. – Atmo Feb 10 '23 at 10:58
  • Thanks, I turned on the option /LARGEADDRESSAWARE, that crash is gone. – Ming Feb 12 '23 at 01:42
  • 1
    Well, you are not removing the crash so much as you are hiding it. And when I say "hiding", it is going to be clear to anyone watching the memory consumption of your applications, since it requires 100's of MB instead of 10's of KB, that something is wrong. For developers I work with, I would expect them to apply a square-root rule by default, that is to process 100MB, make 10k loops of 10kB each and from there, see if a better balance between memory consumption and breaking the data down can be found. Your stubborn approach would grant them a severe rebuke. – Atmo Feb 12 '23 at 02:03
  • Thanks, I'll set up a limit on the size of files to be imported as a workaround. I don't know why the this application have to save a binary file in a xml and comparing the XML files like that, looking for a way to not saving the file at all or comparing the xml files in a more efficient way will be a ultimate solution. – Ming Feb 12 '23 at 05:22

0 Answers0