0

Here is the qt code that crash:

Writing the content of a 130MB binary file will crash, but writing a 2MB binary file will not.

Is there any solution for writeTextElement to write big file content? Thanks

env: qt opensource 4.8.7 Visual Studio 2010 Windows 10

big file: https://1drv.ms/u/s!Ap_EAuwC9QkXijOmXqxp9DEav3gm?e=iha0uI

test project: https://1drv.ms/u/s!Ap_EAuwC9QkXijWzOlpaWmtzOdtz?e=fDpo93

#include <QtCore/QCoreApplication>
#include <QFile>
#include <QXmlStreamWriter>


int main(int argc, char *argv[])
{
    QCoreApplication a(argc, argv);

    QByteArray mContentBuffer;

    QFile file("C:\\Work\\bigfile.xar");
   
    if(!file.open(QFile::ReadOnly))
    {
        return -1;
    }

    mContentBuffer = file.readAll();
    file.close();

    QFile profile("C:\\Work\\profile.xml");

    if(!profile.open(QFile::WriteOnly|QFile::Truncate))
    {
        return -1;
    }

    QXmlStreamWriter stream(&profile);    

    stream.setAutoFormatting(true);
    stream.writeStartDocument();
    stream.writeStartElement("Profile");

    stream.writeTextElement("Content", mContentBuffer.toBase64());

    stream.writeEndElement(); // Profile
    stream.writeEndDocument();

    return a.exec();
}

enter image description here

Ming
  • 379
  • 1
  • 6
  • 20
  • 1
    What debug actions have you performed? If I follow you, your code reads the entire 130MB of data, puts it in memory, then tries to write it back into your XML. Could you have not read bits little bit little instead? – Atmo Feb 02 '23 at 04:53
  • @Atmo , I updated the code sample. the code open a binary file and reads into a QByteArray, and then write the mContent.toBase64() to a xml file. – Ming Feb 03 '23 at 00:16
  • Your edit does not really make it more clear... nor does it address my question about the debugging you have performed. If I can advise, I would recommend you break down the 1st line of code you have shown above into 2: the "read file" part, then the "write into XML" part. At the very least, you will be able to debug and see which half fails. – Atmo Feb 03 '23 at 04:51
  • Also, the code you have added does not look like something I could test; please read [How to create a Minimal, Reproducible Example](https://stackoverflow.com/help/minimal-reproducible-example). – Atmo Feb 03 '23 at 04:53
  • @Atmo, Thanks, I've updated the code example and the big file to reproduce this problem. – Ming Feb 06 '23 at 07:20

1 Answers1

1

The minimal reproducible example was a life saver.

Several things seem to be required to solve your issue in a clean way.

  1. Mainly, the file must be read in chunks.
    To be precise and as mentioned in that other question, in chunks whose size is a multiple of 3 bytes (I have hardcoded 9000 below).
    You will need to look for the value that gives you the best performance while guaranteeing it never fails yourself.
  2. Instead of writing in the XML in 1 go, I made it so the XML file is being written each chunk at a time.
    To achieve that, the code creates the node first, then uses the writeCharacters method to write text in it.
  3. Note the if (mContentBuffer.isEmpty()) right before writing. It is indeed specified in Qt Assistant that the read function has no way of reporting errors; returning an empty QByteArray can mean either that no data was currently available for reading, or that an error occurred. Considering the while loop, it can only be an error, so we abort everything.
    BTW, I think but have not tested that you can just end the document, and the XML stream writer will close everything it needs to have a valid XML.
  4. Just to make it clean, I reordered the lines of codes so that the XML file is opened before the xar file. I also added QtCore/ on every include. Not sure that last point is going to be useful on your real code.

This code implements all the above changes.
It should not crash and while you will not be able to test the output on the 130MB file, you should be able to test your code vs mine on the 2MB file you has successfully read from before.

#include <QtCore/QCoreApplication>
#include <QtCore/QFile>
#include <QtCore/QXmlStreamWriter>

int main(int argc, char* argv[])
{
    QCoreApplication a(argc, argv);

    QFile profile("C:\\Work\\profile.xml");
    if (!profile.open(QFile::WriteOnly | QFile::Truncate))
        return -1;

    QXmlStreamWriter stream(&profile);
    stream.setAutoFormatting(true);
    stream.writeStartDocument();
    stream.writeStartElement("Profile");
    stream.writeStartElement("Content");

    QFile file("C:\\Work\\bigfile.xar");
    if (!file.open(QFile::ReadOnly))
        return -1;

    while (!file.atEnd()) {
        QByteArray mContentBuffer = file.read(9000);
        if (mContentBuffer.isEmpty()) {
            stream.writeEndDocument();
            file.close();
            profile.close();
            return -1;
        }
        stream.writeCharacters(mContentBuffer.toBase64());
    }
    file.close();

    stream.writeEndElement(); // </Content>
    stream.writeEndElement(); // </Profile>
    stream.writeEndDocument();

    return a.exec();
}
Atmo
  • 2,281
  • 1
  • 2
  • 21
  • Thanks @Atmo, do you possibly know why it will crash? now I can successfully output the profile.xml. – Ming Feb 07 '23 at 07:01
  • Most likely because it required too big a contiguous memory block. The fact `QByteArray` uses contiguous memory is documented [here](https://doc.qt.io/qt-6/qbytearray.html#maximum-size-and-out-of-memory-conditions) (read the full section up to the last sentence). You code needs one 130MB block (not a big deal if it was not contiguous) to store the text, then probably 200MB or so for the encoded version (again, not a big deal if it was not contiguous) while the first block was still allocated. Either it could not find the memory or your OS did not try finding it, hence the `null` pointer. – Atmo Feb 07 '23 at 11:33
  • In your code, the encoded byte array gets freed after the call to write it to the XML file, and the original file content gets freed when the object gets destroyed, i.e. when the function returns. – Atmo Feb 07 '23 at 11:34
  • Thanks a lot, @Atmo, could you take look at my new question? https://stackoverflow.com/questions/75405999/writecharacters-of-qxmlstreamwriter-crashed-when-writting-the-content-of-a-big – Ming Feb 10 '23 at 01:04