JaxB marshaler overwriting file contents

Question

I am trying to use JaxB to marshall objects I create to an XML. What I want is to create a list then print it to the file, then create a new list and print it to the same file but everytime I do it over writes the first. I want the final XML file to look like I only had 1 big list of objects. I would do this but there are so many that I quickly max my heap size.

So, my main creates a bunch of threads each of which iterate through a list of objects it receives and calls create_Log on each object. Once it is finished it calls printToFile which is where it marshalls the list to the file.

public class LogThread implements Runnable {
//private Thread myThread;
private Log_Message message = null;
private LinkedList<Log_Message> lmList = null;
LogServer Log = null;
private String Username = null;

public LogThread(LinkedList<Log_Message> lmList){
    this.lmList = lmList;
}

public void run(){
    //System.out.println("thread running");
    LogServer Log = new LogServer();
    //create iterator for list
    final ListIterator<Log_Message> listIterator = lmList.listIterator();

    while(listIterator.hasNext()){
        message = listIterator.next();
        CountTrans.addTransNumber(message.TransactionNumber);
        Username = message.input[2];
        Log.create_Log(message.input, message.TransactionNumber, message.Message, message.CMD);
    }
    Log.printToFile();
    init_LogServer.threadCount--;
    init_LogServer.doneList();
    init_LogServer.doneUser();
    System.out.println("Thread "+ Thread.currentThread().getId() +" Completed user: "+ Username+"... Number of Users Complete: " + init_LogServer.getUsersComplete());
    //Thread.interrupt();
}
}

The above calls the below function create_Log to build a new object I generated from the XSD I was given (SystemEventType,QuoteServerType...etc). These objects are all added to an ArrayList using the function below and attached to the Root object. Once the LogThread loop is finished it calls the printToFile which takes the list from the Root object and marshalls it to the file... overwriting what was already there. How can I add it to the same file without over writing and without creating one master list in the heap?

public class LogServer {
public log Root = null;
public static String fileName = "LogFile.xml";
public static File XMLfile = new File(fileName);

public LogServer(){
    this.Root = new log();
}
//output LogFile.xml
public synchronized void printToFile(){
    System.out.println("Printing XML");
    //write to xml file
    try {
        init_LogServer.marshaller.marshal(Root,XMLfile);
    } catch (JAXBException e) {
        e.printStackTrace();
    }
    System.out.println("Done Printing XML");
}


private BigDecimal ConvertStringtoBD(String input){
    DecimalFormatSymbols symbols = new DecimalFormatSymbols();
    symbols.setGroupingSeparator(',');
    symbols.setDecimalSeparator('.');
    String pattern = "#,##0.0#";
    DecimalFormat decimalFormat = new DecimalFormat(pattern, symbols);
    decimalFormat.setParseBigDecimal(true);
    // parse the string
    BigDecimal bigDecimal = new BigDecimal("0");
    try {
        bigDecimal = (BigDecimal) decimalFormat.parse(input);
    } catch (ParseException e) {
        // TODO Auto-generated catch block
        e.printStackTrace();
    }
    return bigDecimal;
}

public QuoteServerType Log_Quote(String[] input, int TransactionNumber){
    BigDecimal quote = ConvertStringtoBD(input[4]);
    BigInteger TransNumber = BigInteger.valueOf(TransactionNumber);
    BigInteger ServerTimeStamp = new BigInteger(input[6]);
    Date date = new Date();
    long timestamp = date.getTime();
    ObjectFactory factory = new ObjectFactory();
    QuoteServerType quoteCall = factory.createQuoteServerType();

    quoteCall.setTimestamp(timestamp);
    quoteCall.setServer(input[8]);
    quoteCall.setTransactionNum(TransNumber);
    quoteCall.setPrice(quote);
    quoteCall.setStockSymbol(input[3]);
    quoteCall.setUsername(input[2]);
    quoteCall.setQuoteServerTime(ServerTimeStamp);
    quoteCall.setCryptokey(input[7]);

    return quoteCall;
}
public SystemEventType Log_SystemEvent(String[] input, int TransactionNumber, CommandType CMD){
    BigInteger TransNumber = BigInteger.valueOf(TransactionNumber);
    Date date = new Date();
    long timestamp = date.getTime();
    ObjectFactory factory = new ObjectFactory();
    SystemEventType SysEvent = factory.createSystemEventType();

    SysEvent.setTimestamp(timestamp);
    SysEvent.setServer(input[8]);
    SysEvent.setTransactionNum(TransNumber);
    SysEvent.setCommand(CMD);
    SysEvent.setFilename(fileName);

    return SysEvent;
}

public void create_Log(String[] input, int TransactionNumber, String Message, CommandType Command){
    switch(Command.toString()){
    case "QUOTE":  //Quote_Log
        QuoteServerType quote_QuoteType = Log_Quote(input,TransactionNumber);
        Root.getUserCommandOrQuoteServerOrAccountTransaction().add(quote_QuoteType);
        break;

    case "QUOTE_CACHED":
        SystemEventType Quote_Cached_SysType = Log_SystemEvent(input, TransactionNumber, CommandType.QUOTE);
        Root.getUserCommandOrQuoteServerOrAccountTransaction().add(Quote_Cached_SysType);
        break;

}
}

EDIT: The below is code how the objects are added to the ArrayList

    public List<Object> getUserCommandOrQuoteServerOrAccountTransaction() {
    if (userCommandOrQuoteServerOrAccountTransaction == null) {
        userCommandOrQuoteServerOrAccountTransaction = new ArrayList<Object>();
    }
    return this.userCommandOrQuoteServerOrAccountTransaction;
}

Answer 1

Jaxb is about mapping java object tree to xml document or vice versa. So in principle, you need complete object model before you can save it to xml. Of course it would not be possible, for very large data, for example DB dump, so jaxb allows marshalling object tree in fragments, letting the user control moment of the object creation and marshaling. Typical use case would be fetching records from DB one by one and marshaling them one by one to a file, so there would not be problem with the heap.

However, you are asking about appending one object tree to another (one fresh in memory, second one already represented in a xml file). Which is not normally possible as it is not really appending but crating new object tree that contains content of the both (there is only one document root element, not two).

So what you could do,

• is to create new xml representation with manually initiated root element,
  • copy the existing xml content to the new xml either using XMLStreamWriter/XMLStreamReader read/write operations or unmarshaling the log objects and marshaling them one by one.
  • marshal your log objects into the same xml stram
  • complete the xml with the root closing element. -

Vaguely, something like that:

XMLStreamWriter writer = XMLOutputFactory.newInstance().createXMLStreamWriter(new FileOutputStream(...), StandardCharsets.UTF_8.name());
//"mannually" output the beginign of the xml document == its declaration and the root element
writer.writeStartDocument();
writer.writeStartElement("YOUR_ROOT_ELM");

Marshaller mar = ...
mar.setProperty(Marshaller.JAXB_FRAGMENT, true); //instructs jaxb to output only objects not the whole xml document

PartialUnmarshaler existing = ...; //allows reading one by one xml content from existin file, 

while (existing.hasNext()) {
    YourObject obj = existing.next();
    mar.marshal(obj, writer);
    writer.flush();
}

List<YourObject> toAppend = ...
for (YourObject toAppend) {
    mar.marshal(obj,writer);
    writer.flush();
}

//finishing the document, closing the root element
writer.writeEndElement();
writer.writeEndDocument();

Reading the objects one by one from large xml file, and complete implementation of PartialUnmarshaler is described in this answer: https://stackoverflow.com/a/9260039/4483840

That is the 'elegant' solution. Less elegant is to have your threads write their logs list to individual files and the append them yourself. You only need to read and copy the header of the first file, then copy all its content apart from the last closing tag, copy the content of the other files ignoring the document openkng and closing tag, output the closing tag.

If your marshaller is set to marshaller.setProperty(Marshaller.JAXB_FORMATTED_OUTPUT, true); each opening/closing tag will be in different line, so the ugly hack is to copy all the lines from 3rd to one before last, then output the closing tag.

It is ugly hack, cause it is sensitive to your output format (if you for examle change your container root element). But faster to implement than full Jaxb solution.