Why does camel move file with only partial data?

Question

Below is my code:

from("quartz2:report?cron=" + cronExpression)
.routeId("jms:queue:test")
.setHeader("CURRENT_TIME", simple("${date:now:MM-dd-yyyy HH:mm:ss.S}"))     
        //Writing column names for report to the file
        .setBody()
        .simple(summaryHeaders)
        .transform(body().append("\n"))
        .to("file:" + filePath + "?fileName="+scheduledFileName+"${date:now:MM-dd-YYYY}.csv&fileExist=append")
.to("sql:" + dataSummaryQuery + "?dataSource=#dataSource")
        .log(LoggingLevel.INFO, "Summary query executed")
        .marshal(csvFormat)
        .to("file:" + filePath + "?fileName="+scheduledFileName+"${date:now:MM-dd-YYYY}.csv&fileExist=append")
        .log(LoggingLevel.INFO, "Report written to file")
.setBody()
        .simple(fileHeaders)
        .transform(body().append("\n"))
        .to("file:" + filePath + "?fileName="+scheduledFileName+"${date:now:MM-dd-YYYY}.csv&fileExist=append")

        .to("sql:" + dataExtractionQuery + "?dataSource=#dataSource")
        .log(LoggingLevel.INFO, "data query executed")
        .marshal(csvFormat)

        .to("file:" + filePath + "?fileName="+scheduledFileName+"${date:now:MM-dd-YYYY}.csv&fileExist=append");

Till the above code, everything is work fine and all the required data is in the file. When I try to move the file to a different location, the moved file only has the data from the dataextractionquery above. The previous data are not in the file. What is happening here? How can I move the file with all the data added?

Code to move the file:

from("file:" + filePath + "?fileName="+scheduledFileName+"${date:now:MM-dd-YYYY}.csv&fileExist=append")
        .to("file:" + filePath + "?fileName=temp/");

If I do not move the file, I have all the data in the file

score 1 · Accepted Answer · answered Apr 14 '21 at 13:35

What is happening here?

The second route's performing

file consumer scan source file every 500 ms (default delay value is 500)
file producer overwrite target file (default fileExist value is Override)
[Async to above step] file consumer move the file to .camel sub-folder relative to source directory (default move setting)

MOVE AND DELETE OPERATIONS

By default, Camel will move consumed files to the .camel sub-folder relative to the directory where the file was consumed.

Whenever some part of data is stream to source, the route consumed it, overwrite target file (with the data in source) and then throw it away. At last, the route got last part of data and write it to target location.

How can I move the file with all the data added?

There is a warning (Avoid reading files currently being written by another application) and a section cover this. List some possible method for your reference.

Use doneFileName (check USING DONE FILES and WRITING DONE FILES)
Set sufficient large value for delay
Use readLock
Stop move by noop and control idempotent behavior by idempotentKey (set with name and size)

Why does camel move file with only partial data?

1 Answers1