1

I have a spring input channel defined like this

<file:inbound-channel-adapter prevent-duplicates="false" id="inpChannel" directory="file:/Users/abhisheksingh/req" auto-startup="true">
        <int:poller id="poller" fixed-delay="1000" />
</file:inbound-channel-adapter>

<int:service-activator input-channel="inpChannel" ref="inpHandler" />

The file name example as TEST.SQQ. SQQ is the file format which the client uses to place the files in ftp. However, I see that the same file is picked up by the spring ftp adapter again and again with different file names. So the first time it is TEST.SQQ. Then the the next time it is TEST.SQQ-20170204.PQQ and then the next time it is TEST.SQQ-20170204.PQQ.20170304.PQQ. This keeps on continuing. I have a filter on my end which checks the name of the file already processed. But since the file name being polled is different each time, all of these files are picked up for processing.

This is my ftp adapter -

<int-ftp:inbound-channel-adapter id="sqqFtpInbound"
    channel="ftpChannel"
    session-factory="sqqFtpClientFactory"
    auto-create-local-directory="true"
    delete-remote-files="false"
    local-filter="acceptAllFileListFilter"
    local-directory="file:/Users/abhisheksingh/ddrive/everge_ws/sqqReq" auto-startup="true" >
    <int:poller id="poller" fixed-delay="1000" />
</int-ftp:inbound-channel-adapter>

Here is my ftp server image -

enter image description here

Here is my local directory image -

enter image description here

I dont understand why the same file gets picked up again and again. I will appreciate some help !

This is my file list filter code.

public class TestFileListFilter<F> extends AbstractFileListFilter<F> {

    private static final Logger log = LoggerFactory.getLogger(EvergeFileListFilter.class);

    @Override
    protected boolean accept(F file) {
        File f = (File) file;
        if(f.getAbsolutePath().contains(".PQQ")) {

            String newDir = "/Users/abhisheksingh/ddrive/sample/pqqReq/";

            String archiveLocation = "/Users/abhisheksingh/ddrive/sample/pqqArchive/";
            String fullName = archiveLocation + f.getName();
            log.info("Check if the file has already been processed " + fullName);

            File fl = new File(fullName);
            final File dir = new File(archiveLocation);
            for (final File child : dir.listFiles()) {

                String archiveName = FilenameUtils.getBaseName(child.getName());
                String inputName = FilenameUtils.getBaseName(fl.getName());
                log.info("Archive file name is " + archiveName);
                log.info("Input file name is " + inputName);
                if(inputName.contains(archiveName)) {
                    log.info("The file is already processed "+inputName);
                }

            }

            if(fl.exists()) {
                log.error("PQQ file has already been processed.");
                removeFile(f);
                return false;
            }else{
                log.info("PQQ File received " + f.getAbsolutePath());
            }
            moveFile(f, newDir);
            return true;
        }
    }
user3276247
  • 1,046
  • 2
  • 9
  • 24
  • You know, your question isn't clear. The `TEST.SQQ`, `EST.SQQ-20170204.PQQ` and `TEST.SQQ-20170204.PQQ.20170304.PQQ` are really different file names... Also not sure why do you show to us that ``. Please, rephrase your problem. Also I see that your `` doesn't have any `filter` for remote dir. – Artem Bilan Feb 06 '17 at 17:24
  • Remote location has only one file - TEST.SQQ. If the same file keeps getting polled again then it is okay. I archive the files which has been processed. However every time the file gets picked by the sqqFtpInbound adapter, the same file comes back and gets stored in my local file store with different different names. Because of this, I have to process the file again and again. – user3276247 Feb 06 '17 at 18:05
  • Problem statement is very clear. Remote directory has one file. When the inbound adapter polls it and my local filter starts working on it, I get the file names with different different names. I have shown all the configurations just to make sure that I have not messed up with any of the configurations. – user3276247 Feb 06 '17 at 18:07
  • I dont want to add any filter for remote directory. – user3276247 Feb 06 '17 at 18:07
  • Also I had told SQQ to save some client information. Actual name is PQQ. This is evident from the images I have attached. It seems thousand words cant explain what an image can do. – user3276247 Feb 06 '17 at 18:14

1 Answers1

0

I think your custom local-filter has some vulnerabilities to rely on an nonexistent fact to wait for unique files from remote store.

You should ensure that ability because it isn't switched by default.

For this purpose consider to add filter option to the <int-ftp:inbound-channel-adapter> as a reference to the AcceptOnceFileListFilter or FtpPersistentAcceptOnceFileListFilter.

We have a JIRA on the matter.

Please, confirm that it is exactly an issue for you and we might revise a priority for that ticket and will fix it soon.

Artem Bilan
  • 113,505
  • 11
  • 91
  • 118
  • @ artem_bilan, Thanks for spending time on my question. I went through the JIRA. It says that there is no default filter and every time we poll all the files. This is okay with me. However when the same file gets polled again, some time stamp gets appended to it. I dont know how this magic happens. At least name should not change. – user3276247 Feb 07 '17 at 12:24
  • I have updated my question with TestFileListFilter code. Well to me, it looks simple. I don't think there is any vulnerability. Do let me know if you see anything wrong with the code. – user3276247 Feb 07 '17 at 12:27
  • "every time we poll all the files. This is okay with me". ??? You have a question "I dont understand why the same file gets picked up again and again". I give you an answer. What else ? – Artem Bilan Feb 07 '17 at 14:15
  • Those timesptamps in the file names really looks odd for me as well. There is nothing in the Framework what does that for us. So, that looks like you have some custom logic in your application else. Would be great to get from you as small config as possible to play from our side. I don't think that `FileListFilter` should have such a hard work. It's really not its responsibility. – Artem Bilan Feb 07 '17 at 14:16
  • @artem_bilan , So there was some vulnerability in the filter. I have fixed it. Thanks for spending time on this. I just have a follow up question on this spring ftp. Even when I have stopped the application in tomcat, I see the files still getting polled. So what I do to replicate this - I stop the application. I delete all the files in my local directories. I see the polled file reappear. To validate this, I renamed the file on ftp and deleted the polled file from my local. Now The renamed file has come up. I have already executed ./shutdown.sh. So the application is stopped. – user3276247 Feb 08 '17 at 12:48
  • Glad to see that you have fixed original problem! Another one is pretty odd. Let's make up it into a separate SO question! – Artem Bilan Feb 08 '17 at 13:17