0

I have a file containing a bunch of folder names in it. After using getfile to get the data from the file, I'm splitting the data into strings in different queue and each queue contains the name of the folder in the attribute of flowfile.

Image of queues where each queue have a foldername in it in its FolderName attribute

Now I want to pass those queues to getsftp processor where I'll use $FolderName attribute in the Remote Path property of the processor to fetch data based on it.

Here the problem is that when I'm trying to connect any queue's success or failure to getsftp processor, it is not allowing me to connect to it. Here I researched and came to know that getsftp can't be subsequent processor of any other processor and only it can be parent of processors.

Is there any way to make getsftp a subsequent processor of any other processor? If no then is there any way where I can store the values of queues and then use that values in the Remote Path property of the processor to fetch data based on it?

P K
  • 162
  • 12
just learner
  • 33
  • 6
  • Instead of storing bunch of SFTP folder names in the input file, can you store the list of SFTP file names (in the format of `$folder/$filename`)? – P K Jun 14 '23 at 17:52

2 Answers2

0

No processor in NiFi can read a SFTP $foldername dynamically from a flow-file attribute and read all the file contents under the folder. You have to do some workaround - I have listed 2 workarounds as 2 separate answers.
I always use the combination of ListSFTP and FetchSFTP(which accepts an incoming connection). For your use case (you don't need ListSFTP), just connect the queue from your screenshot to the FetchSFTP processor and you should be good to go.
Note: FetchSFTP can only read contents of an SFTP file. So this will only work if you know the SFTP $path and $filename variables (i.e., complete name of the SFTP file to be read) Incoming connection to FetchSFTP I have explained the usage of ListSFTP and FetchSFTP processors as a separate answer.

P K
  • 162
  • 12
  • In getSFTP processor I have a property named Remote Path where I can mention folder name of my sftp and it will get all files present in that folder. In fetchSFTP where can I write the folder name to fetch all its file? – just learner Jun 14 '23 at 06:32
  • @justlearner `FetchSFTP` has a property named `Remote File`. I have updated my answer with the `FetchSFTP` processor properties. If you want to get a list of files from a folder, use `ListSFTP` processor (specify your `$folderName` there) and connect it's output to `FetchSFTP` processor (look at the screenshots in the answer) – P K Jun 14 '23 at 16:58
0

If you want to read all the files from an SFTP folder, use ListSFTP processor (specify your $folderName in the Remote Path property that currently has a .). This will return you one flowFile for every file under the folder (flowFile will have attributes ${path} and ${filename}). Use these attributes in FetchSFTP processor

ListSFTP processor properties

Here is the FetchSFTP processor properties you were interested enter image description here

This is our current PROD setup (which is working fine for years)

P K
  • 162
  • 12
  • in the Remote file parameter, you've mentioned '${path}/${filename}' which indicates that I've passed the folder name through the **path** attribute and name of a specific file using **filename**. Correct? – just learner Aug 28 '23 at 16:13
  • That’s correct @justlearner. Just for clarity, the path is not just the foldername, it should be the absolute path of the folder (example `${path}` is /usr/local/folder) – P K Aug 29 '23 at 01:04
  • suppose by using the path variable I gave the exact location of the folder but will I get all the files present in that folder? I want to get all the file names present in the folder. So if I pass ${path} will I get the names of all the files present? – just learner Aug 29 '23 at 04:28
  • Which processor you are talking about? `ListSFTP` or `FetchSFTP`. <> If you want to read all the files from an SFTP folder, use ListSFTP processor (specify your $folderName in the Remote Path property that currently has a .). This will return you one flowFile for every file under the folder (flowFile will have attributes ${path} and ${filename}). Use these attributes in FetchSFTP processor – P K Aug 29 '23 at 15:36