I have a flatfiles as a source I'm loading 4files at a time using indirect load, here I want to load only distinct rows, if a record present in file1, the same record may contain in file2. So I need to insert only 1 record, the record should ignore, I tried sorter transformation distinct option, but it's not worked as I'm fetching the filename to the target. As filename is differs sorter thinks it's a distinct values.
Please help me with solution how to load only distinct records.
After sorting, utilizing the aggregator transformation with a group by on all columns except "Filename" will help extract distinct records using the maximum file name.
source date : id name file_name 101 anvesh files_1 102 sai krishna files_1 103 naren files_1 101 anvesh files_2 104 ankitha files_2 105 lakshmi files_2 101 anvesh files_3 102 sai krishna files_3 107 rajesh files_3
Target data: after aggregate id name file_name 101 anvesh files_3 102 sai krishna files_3 103 naren files_1 104 ankitha files_2 105 lakshmi files_2 107 rajesh files_3
