1

I'm looking for ideas for an Open Source ETL or Data Processing software that can monitor a folder for CSV files, then open and parse the CSV.

For each CSV row the software will transform the CSV into a JSON format and make an API call to start a Camunda BPM process, passing the cell data as variables into the process.

Looking for ideas,

Thanks

George
  • 11
  • 2
  • Given that almost any coding framework or ETL tool can do this, you would need to provide more detailed requirements before anyone could sensibly help you i.e. given a list of 20 tools that all meets the requirements you’ve listed, how would you narrow down your selection? At the moment, any recommendation would be no different to what you could google for yourself – NickW Aug 30 '21 at 09:21
  • Ok, thanks for your comment. I was hoping to find a low-code solution. I'm kinda stuck and have already spent several days googling for a solution but I'm just getting stuck in analysis paralysis so I'm looking for a way to get unstuck. – George Aug 30 '21 at 23:29
  • Please edit the question to limit it to a specific problem with enough detail to identify an adequate answer. – Community Sep 03 '21 at 09:53

1 Answers1

2

  1. You can use a Java WatchService or Spring FileSystemWatcher as discussed here with examples:
    How to monitor folder/directory in spring?
    referencing also:
    https://www.baeldung.com/java-nio2-watchservice

  2. Once you have picked up the CSV you can use my example here as inspiration or extend it: https://github.com/rob2universe/csv-process-starter specifically https://github.com/rob2universe/csv-process-starter/blob/main/src/main/java/com/camunda/example/service/CsvConverter.java#L48
    The example starts a configurable process for every row in the CSV and includes the content of the row as a JSON process data.

  3. I wanted to limit the dependencies of this example. The CSV parsing logic applied is very simple. Commas in the file may break the example, special characters may not be handled correctly. A more robust implementation could replace the simple Java String .split(",") with an existing CSV parser library such as Open CSV

The file watcher would actually be a nice extension to the example. I may add it when I get around to it, but would also accept a pull request in case you fork my project.

rob2universe
  • 7,059
  • 39
  • 54