Parsing csv data into object with different column length

Question

I am new to Java and practicing parsing csv file into the object. I've tried but cannot figure it out.

The file looks like this:

  [0],    [1], [2],    [3]  ,    [4]    ,   [5]   ,  [6] ,   [7]  ,  [8] , [9]
class, gender, age, bodyType, profession, pregnant, isYou ,species, isPet, role
scenario:green,   ,         ,           ,         ,        ,      ,      ,
person, female, 24, average ,           , FALSE   ,        ,      ,      , passenger
animal, male  ,  4,         ,           , FALSE   ,        , dog  , TRUE , pedestrian
scenario:red
person, male  , 16, athletic, boxer     , FALSE   ,  TRUE  ,      ,      , passenger
person, female, 25, athletic, doctor    , TRUE    ,  FALSE ,      ,      , pedestrian

I need to parse it by any number of passengers and pedestrians with any scenarios. Finally, add these scenarios into an ArrayList for analyzing.

What I think is to:

1. loop through each line, stops when reaches to the next scenario:red, adds the passengers and the pedestrians to the Character ArrayList. (I've done adding, but don't how to stop).
2. Create a scenario using constructor scenario(ArrayList<Character> passenger, ArrayList<Character> pedestrians, boolean redOrGreen);
3. The ArrayList scenarios add the created scenarios.

What I've done is put everything together instead of separate them. Any help or hint is highly appreciated.

Thanks for this community who helped me, here is what I've got so far.

    public void loadCsv() throws IOException {

    String csvFile = "config.csv";
    String line = "";
    String csvSplit = "\\s*,\\s*";

    Scenario scenario = new Scenario();
    Person person = new Person();
    Animal animal = new Animal();

    ArrayList<Scenario> scenaios = new ArrayList<Scenario>();
    ArrayList<String> csvContents = new ArrayList<String>();

    ArrayList<Character> passengers = new ArrayList<Character>();
    ArrayList<Character> pedestrians = new ArrayList<Character>();

    try (BufferedReader csvReader = new BufferedReader(new FileReader(csvFile));) {
        String headerLine = csvReader.readLine(); //get rid of the header
        //add each line to the arrayList
        while ((line = csvReader.readLine()) != null) { 
            csvContents.add(line);   
        }

        for(String csvLine : csvContents) {                
            String[] data = csvLine.split(csvSplit); // split by comma and remove redundant spaces

            if (data.length == NO_OF_FIELD) { //check and avoid indexOutOfBoundException

                String clazz = data[0].toLowerCase();// cannot use word "class" as a variable

                if (clazz.startsWith("scenario") && data.length == 1) { 
                    scenario = new Scenario();
                    scenario.setLegalCrossing(clazz.endsWith("green"));
                    continue;
                } 

                else if ("person".equals(clazz) && data.length ==10) {

                    person = loadCsvPerson(data);
                    addCharacter(person, data);

                } 

                else if ("animal".equals(clazz) && data.length ==10) {

                    animal = loadCsvAnimal(data);
                    addCharacter(animal, data);

                    }                        
                }
            }                               
        }
        //passenger and pedestrians are in position
        System.out.println("passengers: " + passengers);
        System.out.println("pedestrians: " + pedestrians);

        if (null != scenario) {
            scenario.setPassengers(passengers);
            scenario.setPedestrians(pedestrians);
        }

    } catch (FileNotFoundException e) {
        e.printStackTrace();
    } 
}

Answer 1

If it is possible to change the csv file format, I would add scenario type column (and scenario id or name if required), so you can work with csv file as a result set from database when you join tables (scenario + passenger + pedestrian) and return plain rows.

With this approach you will be able to delegate parsing to any csv library and do your logic (group by scenario id/name/type) separately. With surrogate rows you have (scenario:green...) you have to write your custom parser.

For example, you can use univocity to simply parse file into your model (even using annotations) and iteratively group it and handle.

Answer 2

Or if you need to work with existing file format do something like that:

if (clazz.startsWith("scenario") && data.length == 1) { 
    // collect existing scenario before starting processing new one
    if (scenario != null) {
        scenario.setPassengers(passengers);
        scenario.setPedestrians(pedestrians);

        passengers = new ArrayList();
        pedestrians = new ArrayList();

        scenarios.add(scenario);
    }

    // now start new group (scenario)
    scenario = new Scenario();
    scenario.setLegalCrossing(clazz.endsWith("green"));
    continue;
} 

Answer 3

Following things need to be addressed in your code:

1. Strive to avoid using the name of a class which is already used by the standard library (and especially when it is in the default package, java.lang) e.g. there is already a class Character in Java library and therefore you should use a different name for your custom class.

2. Use continue to skip the line, scenario:red

   for(String csvLine : csvContents) { 
       if(csvLine.equals("scenario:red")){
           continue;
       }
       String[] data = csvLine.split(csvSplit); // split by comma and remove redundant spaces
   
       if (data.length == NO_OF_FIELD) {
           //..
       }
       //..
   }
   

3. If you have already defined final int NO_OF_FIELD = 10, you can use the same instead of using the value 10 directly i.e. you should use NO_OF_FIELD instead of 10 in the following code:

   if (data.length == NO_OF_FIELD) { //check and avoid indexOutOfBoundException
   
       String clazz = data[0].toLowerCase();// cannot use word "class" as a variable
   
       //...
   
       else if ("person".equals(clazz) && data.length ==10) {
   

   However, you also need to understand that && data.length ==10 is unnecessary here as you have already checked data.length == NO_OF_FIELD in the enclosing if condition.

I couldn't understand the rest of your points. If you clarify them, I'll be able to help you further.

Answer 4

I need to add the previous scenario in the second round. Since the last set of data won't be captured, I need to set another new scenario to add it in. Thanks for the art sir.

Character character = null;
    try (BufferedReader csvReader = new BufferedReader(new FileReader(csvFile));) {
        String headerLine = csvReader.readLine(); //get rid of the header
        //add each line to the arrayList
        while ((line = csvReader.readLine()) != null) { 
            csvContents.add(line);   
        }
        final int NO_OF_FIELDS = 10;

        for(String csvLine : csvContents) {                
            String[] data = csvLine.split(csvSplit); // split by comma and remove redundant spaces                
            String clazz = data[0].toLowerCase();// cannot use word "class" as a variable

            if (clazz.startsWith("scenario") && data.length == 1) {
                // adding scenario after one set of data
                // i.e second round adding the first round data
                if (passengers.size() != 0 && pedestrians.size() != 0) {
                    Scenario scenario = new Scenario();
                    scenario.setPassengers(passengers);
                    scenario.setPedestrians(pedestrians);
                    scenarios.add(scenario);
                }

                passengers = new ArrayList<Character>();
                pedestrians = new ArrayList<Character>();

                if (clazz.endsWith("green")) {
                    scenario.setLegalCrossing(true);
                    System.out.println("green light");
                }

                else if (clazz.endsWith("red")){
                    scenario.setLegalCrossing(false);
                    System.out.println("red light");
                } 

                continue;
            }
        //...
            Scenario scenario = new Scenario();
            scenario.setPassengers(passengers);
            scenario.setPedestrians(pedestrians);
            scenarios.add(scenario);

            scenario.setPassengers(passengers);
            scenario.setPedestrians(pedestrians);

            Audit audit = new Audit();
            audit.setScenario(scenarios);