How to validate CSV headers using Univocity routines?

Question

Im using Univocity CSV parser with routines when I iterate over Java beans. Is there a way to validate CSV header? When I edit the CSV and add invalid header, it just insert into given bean null without any error.

Model class:

public class Customer {

@Format(formats ="yyyy-MM-dd")
@Parsed(field="C_DAY")
private Date day;

@Parsed(field="C_ID")
private Long id;

@Parsed(field="C_TYPE")
private String type;

@Format(formats ="yyyy-MM-dd")
@Parsed(field="C_ORIGIN_DATE")
private Date originDate;

@Format(formats ="yyyy-MM-dd")
@Parsed(field="C_REL_DATE")
private Date relDate;

@Parsed(field="C_LEGAL_ID")
private String legalId;

@Parsed(field="C_NAME")
private String name;}

Parser:

    @Autowired
private CustomerDAO dao;

public void parse(File file) throws IOException, SQLException, CustomerValidationException, ParseException {
    CsvParserSettings parserSettings = new CsvParserSettings();
    parserSettings.getFormat().setLineSeparator("\n");
    parserSettings.setHeaderExtractionEnabled(false);
    CsvRoutines routines = new CsvRoutines(parserSettings);
    List<Customer> customers = new ArrayList<>();
    java.util.Date stamp = getTimestamp(file);
    dao.checkTimestampDate(stamp);


    for (Customer customer : routines.iterate(Customer.class, file, "UTF-8")) {
        validateFileDateWithFileName(stamp, customer.getDay());
        validateCustomer(customer);
        customers.add(customer);
    }
    dao.save(customers);
}

score 3 · Accepted Answer · answered Oct 09 '18 at 17:44

Author of the library here. The BeanListProcessor has a strictHeaderValidationEnabled property you can set to true to ensure all headers in your class exist in the input.

You just can't use the CsvRoutines in that case as that class implements convenience methods that use their own internal row processors, so yours will be ignored. Try this code:

    CsvParserSettings parserSettings = new CsvParserSettings();
    parserSettings.getFormat().setLineSeparator("\n");

    final List<Customer> customers = new ArrayList<>();
    final java.util.Date stamp = getTimestamp(file);
    dao.checkTimestampDate(stamp);

    parserSettings.setProcessor(new BeanProcessor<Customer>() {
        @Override
        public void beanProcessed(Customer customer, ParsingContext context) {
            validateFileDateWithFileName(stamp, customer.getDay());
            validateCustomer(customer);
            customers.add(customer);
        }
    });

    new CsvParser(parserSettings).parse(file, "UTF-8");

    dao.save(customers);

Hope this helps.

Terran · Answer 2 · 2020-11-05T16:35:04.763

Based on answer by Jeronimo Backes.

In case you have @Header annotation on the bean or know the exact headers but still need setHeaderExtractionEnabled(true):

public <T> List<T> parse(File file, Class<T> beanType, char delimiter, Charset charset) {
    String[] headers = beanType.getDeclaredAnnotation(Headers.class).sequence(); // or other source

    CsvParserSettings parserSettings = Csv.parseRfc4180(); // or some other
    parserSettings.detectFormatAutomatically(delimiter);
    parserSettings.setHeaderExtractionEnabled(true);

    // initialize new processor (stateful, should not be reused! See implementation of parseAll)
    BeanListProcessor<T> processor= new BeanListProcessor<>();
    processor.setStrictHeaderValidationEnabled(true);
    parserSettings.setProcessor(processor);

    CsvParser csvParser = new CsvParser(parserSettings);

    csvParser.parse(file, charset);

    // header validation
    String[] headersParsed = processor.getHeaders();
    if (!Arrays.equals(headers, headersParsed)) {
        String message = String.format("Header validation failed. Expected: %s, but was: %s",
            Arrays.toString(headers), Arrays.toString(headersParsed));
        throw new DataProcessingException(message);
    }

    return beanListProcessor.getBeans();
}

How to validate CSV headers using Univocity routines?

2 Answers2