1

I have performance issues on a batch which is parsing big xml files in order to insert objects linked by foreign key in the database (Postgresql 9.1). The insertion of one record takes 16ms and the target of the batch will be to insert about 8 millions of rows, do the math... (i'm using threads, dividing the duration but it's not enough)

At the moment, the sequence is

  • parsing xml, creating a bean "Piece" and all his children
  • insert the bean Piece

    public Piece insert(Piece piece)    {
    final MapSqlParameterSource paramMap = new MapSqlParameterSource();
    final KeyHolder generatedKeyHolder = new GeneratedKeyHolder();
    String[] keyColumnNames = new String[1];
    keyColumnNames[0] = HeraConstantes.OID_PIECE;
    
    
    String sql = UtilsBatchSql.REQ_INSERT_PIECE;
    
    NamedParameterJdbcTemplate parameterJdbcTemplate = new
        NamedParameterJdbcTemplate(getSimpleJdbcTemplate().getJdbcOperations());
    
    
    paramMap.addValue(HeraConstantes.PCE_NUM_PIECE, piece.getNumeroPiece(), Types.BIGINT);
    paramMap.addValue(HeraConstantes.PCE_TYPE_PIECE, piece.getTypePiece(), Types.VARCHAR);
    paramMap.addValue(HeraConstantes.PCE_ID_BORD, piece.getIdBordereau(), Types.BIGINT);
    paramMap.addValue(HeraConstantes.PCE_EXE_RATTACH, piece.getExerciceRattachement(), Types.INTEGER);
    paramMap.addValue(HeraConstantes.PCE_DT_EMIS, piece.getDateEmission(), Types.DATE);
    paramMap.addValue(HeraConstantes.BUDGET_OID_FROM_RAR_PIECES, piece.getBudgetCollectiviteOID(), Types.BIGINT);
    
    try
    {
        parameterJdbcTemplate.update(sql, paramMap, generatedKeyHolder, keyColumnNames);
    
    }
    catch (DataAccessException e)
    {
        log.error(e.getMessage());
        throw new GerableException(e.getMessage());
    }
    piece.setPieceOID(generatedKeyHolder.getKey().longValue());
    
    
    
    return piece;}
    
  • I create another bean called "Line"

  • do line.setPieceOID(piece.getOid())
  • insert the bean Line (same way as Piece insertion)
  • and so on, up to 4 different levels (each need the primary key of the previous insert to set the foreign key).

So, is there a way to insert cluster of object using an only transaction, avoiding to connect 5 times to server ?

Maybe i'm missing the way to insert with a batchUpdate ?

Any earned millisecond would be appreciated.

saka
  • 49
  • 5
  • Parse the XML and save to CSV and use COPY FROM – e4c5 Nov 23 '16 at 09:24
  • is there a particular reason JDBC has to be used because i think it doesnot have batch query support ? Check out http://stackoverflow.com/questions/9565481/how-to-do-multiple-inserts-in-database-using-spring-jdbc-template-batch. If your fine you can move to JPA which will have the support and will optimize the process. – Goro Nov 23 '16 at 09:35
  • @e4c5 how could i get generatedkey with this method ? – saka Nov 23 '16 at 09:40

0 Answers0