I have performance issues on a batch which is parsing big xml files in order to insert objects linked by foreign key in the database (Postgresql 9.1). The insertion of one record takes 16ms and the target of the batch will be to insert about 8 millions of rows, do the math... (i'm using threads, dividing the duration but it's not enough)
At the moment, the sequence is
- parsing xml, creating a bean "Piece" and all his children
insert the bean Piece
public Piece insert(Piece piece) { final MapSqlParameterSource paramMap = new MapSqlParameterSource(); final KeyHolder generatedKeyHolder = new GeneratedKeyHolder(); String[] keyColumnNames = new String[1]; keyColumnNames[0] = HeraConstantes.OID_PIECE; String sql = UtilsBatchSql.REQ_INSERT_PIECE; NamedParameterJdbcTemplate parameterJdbcTemplate = new NamedParameterJdbcTemplate(getSimpleJdbcTemplate().getJdbcOperations()); paramMap.addValue(HeraConstantes.PCE_NUM_PIECE, piece.getNumeroPiece(), Types.BIGINT); paramMap.addValue(HeraConstantes.PCE_TYPE_PIECE, piece.getTypePiece(), Types.VARCHAR); paramMap.addValue(HeraConstantes.PCE_ID_BORD, piece.getIdBordereau(), Types.BIGINT); paramMap.addValue(HeraConstantes.PCE_EXE_RATTACH, piece.getExerciceRattachement(), Types.INTEGER); paramMap.addValue(HeraConstantes.PCE_DT_EMIS, piece.getDateEmission(), Types.DATE); paramMap.addValue(HeraConstantes.BUDGET_OID_FROM_RAR_PIECES, piece.getBudgetCollectiviteOID(), Types.BIGINT); try { parameterJdbcTemplate.update(sql, paramMap, generatedKeyHolder, keyColumnNames); } catch (DataAccessException e) { log.error(e.getMessage()); throw new GerableException(e.getMessage()); } piece.setPieceOID(generatedKeyHolder.getKey().longValue()); return piece;}
I create another bean called "Line"
- do line.setPieceOID(piece.getOid())
- insert the bean Line (same way as Piece insertion)
- and so on, up to 4 different levels (each need the primary key of the previous insert to set the foreign key).
So, is there a way to insert cluster of object using an only transaction, avoiding to connect 5 times to server ?
Maybe i'm missing the way to insert with a batchUpdate ?
Any earned millisecond would be appreciated.