How to retrieve data in chunks using Pagination(Limit/offset) from a select query in this following example code?

Question

I have an select sql query statement that returns 900000+ rows.But i want data to be retrieved in small chunks using limit/offset clause,so that it can be easily inserted into another table.

public class Snomedinfo {
    public void snomedinfoinsert()
    {
    Root oRoot = null;
    ResultSet oRsSelect = null;
    PreparedStatement oPrStmt = null;
    PreparedStatement oPrStmt2 = null;
    PreparedStatement oPrStmtSelect = null;
    String strSql = null;
    int count=0;
    int batchSize=1000;

    ResultSet oRs = null;
    String refid = null;
    String id = null;
    String effectivetime = null;
    String active = null;
    String moduleid = null;
    String conceptid = null;
    String languagecode = null;
    String typeid = null;
    String term = null;
    String caseSignificanceid = null;

    try{
    oRoot = Root.createDbConnection(null);

    strSql = "SELECT d.* FROM snomed_conceptdata c, snomed_descriptiondata d WHERE c.active = 1 and conceptid = c.id AND d.active = 1 ";//this is the query that returns 900000 rows
    oPrStmt2 = oRoot.con.prepareStatement(strSql);

    oRs = oPrStmt2.executeQuery();
    String sql = "INSERT INTO snomedinfo_date (refid,id,effectivetime,active,moduleid,conceptid,languagecode,typeid,term,caseSignificanceid) VALUES( ?, ?, ?,?,?,?,?,?,?,?)";
    oPrStmt = oRoot.con.prepareStatement(sql);

        while (oRs.next())
        {
            refid = Root.TrimString(oRs.getString("refid")); 
            id = Root.TrimString(oRs.getString("id"));
            effectivetime = Root.TrimString(oRs.getString("effectivetime"));
            active = Root.TrimString(oRs.getString("active"));
            moduleid = Root.TrimString(oRs.getString("moduleid"));
            conceptid = Root.TrimString(oRs.getString("conceptid"));
            languagecode = Root.TrimString(oRs.getString("languagecode"));
            typeid = Root.TrimString(oRs.getString("typeid"));
            term = Root.TrimString(oRs.getString("term"));
            caseSignificanceid = Root.TrimString(oRs.getString("caseSignificanceid"));  

            oPrStmt.setString(1, refid);
            oPrStmt.setString(2, id);
            oPrStmt.setString(3, effectivetime);
            oPrStmt.setString(4, active);
            oPrStmt.setString(5, moduleid);
            oPrStmt.setString(6, conceptid);
            oPrStmt.setString(7, languagecode);
            oPrStmt.setString(8, typeid );
            oPrStmt.setString(9, term);
            oPrStmt.setString(10, caseSignificanceid);
            oPrStmt.addBatch();
            if (++count % batchSize == 0) {
                oPrStmt.executeBatch();
                oPrStmt.clearBatch();
            }

        }


        oPrStmt.executeBatch();


    System.out.println("done");
    }

    catch (Exception e) {
        e.printStackTrace();

    }

    finally {


        oRs = Root.EcwCloseResultSet(oRs);
        oPrStmt = Root.EcwClosePreparedStatement(oPrStmt);
        oPrStmt = Root.EcwClosePreparedStatement(oPrStmt2);

        oRoot = Root.closeDbConnection(null, oRoot);
    }
}

How do i implement the logic for setting limit/offset in my program ?? I have an select sql query statement that returns 900000+ rows.But i want data to be retrieved in small chunks using limit/offset clause,so that it can be easily inserted into another table.

score 0 · Answer 1 · answered Aug 14 '18 at 16:17

SELECT d.* FROM snomed_conceptdata c, snomed_descriptiondata d WHERE c.active = 1 and conceptid = c.id AND d.active = 1 limit offset,limit;

Offset - The row from which you want to start reading data.

Limit - The number of rows you want to read.

For example- SELECT d.* FROM snomed_conceptdata c, snomed_descriptiondata d WHERE c.active = 1 and conceptid = c.id AND d.active = 1 limit 10,20;

If you are using this in a web page then you need pagination.

Otherwise use a simple loop like this -

for(int i=0;i<=(90000/100);i+=10){

strSql = " SELECT d.* FROM snomed_conceptdata c, snomed_descriptiondata d WHERE c.active = 1 and conceptid = c.id AND d.active = 1 limit "+i+",10";

// run other code

}

This offsets the pointer by 10 rows with each iteration.

How to retrieve data in chunks using Pagination(Limit/offset) from a select query in this following example code?

1 Answers1