I am developing a system in Java which reads all the MySQL database tables, performs some operation and finally writes all data into a file (separate file for each table).
Since all the database table have different number of columns and different number of rows, there can be memory issue if the data is higher than our system can handle. Therefore, I need to write code that reads the tables values block by block and writes that data into file; and after some iterating all the data are written into that file.
I believe this approach would run in any system with any RAM size so that this system works without running into memory issues. Currently, for any table I am limiting the query result and writing that result in a one file, and iterating this process over and over until all the results are not processed. Here the value of limit size and number of iteration for all tables are dynamic, i.e. depends upon number of rows, columns and RAM size.
Following is the code written so far.
public static void main(String[] args) throws Exception {
List<String> dbList = MySqlUtils.getAllTableNames("datahouse");
for (String tableName : dbList) {
processTable(tableName);
}
}
public static void processTable(String tableName) throws Exception {
String dbname = "datahouse";
int startIndex = 0;
int limit = getMySqlQueryLimit(dbname, tableName);
int endIndex = limit;
int iteratorLength = getIteratorLength(dbname, tableName);
for (int i = 1; i <= iteratorLength; i++) {
ResultSet resultSet = getResultSet(tableName, startIndex, endIndex);
while (resultSet.next()) {
// Write into file after some operation
}
startIndex = endIndex;
endIndex += limit;
}
}
public static ResultSet getResultSet(String tableName, int startLimit, int endLimit) throws SQLException {
StringBuilder builder = new StringBuilder();
builder.append("SELECT * FROM " + tableName);
builder.append("ORDER BY id ASC limit (");
builder.append(startLimit);
builder.append(",");
builder.append(endLimit);
builder.append(")");
return MySqlUtils.getStatement().executeQuery(builder.toString());
}
public static int getMySqlQueryLimit(String dbName, String tableName) throws SQLException {
long ramSize = SystemUtils.getPhysicalMemorySize();
int columnSize = getColumnCount(dbName, tableName);
int totalRows = getRowsCount(dbName, tableName);
//TODO
return 0;
}
public static int getIteratorLength(String dbName, String tableName) {
try {
long ramSize = SystemUtils.getPhysicalMemorySize();
int columnSize = getColumnCount(dbName, tableName);
int totalRows = getRowsCount(dbName, tableName);
//TODO
return 0;
} catch (SQLException e) {
e.printStackTrace();
return 0;
}
}
In processTable()
method, there is a dependency between limit
and iteratorLength
. Is there any algorithm (or any mathematical formula) that can calculate the values for getMySqlQueryLimit()
and getIteratorLength()
, so that this code can be executed in any of the system independent of RAM size i.e. without running into memory issue?