0

if i put more than 5 excel files in a folder i getting this

java.lang.OutOfMemoryError: Java heap space.

Totally i have 40 excel files and each file may contain 5 to 10 sheets. my code will find out the errors in the excel files and write it down in the Error.txt file.

public class Hvd {

    public static int getExcelColumnNumber(String column) {
        int result = 0;
        for (int i = 0; i < column.length(); i++) {
            result *= 26;
            result += column.charAt(i) - 'A' + 1;
        }
        return result;
    }

    public static String getExcelColumnName(int number) {
        final StringBuilder sb = new StringBuilder();

        int num = number - 1;
        while (num >=  0) {
            int numChar = (num % 26)  + 65;
            sb.append((char)numChar);
            num = (num  / 26) - 1;
        }
        return sb.reverse().toString();
    }


    public static void main(String[] args) {
        try {

            //File directory = new File("C://Users//zkuwscr//Documents//new practice tests//Regression_Dev_Dec12th");
            File directory = new File("C://Users//kondeti.venkatarao//Documents//Regresion_sheets");
            File[] files = directory.listFiles();

            //File errorBW = new File("C://Users//zkuwscr//Documents//new practice tests//Regression_Dev_Dec12th//ErrorBW.txt");
            File errors = new File("C://Users//kondeti.venkatarao//Documents//Regresion_sheets//Error.txt");
            FileOutputStream errorsFileOutputStream = new FileOutputStream(errors);
            BufferedWriter errorsBufferedWriter = new BufferedWriter(new OutputStreamWriter(errorsFileOutputStream));

            File mismatch = new File("C://Users//kondeti.venkatarao//Documents//Regresion_sheets//Mismatch.txt");
            FileOutputStream mismatchFileOutputStream = new FileOutputStream(mismatch);
            BufferedWriter mismatchBufferedWriter = new BufferedWriter(new OutputStreamWriter(mismatchFileOutputStream));


            for (File file : files) {
                if (file.getName().endsWith(".xlsx")) {
                    //FileInputStream fis = new FileInputStream(file);
                    // Create Workbook instance holding reference to .xlsx file

                    OPCPackage pkg = OPCPackage.open(file);
                    XSSFWorkbook workbook = new XSSFWorkbook(pkg);
                    int i = 1;
                    while (i < workbook.getNumberOfSheets()) {

                        // System.out.println(workbook.getNumberOfSheets());
                        // Get first/desired sheet from the workbook
                        XSSFSheet sheet = workbook.getSheetAt(i);

                        // Iterate through each rows one by one
                        Iterator<Row> rowIterator = sheet.iterator();
                        while (rowIterator.hasNext()) {
                            Row row = rowIterator.next();
                            // For each row, iterate through all the columns
                            Iterator<Cell> cellIterator = row.cellIterator();
                            while (cellIterator.hasNext()) {
                                Cell cell = cellIterator.next();
                                // Check the cell type and format accordingly
                                switch (cell.getCellType()) {
                                /*
                                 * case Cell.CELL_TYPE_NUMERIC:
                                 * System.out.print(cell.getNumericCellValue());
                                 * break; case Cell.CELL_TYPE_STRING:
                                 * System.out.print(cell.getStringCellValue());
                                 * break;
                                 */
                                // case Cell.CELL_TYPE_FORMULA:
                                case Cell.CELL_TYPE_FORMULA:

                                    if (cell.getCellFormula().startsWith("IF("))
                                        if (sheet.getRow(row.getRowNum())
                                                .getCell(cell.getColumnIndex())
                                                .getRawValue().equals("1")) {

                                            HashSet<Integer> number= new HashSet<Integer>();
                                            ArrayList<String> alphas=new ArrayList<String>();

                                            String formula=sheet.getRow(row.getRowNum()).getCell(cell.getColumnIndex()).toString();
                                            Matcher digitMatcher = Pattern.compile("\\d+").matcher(formula);
                                            Matcher alphabetMatcher = Pattern.compile("[a-zA-Z]+").matcher(formula);

                                            while(alphabetMatcher.find()) {
                                                if(!alphabetMatcher.group().equals("TYPE"))
                                                    alphas.add(alphabetMatcher.group());
                                            }

                                            int countIF = Collections.frequency(alphas, "IF");
                                            int countABS = Collections.frequency(alphas, "ABS");
                                            HashSet<String> alphaSet=new HashSet<String>(alphas);
                                            if(countIF!=5 && countIF!=6)
                                                alphaSet.remove("IF");
                                            if(countABS != 3 && countABS!=4)
                                                alphaSet.remove("ABS");

                                            while(digitMatcher.find()) {
                                              if(!digitMatcher.group().equals("0") && !digitMatcher.group().equals("1") && !digitMatcher.group().equals("01"))
                                              number.add(Integer.parseInt(digitMatcher.group()));
                                            }


                                            ArrayList<Integer> numberList = new ArrayList<Integer>(number);
                                            ArrayList<String> alphaList = new ArrayList<String>(alphaSet);
                                            System.out.println("alphaSet"+alphaSet);
                                            System.out.println("numberList"+numberList);

                                            int rowIndex=numberList.get(0)-1;
                                            int originalColumnIndex = getExcelColumnNumber(alphaList.get(0))-1;
                                            int referenceColumnIndex = getExcelColumnNumber(alphaList.get(1))-1;
                                            if(originalColumnIndex > referenceColumnIndex){
                                                int temp = referenceColumnIndex;
                                                referenceColumnIndex =originalColumnIndex;
                                                originalColumnIndex=temp;
                                            }


                                            //System.out.println(sheet.getRow(row.getRowNum()));
                                            System.out.println("File Name: "+ file.getName());
                                            System.out.println("Sheet Name: "+ sheet.getSheetName());
                                            System.out.println(sheet.getRow(row.getRowNum()).getCell(cell.getColumnIndex()).toString());
                                            if(sheet.getRow(rowIndex).getCell(originalColumnIndex).getCellFormula().equals(""))
                                                System.out.println("please help me out");
                                            System.out.println("Function Name: "+ sheet.getRow(rowIndex).getCell(originalColumnIndex).getCellFormula());
                                            System.out.println("row indext"+rowIndex);
                                            System.out.println("original column index"+originalColumnIndex);
                                            System.out.println("ref column index"+referenceColumnIndex);
                                            /*System.out.println("File Name: "
                                                    + file.getName());
                                            System.out.println("Sheet Name: "
                                                    + sheet.getSheetName());
                                            System.out.println(cell
                                                    .getCellFormula());*/
                                            if(sheet.getRow(rowIndex).getCell(originalColumnIndex).getCellFormula().contains("qCRA_") && sheet.getRow(rowIndex).getCell(originalColumnIndex).getRawValue().contains("Error:")){
                                                errorsBufferedWriter.newLine();
                                                errorsBufferedWriter.write("File Name: "+ file.getName());
                                                errorsBufferedWriter.newLine();
                                                errorsBufferedWriter.write("Sheet Name: "+ sheet.getSheetName());
                                                errorsBufferedWriter.newLine();
                                                errorsBufferedWriter.write("Function Name: "+ sheet.getRow(rowIndex).getCell(originalColumnIndex).getCellFormula());
                                                errorsBufferedWriter.newLine();
                                                errorsBufferedWriter.write("Cell Number: "+getExcelColumnName(originalColumnIndex+1)+numberList.get(0));
                                                errorsBufferedWriter.newLine();
                                                errorsBufferedWriter.write("Orginal Value : "+sheet.getRow(rowIndex).getCell(originalColumnIndex).getRawValue());
                                                errorsBufferedWriter.newLine();
                                                errorsBufferedWriter.write("Reference Value : "+sheet.getRow(rowIndex).getCell(referenceColumnIndex));
                                                errorsBufferedWriter.newLine();
                                            } else {
                                                mismatchBufferedWriter.newLine();
                                                mismatchBufferedWriter.write("File Name: "+ file.getName());
                                                mismatchBufferedWriter.newLine();
                                                mismatchBufferedWriter.write("Sheet Name: "+ sheet.getSheetName());
                                                mismatchBufferedWriter.newLine();
                                                mismatchBufferedWriter.write("Function Name: "+ sheet.getRow(rowIndex).getCell(originalColumnIndex).getCellFormula());
                                                mismatchBufferedWriter.newLine();
                                                mismatchBufferedWriter.write("Cell Number: "+getExcelColumnName(originalColumnIndex+1)+numberList.get(0));
                                                mismatchBufferedWriter.newLine();
                                                mismatchBufferedWriter.write("Orginal Value : "+sheet.getRow(rowIndex).getCell(originalColumnIndex).getRawValue());
                                                mismatchBufferedWriter.newLine();
                                                mismatchBufferedWriter.write("Reference Value : "+sheet.getRow(rowIndex).getCell(referenceColumnIndex));
                                                mismatchBufferedWriter.newLine();
                                            }
                                        }
                                    break;

                                }
                                cell=null;
                            }
                            row=null;
                        }
                        i++;
                        //fis.close();
                        pkg.close();
                        sheet=null;
                    }
                    workbook=null;
                }
            }
            errorsBufferedWriter.close();
            errorsFileOutputStream.close();
        } catch (Exception e) {
            e.printStackTrace();
        }
    }
}
Venkat Kondeti
  • 81
  • 1
  • 12
  • 3
    Please don't use StringBuffer. StringBuilder has been a drop in replacement for ten years now. – Peter Lawrey Dec 15 '14 at 11:59
  • 1
    You don't need to escape forward slash, just use `c:/Users/` – Peter Lawrey Dec 15 '14 at 12:00
  • Have you tried to follow the answer from this question http://stackoverflow.com/questions/2381849/exception-in-thread-main-java-lang-outofmemoryerror-java-heap-space ? – reto Dec 15 '14 at 12:00
  • 1
    If you only get a problem after reading multiple files, you might have a memory leak, I suggest you use a memory profiler to see where memory is being retained. You can start with VisualVM which is built into Java. – Peter Lawrey Dec 15 '14 at 12:02
  • @PeterLawrey how to use VisualVM ? Could you please point me to a link, where i can read and apply to my program ? – Venkat Kondeti Dec 15 '14 at 12:09
  • You could start with http://visualvm.java.net/gettingstarted.html and http://visualvm.java.net/profiler.html – Peter Lawrey Dec 15 '14 at 12:13
  • 1
    @PeterLawrey those are the first two hits in Google you got when you literally post the question "how to use VisualVM" in it, right? – Gimby Dec 15 '14 at 13:37
  • @PeterLawrey I launched my application using VisualVM, there im only able to see the heap graph. i want to know exactly in which part of code is occupying the more memory? Please help me – Venkat Kondeti Dec 16 '14 at 07:45
  • There is a tab where you instrument memory profiling. – Peter Lawrey Dec 16 '14 at 08:35
  • could you please look at the below the @Axel comments. – Venkat Kondeti Dec 16 '14 at 11:47

1 Answers1

1
  1. If you get OOM, use -Xmx... to increase heap size.
  2. Throw out your StringBuffers and write directly to files (BufferedWriter or even PrintStream). No need to keep all the output in memory. This alone might solve your problem.
Axel
  • 13,939
  • 5
  • 50
  • 79
  • now i have used BufferedWriter but still getting Exception in thread "main" java.lang.OutOfMemoryError: Java heap space. I completely removed StringBuffer/StringBuilder. – Venkat Kondeti Dec 16 '14 at 09:24
  • Axel @Axel i have updated the question with new code. – Venkat Kondeti Dec 16 '14 at 09:56
  • You shouldn't need the `workbook=null` and `sheet=null` statements, but looking at the apache POI documentation, you might need a call to `workbook.close()`. BTW how large are the Excel-files? And have you tried passing `-Xmx1024m` (or even higher) to the JVM? – Axel Dec 16 '14 at 10:17
  • PS: `FileInputStream fis = new FileInputStream(file.getAbsolutePath())` should rather be `FileInputStream fis = new FileInputStream(file)`. – Axel Dec 16 '14 at 10:22
  • we dont have such a function workbook.close();. And almost all the files are less than 300KB and one file is 3441KB. Yes i tried passed -Xmx1024m. – Venkat Kondeti Dec 16 '14 at 10:25
  • Which version of POI are you using? I found it here: http://poi.apache.org/apidocs/org/apache/poi/xssf/usermodel/XSSFWorkbook.html – Axel Dec 16 '14 at 10:31
  • i have edited my code. Now im using OPCPackage pkg = OPCPackage.open(file); XSSFWorkbook workbook = new XSSFWorkbook(pkg); now im getting different error org.apache.poi.openxml4j.exceptions.OpenXML4JRuntimeException: Fail to save: an error occurs while saving the package : Zip File is closed – Venkat Kondeti Dec 16 '14 at 10:49
  • could you please look at the above comment ? – Venkat Kondeti Dec 16 '14 at 11:40