Why does my program read only the first record in my files?

Question

I have written this program to compare 2 files. They are 500mb to 2.8gb in size and are created every 6 hours. I have 2 files from 2 sources (NMD and XMP). They are broken up into lines of text that have fields separated by the pipe(|) character. Each line is a single record and may be up to 65,000 characters long. The data is about TV shows and movies, showing times and descriptive content. I have determined that any particular show or movie has a minimum of 3 pieces of data that will uniquely identify that show or movie. IE: CallSign, ProgramId and StartLong. The two sources for this data are systems called NMD and XMP hence that acronym added to various variables. So my goal is to compare a file created by NMD and one created by XMP and confirm that everything that NMD produces is also produced by XMP and that the data in each matched record is the same.

What I am trying to accomplish here is this: 1. Read the NMD file record by record for the 3 unique data fields. 2. Read the XMP file record by record and look for a match for the current record in the NMD file. 3.The NMD file should iterate one record at a time. Each NMD record should then be searched for in the entire XMD file, record by record for that same record. 4. Write a log entry in one of 2 files indicating success or failure and what that data was.

What is happening is the first record in each file is read, but then no records after that get read. As a result, the end of neither file is ever reached and no matches are ever found. My success.log and failure.log file never show any data in them. In the outer do/while loop, System.out displays a single line of text. IE: The first record in the file. In the inner do/while loop System.out prints the same data over and over and over which is also from the first record in the file. Isn't this proof that the program isn't iterating record by record through the two source files?

So onto the actual code...

import java.io.*;

public class FileParse {

    public static void main(String[] args) throws java.io.IOException {

        String epgsRecordNMD = null;
        String epgsRecordXMP = null;
        BufferedWriter logSuccessWriter = null;
        BufferedWriter logFailureWriter = null;
        BufferedReader readXMP = null;
        BufferedReader readNMD = null;

        readNMD = new BufferedReader(new FileReader("d:testdataNMD.txt"));

            do {

                epgsRecordNMD = readNMD.readLine();
                String[] epgsSplitNMD = epgsRecordNMD.split("\\|");
                String epgsCallSignNMD = epgsSplitNMD[0];
                String epgsProgramIdNMD = epgsSplitNMD[2];
                String epgsStartLongNMD = epgsSplitNMD[9];
                System.out.println("epgsCallsignNMD: " + epgsCallSignNMD + " epgsProgramIdNMD: " + epgsProgramIdNMD + " epgsStartLongNMD: " + epgsStartLongNMD );


                    do {

                        readXMP = new BufferedReader(new FileReader("d:testdataXMP.txt"));
                        epgsRecordXMP = readXMP.readLine();
                        String[] epgsSplitXMP = epgsRecordXMP.split("\\|");
                        String epgsCallSignXMP = epgsSplitXMP[0];
                        String epgsProgramIdXMP = epgsSplitXMP[2];
                        String epgsStartLongXMP = epgsSplitXMP[9];
                        System.out.println("epgsCallsignXMP: " + epgsCallSignXMP + " epgsProgramIdXMP: " + epgsProgramIdXMP + " epgsStartLongXMP: " + epgsStartLongXMP);


                        if (epgsCallSignXMP.equals(epgsCallSignNMD) && epgsProgramIdXMP.equals(epgsProgramIdNMD) && epgsStartLongXMP.equals(epgsStartLongNMD)) {
                            logSuccessWriter = new BufferedWriter (new FileWriter("d:success.log", true));
                            logSuccessWriter.write("NMD match found in XMP" + "epgsCallsignNMD: " + epgsCallSignNMD + " epgsProgramIdNMD: " + epgsProgramIdNMD + " epgsStartLongNMD: " + epgsStartLongNMD);
                            logSuccessWriter.write("\n");
                            logSuccessWriter.close();
                            System.out.println ("Match found");

                        }

                    } while (epgsRecordXMP != null); 

                        logFailureWriter = new BufferedWriter (new FileWriter("d:failure.log", true));
                            logFailureWriter.write("NMD match not found in XMP" + "epgsCallsignNMD: " + epgsCallSignNMD + " epgsProgramIdNMD: " + epgsProgramIdNMD + " epgsStartLongNMD: " + epgsStartLongNMD);
                            logFailureWriter.write("\n");
                            logFailureWriter.close();
                            System.out.println ("Match NOT found");

           } while (epgsRecordNMD != null);
            readNMD.close();
            readXMP.close();

        }
    }

I noticed one thing that was a problem just now. Inside my second do/while loop I have this command: readXMP = new BufferedReader(new FileReader("d:testdataXMP.txt")); This causes the testdataXMP.txt file to reopen over and over, hence the records not iterating. I've moved this line just below the readNMD line. — JavaNoob, Jul 19 '17 at 20:08
Possible duplicate of [What is a debugger and how can it help me diagnose problems?](https://stackoverflow.com/questions/25385173/what-is-a-debugger-and-how-can-it-help-me-diagnose-problems) — Raedwald, Jul 20 '17 at 08:37

score 0 · Answer 1 · answered Jul 19 '17 at 20:08

0

   do {

     readXMP = new BufferedReader(new FileReader("d:testdataXMP.txt"));

You sure you want this in a loop? You are opening the same file over and over again.

answered Jul 19 '17 at 20:08

OldProgrammer

12,050
4
24
45

see what I just wrote...and no I don't – JavaNoob Jul 19 '17 at 20:16
Things are iterating through the records now in testdataXMP.txt, but when the last record is read, i get an NPE. Why doesn't the outer do/while loop iterate to the next record in the testdataNMD.txt file? The success.log now shows a match which I expected. The failure.log is still empty since the outer do/while loop isn't "looping". – JavaNoob Jul 19 '17 at 20:17
In both of my testdata files, I have records that won't match. I want to see those failures. – JavaNoob Jul 19 '17 at 20:20
I see another problem in my code. The inner do/while ends and then the logFailureWriter will put an entry in failure.log whether a successful match happens or not. How do I make my inner do/while iterate through completely and then if a success anywhere is made to get the logFailureWriter not run. This sounds like a variable greater than 0 needs to be checked. – JavaNoob Jul 19 '17 at 20:28
If you have a different problem now, please open a separate question, not add comments to existing question. – OldProgrammer Jul 19 '17 at 20:37
seriously? new thread for further help on the same code? How about if I make the title more generic? – JavaNoob Jul 19 '17 at 20:42
https://meta.stackoverflow.com/questions/266767/what-is-the-the-best-way-to-ask-follow-up-questions – OldProgrammer Jul 19 '17 at 20:54
I've started a new thread...just can't create it becasue of the 90 minute limit. This thread would have done just fine for my add-on comments. – JavaNoob Jul 19 '17 at 21:26

Bryan · Answer 2 · 2017-07-20T07:43:13.067

0

Why not do something like this?

BufferedReader br = new BufferedReader(new FileReader("whateverfile.dat"));
BufferedReader br2 = new BufferedReader(new FileReader("whateverfile2.dat"));
String data;
String data2;

while(data = br.readLine() != null)
{
    // do whatever you want, for example found 3 unique records

    while(data2 = br2.readLine() != null)
    {
        // do whatever you want
    }
}

Just like what OldProgrammer said, you're opening the file multiple times. You just need to ensure that the reader / streamer is open and iterate whatever you want then close it.

edited Jul 20 '17 at 07:43

answered Jul 19 '17 at 21:03

Bryan

8,488
14
52
78

I'm really new to Java...this is my 3rd program ever (including hello world)and I haven't written anything else in 27 years so feel free to correct me liberally...since I don't know any better. – JavaNoob Jul 19 '17 at 22:53
I tried while loops, but then I needed the readLine outside the loop to get a condition for the while statement and that lost me iterating through each record. So I went to a do/while instead since the readLine is now inside the loop. – JavaNoob Jul 19 '17 at 22:57
That second fileReader statement was staring me in the face, but I just couldn't see that it was in the wrong place until after I posted the thread. I must have stared at it and mentally gone through the steps in the program a hundred times and never caught it. – JavaNoob Jul 19 '17 at 22:58
I've updated the answer. You should never try to re-open something that is already open. Open it once, do whatever you want, then close it back. – Bryan Jul 20 '17 at 07:44

Why does my program read only the first record in my files?

2 Answers2