Reading a File's Content and then doing analysis upon them

Question

the project I am currently working on has me reading a file and then doing analysis upon the data inside the data. Using FileReader I have read each line of the file into an Array. The file looks like the following:

01 02 03 04 05 06
02 03 04 05 06 07
03 04 05 06 07 08
04 05 06 07 08 09

These are not the exact numbers but they make a good example. I am now trying to find out how many times say the number "04" appears in my list of data. I was thinking of putting all the data in a two dimensional array by breaking each line apart but I am not quite sure how to do this. Will I need a parser or is it possible to use some type of string function (like split) to split this data apart and then store it into the Array?

Answer 1

If you only need to count the 04's, you REALLY don't need to store the whole file. You could, for example, read each line and check it for 04's (and adding to a counter, or whatever). You could even just read character by character, but that might be a bit tedious for the slight (if any) efficiency gains.

If the processing you need to do on the file is more complex, this approach may not be up to the task. But unless you specify what that is, I can't say if it is or not.

Answer 2

You should use a Map to hold the count of occurrences, like so:

public static void main(String[] args) throws IOException {
Pattern splitter = Pattern.compile("\\s+");
try(Stream<String> stream = Files.lines(Paths.get("input.txt"))) {
    Map<String,Long> result = stream.flatMap(splitter::splitAsStream)
            .collect(Collectors.groupingBy(Function.identity(),
                    Collectors.counting()));
    System.out.println(result);
}}

Or load the data and parse it in multiple stages:

public static void main(String[] args) throws IOException {
    // 1. load the data array
    String[][] data;
    try(Stream<String> stream = Files.lines(Paths.get("numbers.txt"))) {
        data = stream.map(line -> line.split("\\s+")).toArray(String[][]::new);
    }
    System.out.format("Total lines = %d%n", data.length);

    // 2. count the occurrences of each word
    Map<String,Long> countDistinct = Arrays.stream(data).flatMap(Arrays::stream)
            .collect(Collectors.groupingBy(Function.identity(),
                    Collectors.counting()));
    System.out.println("Count of 04 = " + countDistinct.getOrDefault("04", 0L));

    // 3. calculate correlations 
    Map<String,Map<String,Long>> correlations;
    correlations = Arrays.stream(data).flatMap((String[] row) -> {
        Set<String> words = new HashSet<>(Arrays.asList(row));
        return words.stream().map(word -> new AbstractMap.SimpleEntry<>(word, words));
    }).collect(Collectors.toMap(kv -> kv.getKey(),
            kv -> kv.getValue().stream()
                    .collect(Collectors.toMap(Function.identity(), v -> 1L)),
            (map1, map2) -> {
                map2.entrySet().forEach(kv -> map1.merge(kv.getKey(), kv.getValue(), Long::sum));
                return map1;
            }));
    System.out.format("Lines with 04 = %d%n",           
        correlations.getOrDefault("04", Collections.EMPTY_MAP).getOrDefault("04", 0L));
    System.out.format("Lines with both 04 and 07 = %d%n",           
        correlations.getOrDefault("04", Collections.EMPTY_MAP).getOrDefault("07", 0L));
}

EDIT:

Here is a (perhaps) easier to read version that doesn't use a Stream/functional approach:

public static void main(String[] args) throws IOException {
    long lineCount = 0;
    Map<String,Long> wordCount = new HashMap<>();
    Map<String,Map<String,Long>> correlations = new HashMap<>();
    try(Stream<String> stream = Files.lines(Paths.get("numbers.txt"))) {
        Iterable<String> lines = stream::iterator;
        Set<String> lineWords = new HashSet<>();
        for(String line : lines) {
            lineCount++;
            for(String word : line.split("\\s+")) {
                lineWords.add(word);
                wordCount.merge(word, 1L, Long::sum);
            }
            for(String wordA : lineWords) {
                Map<String,Long> relate = correlations.computeIfAbsent(wordA,
                        key -> new HashMap<>());
                for(String wordB : lineWords) {
                    relate.merge(wordB, 1L, Long::sum);
                }
            }
        }
    }
    System.out.format("Total lines = %d%n", lineCount);
    System.out.println("Count of 04 = " + wordCount.getOrDefault("04", 0L));
    System.out.format("Lines with 04 = %d%n",           
        correlations.getOrDefault("04", Collections.EMPTY_MAP).getOrDefault("04", 0L));
    System.out.format("Lines with both 04 and 07 = %d%n",           
        correlations.getOrDefault("04", Collections.EMPTY_MAP).getOrDefault("07", 0L));
}

Output:

Total lines = 4

Count of 04 = 4

Lines with 04 = 4

Lines with both 04 and 07 = 3

Answer 3

Edit: I misread that you've already read the file into an array. So you could just skip to processing each entry in the array for the substrings.

Assuming you are using a text file or similar for the input, you can read the file line-by-line and count the number of "04" in each line as you read it. You could use a buffered reader like this:

String line;
while ((line = br.readLine()) != null) {
    //process each line
}

To count for the number of occurrences of your desired string, you can reference this other answer:

Occurrences of substring in a string

Answer 4

You are "premature" in your design ideas; for example about using a 2D array here.

You see, you really have to get a better understanding of your requirements before you start thinking about design/implementation choices.

Example: when you only care about measuring how often some number shows up overall then using a 2D array won't do any good. Instead, you could just put all numbers into one long List<Integer>; to then use some of the fancy java8 stream operations on that for example.

But if that was just one example out of many, then other ways to manage your data in memory might be more efficient.

Beyond that: if you find that the things you will be doing with this data goes beyond simple calculations - Java might probably not be the best choice here. You see, languages like R are specially designed to do just that: crunch incredible amounts of data; and giving you "instant" access to a wide range of statistical operations of all kind.

And to answer your idea about counting the occurrences of all the different numbers; that is really simple: you use a Map<Integer, Integer> here; like in:

Map<Integer, Integer> numbersWithCount = new HashMap<>();

now you loop over your data; and for each data point:

int currentNumber = ... next number from your input data

int counterForNum;
if (numbersWithCount.containsKey(currentNumber)) {
  counterForNum = numbersWithCount.get(currentNumber) + 1;
} else {
   // currentNumber found the first time
  counterForCurrentNumber = 1;
}
numbersWithCount.put(currentNumber);

In other words: you just iterate over all incoming numbers, and for each of those, you either create a new counter; or increase one that was already stored.

And if you use a TreeMap instead of a HashMap, you even get your keys sorted. Many possibilities there ...