When Will Storage Requirements Exceed RAM in a System that Constantly Computes Higher Values and Stores Them on the Disk?

Question

I'm working on a project that can generate an infinite amount of numbers <More on this>: Threading Isues With Fixed Thread Pool and Large Number of Tasks and I've been successful in solving the threading issues involved with it. Also, I've decided to store the results of the sequencer on the hard disk so I can compute more numbers (and make sure that they all converge to 1), I do this every time the HashMap that is storing the results reaches 400,000 (a reasonable size on my computer, about 1GB/8GB) key/value pairs, then load in the next 400,000. Assuming that the values are FinalSequencerReports, as the values get larger and larger tending towards ∞ the values stored should eventually exceed my RAM capacity, is there any way to get around this limitation (or even at what number I'll even encounter a situation where storing 400,000 instances will exceed my RAM capacity? Or a formula for any RAM capacity?) NOTE: inside the FinalSequencerReport class there is a String, it will always be empty.

Class being stored in the HashMap<BigInteger, FinalSequencerReport<BigInteger>>:

public static final class FinalSequencerReport<T extends Number> extends SequencerReport<T> implements Comparable<FinalSequencerReport<T>> {

    public static Comparator<? super FinalSequencerReport<? extends Number>> compareByInitialValue() {
        return (FinalSequencerReport<? extends Number> o1, FinalSequencerReport<? extends Number> o2)
                -> new BigInteger(o1.getInitialValue().toString()).compareTo(new BigInteger(o2.getInitialValue().toString()));
    }
    private final T initialValue;
    private final String finalFormattedString;

    public FinalSequencerReport(SequencerReport<T> finalReport, T initialValue) {
        super(finalReport.getResult(), finalReport.getIterations(), finalReport.getSequence());
        this.initialValue = initialValue;
        this.finalFormattedString = "Initial Value: "
                + getInitialValue() + "\nFinal Value: " + getResult() + "\nIterations:  "
                + getIterations() + "\nAlgebraic Sequence:\n" + getSequence();
    }

    public String getFinalFormattedString() {
        return this.finalFormattedString;
    }

    public Number getInitialValue() {
        return this.initialValue;
    }

    @Override
    public int compareTo(FinalSequencerReport<T> o) {
        return FinalSequencerReport.compareByInitialValue().compare(this, o);
    }
}

public static class SequencerReport<T extends Number> {

    private final T result, iterations;
    private final String sequence;

    public SequencerReport(T result, T iterations) {
        this(result, iterations, "");
    }

    public SequencerReport(T result, T iterations, String sequence) {
        this.result = result;
        this.iterations = iterations;
        this.sequence = sequence;
    }

    public T getResult() {
        return this.result;
    }

    public T getIterations() {
        return this.iterations;
    }

    public String getSequence() {
        return this.sequence;
    }

}

Answer 1

you will need to periodically check the available HDD space, paired with a RAM check - i recommend checking the RAM much more often than the HDD.

If you want to estimate space-usage, you could approximate the size of your objects, but beware : as the internal representation of objects may change at any time you might get incorrect results, this is implementation-specific, object-sizes cant be accurately measured in java, AFAIK

Also note that the upper limit of available RAM always depends on your JVM-settings, the Oracle-JVM setting can be altered via -Xmx, possibles suffixes include m and g, example : -Xmx2g - dont fiddle with -Xms since it can seriously slow down everything if not produce weird crashes because you're effectively circumventing every JVM-optimization

free filesystem space & available RAM:

 /* Total number of processors or cores available to the JVM */
  System.out.println("Available processors (cores): " + 
  Runtime.getRuntime().availableProcessors());

  /* Total amount of free memory available to the JVM */
  System.out.println("Free memory (bytes): " + 
  Runtime.getRuntime().freeMemory());

  /* This will return Long.MAX_VALUE if there is no preset limit */
  long maxMemory = Runtime.getRuntime().maxMemory();
  /* Maximum amount of memory the JVM will attempt to use */
  System.out.println("Maximum memory (bytes): " + 
  (maxMemory == Long.MAX_VALUE ? "no limit" : maxMemory));

  /* Total memory currently in use by the JVM */
  System.out.println("Total memory (bytes): " + 
  Runtime.getRuntime().totalMemory());

  /* Get a list of all filesystem roots on this system */
  File[] roots = File.listRoots();

  /* For each filesystem root, print some info */
  for (File root : roots) {
    System.out.println("File system root: " + root.getAbsolutePath());
    System.out.println("Total space (bytes): " + root.getTotalSpace());
    System.out.println("Free space (bytes): " + root.getFreeSpace());
    System.out.println("Usable space (bytes): " + root.getUsableSpace());
  }

object size approximation :

import java.lang.instrument.Instrumentation;

public class ObjectSizeFetcher {
    private static Instrumentation instrumentation;

    public static void premain(String args, Instrumentation inst) {
        instrumentation = inst;
    }

    public static long getObjectSize(Object o) {
        return instrumentation.getObjectSize(o);
    }
}

public class C {
    private int x;
    private int y;

    public static void main(String [] args) {
        System.out.println(ObjectSizeFetcher.getObjectSize(new C()));
    }
}