Generating millions of random string in Java

Question

I would like to generate millions of passwords randomly between 4 millions and 50 millions. The problem is the time it takes the processor to process it.

I would like to know if there is a solution to generate a lot of passwords in only a few seconds (max 1 minute for 50 millions).

I've done that for now but it takes me more than 3 min (with a very good config and I would like to run it on small config).

private final static String policy = "azertyuiopqsdfghjklmwxcvbnAZERTYUIOPQSDFGHJKLMWXCVBN1234567890";
    private static List<String> names = new ArrayList<String>();
    
    
    public static void main(String[] args) {
        names.add("de");
        init();
    }
    
    
    
    private static String generator(){
        String password="";
        int randomWithMathRandom = (int) ((Math.random() * ( - 6)) + 6);
        for(var i=0;i<8;i++){
            randomWithMathRandom = (int) ((Math.random() * ( - 6)) + 6);
            password+= policy.charAt(randomWithMathRandom);
        }
        return password;
    }
    
    public static void init() {
        for (int i = 0; i < 40000000; i++) {
            names.add(generator());     
        }
    }

btw I can't take a ready-made list. I think the most 'expensive' waste of time is the input into the list.

My current config : ryzen 7 4800h rtx 2600 SSD NVME RAM 3200MHZ

UPDATE : I tried with 20Millions and it's display an error: java.lang.OutOfMemoryError thrown from the UncaughtExceptionHandler in thread "main"

Answer 1

Storing 50 million passwords as Strings in-memory could cause problems since either the stack or the heap may overflow. From this point of view, I think the best we can do is to generate a chunk of passwords, store them in a file, generate the next chunk, append them to the file... until the desired amount of passwords is created. I hacked together a small program that generates random Strings of length 32. As alphabet, I used all ASCII-characters between '!' (ASCII-value 33) and '~' (ASCII-value 126).

import java.io.IOException;
import java.nio.file.Files;
import java.nio.file.Path;
import java.nio.file.StandardOpenOption;
import java.text.DecimalFormat;
import java.text.DecimalFormatSymbols;
import java.util.Random;
import java.util.concurrent.TimeUnit;

class Scratch {
  private static final int MIN = '!';
  private static final int MAX = '~';
  private static final Random RANDOM = new Random();
  
  public static void main(final String... args) throws IOException {
    final Path passwordFile = Path.of("passwords.txt");
    if (!Files.exists(passwordFile)) {
      Files.createFile(passwordFile);
    }
    final DecimalFormat df = new DecimalFormat();
    final DecimalFormatSymbols ds = df.getDecimalFormatSymbols();
    ds.setGroupingSeparator('_');
    df.setDecimalFormatSymbols(ds);
    final int numberOfPasswordsToGenerate = 50_000_000;
    final int chunkSize = 1_000_000;
    final int passwordLength = 32;
    int generated = 0;
    int chunk = 0;
    final long start = System.nanoTime();
    while (generated < numberOfPasswordsToGenerate) {
      final StringBuilder passwords = new StringBuilder();
      for (
          int index = chunk * chunkSize;
          index < (chunk + 1) * chunkSize && index < numberOfPasswordsToGenerate;
          ++index) {
        final StringBuilder password = new StringBuilder();
        for (int character = 0; character < passwordLength; ++character) {
          password.append(fetchRandomLetterFromAlphabet());
        }
        passwords.append(password.toString()).append(System.lineSeparator());
        ++generated;
        if (generated % 500_000 == 0) {
          System.out.printf(
              "%s / %s%n",
              df.format(generated),
              df.format(numberOfPasswordsToGenerate));
        }
      }
      ++chunk;
      Files.writeString(passwordFile, passwords.toString(), StandardOpenOption.APPEND);
    }
    final long consumed = System.nanoTime() - start;
    System.out.printf("Done. Took %d seconds%n", TimeUnit.NANOSECONDS.toSeconds(consumed));
  }

  private static char fetchRandomLetterFromAlphabet() {
    return (char) (RANDOM.nextInt(MAX - MIN + 1) + MIN);
  }
}

On my laptop, the program yields good results. It completes in about 33 seconds and all passwords are stored in a single file.

This program is a proof of concept and not production-ready. For example, if a password.txt does already exist, the content will be appended. For me, the file already has 1.7 GB after one run, so be aware of this. Furthermore, the generated passwords are temporarily stored in a StringBuilder, which may present a security risk since a StringBuilder cannot be cleared (i.e. its internal memory structured cannot be zeroed). Performance could further be improved by running the password generation multi-threaded, but I will leave this as an exercise to the reader.

To use the alphabet presented in the question, we can remove static fields MIN and MAX, define one new static field private static final char[] ALPHABET = "azertyuiopqsdfghjklmwxcvbnAZERTYUIOPQSDFGHJKLMWXCVBN1234567890".toCharArray(); and re-implement fetchRandomLetterFromAlphabet as:

  private static char fetchRandomLetterFromAlphabet() {
    return ALPHABET[RANDOM.nextInt(ALPHABET.length)];
  }

We can use the following code-snippet to read-back the n-th (starting at 0) password from the file in constant time:

final int n = ...;
final RandomAccessFile raf = new RandomAccessFile(passwordFile.toString(), "r");
final long start = System.nanoTime();
final byte[] bytes = new byte[passwordLength];

// byte-length of the first n passwords, including line breaks:
final int offset = (passwordLength + System.lineSeparator().toCharArray().length) * n;

raf.seek(offset); // skip the first n passwords
raf.read(bytes);

// reset to the beginning of the file, in case we want to read more passwords later:
raf.seek(0); 

System.out.println(new String(bytes));

Answer 2

I can give you some tips to optimize your code and make it faster, you can use them along with other.

1. If you know the number of passwords you need, you should create a string array and fill it with the variable in your loop.
2. If you have to use a dynamic size data structure, use linked list. Linked list is better than array list when adding elements is your main target, and worse if you want to access them more then add them.
3. Use string builder instead of += operator on strings. The += operator is very 'expensive' in complexity of time, because it always creates new strings. Using string builder append method can speed up your code.
4. Instead of using Math.random() and multiple the result to your range number, create a static Random object and use yourRandomInstance.next(int range).
5. Consider use the ascii table to get random character instead of using str.charAt(int index) method, it may speed up your code too, i offer you to check it.