I cannot answer your question directly, as I'm not using Java; however I can share a similar experience in Perl that may be partially relevant.
The OOXML format is indeed very large and complex, so any software that aims at covering the full specification is likely to be quite costly in terms of resources. In Perl, the most well-known module for reading .xlsx
files is https://metacpan.org/pod/Spreadsheet::ParseXLSX, which does the job well for small and medium files; however it is far too slow on large amounts of data. So I ended up writing another module https://metacpan.org/pod/Excel::ValueReader::XLSX, with far less features, but optimized for fast parsing of large files.
The moral is : there is no one-size-fits-all solution. If you are willing to sacrifice some features for better speed or less memory consumption, you might find other libraries. In Java, https://github.com/dhatim/fastexcel could perhaps be a good candidate (just from reading the documentation).