1

Since the file is too large to load into memory, I found a technique called incremental hash to iteratively hash a file stream. Are there any widely used hash function implementing this technique?

Also regardless of the exact hashing function, do web languages implement ways to hash a large file without executing external commands? My project uses PHP and Javascript but this question could be relevant to any settings.

Shukai Ni
  • 457
  • 2
  • 6
  • 17
  • Most hash function APIs I've used allow iterative submission of content to be hashed. Java supports hashing APIs for various hashing algorithms. – Pointy Jun 29 '22 at 17:03
  • in php you can open the file with fopen (https://www.php.net/manual/en/function.hash-file.php#122549), then feed it into hash_update after hash_init, nodejs can also do it just don't call the end method https://stackoverflow.com/questions/18658612/obtaining-the-hash-of-a-file-using-the-stream-capabilities-of-crypto-module-ie – Lawrence Cherone Jun 29 '22 at 17:55
  • @LawrenceCherone thanks, do you know if `hash_update_stream()` also takes chunk by chunk or by the entire file? – Shukai Ni Jun 30 '22 at 05:33
  • It seems `hash_update_stream()` also achieves incremental hash with buffer size = [1024](https://github.com/php/php-src/blob/bfe6f9e66a65d7c40fd486249097f932e2b237c3/ext/hash/hash.c#L725) – Shukai Ni Jun 30 '22 at 06:13

0 Answers0