0

I'm have created a java app which transfers files from FTP location to HDFS. This is done using copying the file first to temp folder in the edge node then move it to HDFS (cloudera cluster). Currently I'm doing byte check on the file - I'm comparing total file size in bytes of the FTP file and the copied HDFS file. I want to implement checksum. I have googled but I couldn't get any answers. Please let me know if you any ideas. (FTP server is a Windows machine where we enabled FTP)

Sidharthan PV
  • 38
  • 5
  • Standard java [CRC32](https://docs.oracle.com/javase/8/docs/api/java/util/zip/CRC32.html) – Joop Eggen Oct 04 '19 at 11:11
  • @MartinPrikryl, thanks for the link to the answer. Actually I need to get the checksum of the file which I transferred to HDFS(hadoop file system) and compare it with the FTP checksum to verify the transfer is correct! – Sidharthan PV Oct 06 '19 at 11:26
  • @user3600673 I understand. That's why I've posted the above link. It shows, that you will have troubles calculating the checksum of the file on the FTP server. – Martin Prikryl Oct 06 '19 at 18:57

1 Answers1

0

If you want to calculate checksum, I think that the easiest solution is to use Apache Common Codec.

String sha256hex = org.apache.commons.codec.digest.DigestUtils.sha256Hex(stringText);

Another alternative is Guava which has an easy-to-use suite of Hashing utilities. For example, to hash a string using SHA256 as a hex-string you would simply do:

final String hashed = Hashing.sha256()
        .hashString("your input", StandardCharsets.UTF_8)
        .toString();
wolfdale
  • 1
  • 1
  • 2