How do I get a Readable File?

Question

I have a directory filled with 99 files, I want to read these files and then hash them into a sha256 checksum. I eventually want to output them to a JSON file with a key-value pair so for example (File 1, 092180x0123). Currently I am having trouble passing my ParDo function a readable File I must be missing something very easy. This is my first time using Apache beam so any help would be amazing. Here is what I have so far

public class BeamPipeline {

    public static void main(String[] args)  {

        PipelineOptions options = PipelineOptionsFactory.create();
        Pipeline p = Pipeline.create(options);

            p
            .apply("Match Files", FileIO.match().filepattern("../testdata/input-*"))
            .apply("Read Files", FileIO.readMatches())
            .apply("Hash File",ParDo.of(new DoFn<FileIO.ReadableFile, KV<FileIO.ReadableFile, String>>() {
        @ProcessElement
        public void processElement(@Element FileIO.ReadableFile file, OutputReceiver<KV<FileIO.ReadableFile, String>> out) throws
        NoSuchAlgorithmException, IOException {
            // File -> Bytes
            String strfile = file.toString();
            byte[] byteFile = strfile.getBytes();


            // SHA-256
            MessageDigest md = MessageDigest.getInstance("SHA-256");
            byte[] messageDigest = md.digest(byteFile);
            BigInteger no = new BigInteger(1, messageDigest);
            String hashtext = no.toString(16);
            while(hashtext.length() < 32) {
                hashtext = "0" + hashtext;
            }
            out.output(KV.of(file, hashtext));
        }
    }))
            .apply(FileIO.write());
        p.run();
    }
}

Guillem Xercavins · Accepted Answer · 2019-04-21T20:14:28.230

One example to have a KV pair containing the matched filename (from MetadataResult) and the corresponding SHA-256 of the whole file (instead of reading it line by line):

p
  .apply("Match Filenames", FileIO.match().filepattern(options.getInput()))
  .apply("Read Matches", FileIO.readMatches())
  .apply(MapElements.via(new SimpleFunction <ReadableFile, KV<String,String>>() {
      public KV<String,String> apply(ReadableFile f) {
            String temp = null;
            try{
                temp = f.readFullyAsUTF8String();
            }catch(IOException e){

            }

            String sha256hex = org.apache.commons.codec.digest.DigestUtils.sha256Hex(temp);   

            return KV.of(f.getMetadata().resourceId().toString(), sha256hex);
        }
      }
  ))
  .apply("Print results", ParDo.of(new DoFn<KV<String, String>, Void>() {
      @ProcessElement
      public void processElement(ProcessContext c) {
        Log.info(String.format("File: %s, SHA-256: %s ", c.element().getKey(), c.element().getValue()));
      }
    }
 ));

Full code here. The output in my case was:

Apr 21, 2019 10:02:21 PM com.dataflow.samples.DataflowSHA256$2 processElement
INFO: File: /home/.../data/file1, SHA-256: e27cf439835d04081d6cd21f90ce7b784c9ed0336d1aa90c70c8bb476cd41157 
Apr 21, 2019 10:02:21 PM com.dataflow.samples.DataflowSHA256$2 processElement
INFO: File: /home/.../data/file2, SHA-256: 72113bf9fc03be3d0117e6acee24e3d840fa96295474594ec8ecb7bbcb5ed024

Which I verified with an online hashing tool:

By the way I don't think you need OutputReceiver for a single output (no side outputs). Thanks to these questions/answers that were helpful: 1, 2, 3.

This is awesome, do you have any idea on where to point me for the part where I have to write the KV pairs to a json file? — dmc94, Apr 21 '19 at 20:46

How do I get a Readable File?

1 Answers1