7

I need to convert Nippy data structures stored on disk into something that can be read by Nippy? Nippy uses byte arrays, so I need some way to convert the file into a byte array. I have tried 

(clojure.java.io/to-byte-array (clojure.java.io/file folder-path file-path))

but this gives

java.lang.IllegalArgumentException: Value out of range for byte: ?

Then I try:

(into-array Byte/TYPE  (map byte (slurp (clojure.java.io/file folder-path file-path))))

but somehow the namespace is wrong, and I can't find the right one.

To write the Nippy structures in the first place, I am using:

(with-open [w (clojure.java.io/output-stream file-path)]
    (.write w (nippy/freeze data)))))
user1559027
  • 343
  • 2
  • 13

6 Answers6

33

Here's how I do it generically with clojure built-ins

(defn slurp-bytes
  "Slurp the bytes from a slurpable thing"
  [x]
  (with-open [in (clojure.java.io/input-stream x)
              out (java.io.ByteArrayOutputStream.)]
    (clojure.java.io/copy in out)
    (.toByteArray out)))

EDIT: Updated answer based on Jerry101's suggestion in comments.

Matt W-D
  • 1,605
  • 2
  • 19
  • 22
  • Thank you so much for writing this answer, it helped me so much in my quest of converting an image to a base64 UTF8 string. – asiegf Aug 16 '19 at 08:51
  • 2
    This is good, but the input-stream should be in a `with-open` binding. You can bind the input and the output in the same `with-open`. – Jerry101 Aug 17 '19 at 01:02
16

I'm not aware of anything built-in to Clojure that will handle this. You definitely don't want slurp because that will decode the stream contents as text.

You could write your own method to do this, basically reading from the InputStream into a buffer and writing the buffer to a java.io.ByteArrayOutputStream. Or you could use the IOUtils class from Apache Commons IO:

 (require '[clojure.java.io :as io])
 (import '[org.apache.commons.io IOUtils])

 (IOUtils/toByteArray (io/input-stream file-path))

You should also take a look at Nippy's thaw-from-in! and freeze-to-out! functions:

 (import '[java.io DataInputStream DataOutputStream])

 (with-open [w (io/output-stream file-path)]
   (nippy/freeze-to-out! (DataOutputStream. w) some-data))

 (with-open [r (io/input-stream file-path)]
   (nippy/thaw-from-in! (DataInputStream. r)))
Alex
  • 13,811
  • 1
  • 37
  • 50
7

Since you know the .length of the file, you can allocate once and use DataInputStream's readFully method. No additional libraries, buffer copies, or loops required.

(defn file-to-byte-array
  [^java.io.File file]
  (let [result (byte-array (.length file))]
    (with-open [in (java.io.DataInputStream. (clojure.java.io/input-stream file))]
      (.readFully in result))
    result))
ɲeuroburɳ
  • 6,990
  • 3
  • 24
  • 22
2

A quick make-shift solution may be this code:

(defn slurpb [is]
  "Convert an input stream is to byte array"
  (with-open [baos (java.io.ByteArrayOutputStream.)]
    (let [ba (byte-array 2000)]
      (loop [n (.read is ba 0 2000)]
        (when (> n 0)
          (.write baos ba 0 n)
          (recur (.read is ba 0 2000))))
      (.toByteArray baos))))

;;test
(String. (slurpb (java.io.ByteArrayInputStream. (.getBytes "hello"))))
user3742065
  • 21
  • 1
2

Please note that I just cut Nippy v2.13.0 which now includes a pair of helper utils to help simplify this use case: freeze-to-file and thaw-from-file.

Release details at: https://github.com/ptaoussanis/nippy/releases/tag/v2.13.0

Cheers!

Peter Taoussanis
  • 350
  • 2
  • 4
0

You can give a try to ClojureWerk's Buffy : https://github.com/clojurewerkz/buffy.

Buffy is a Clojure library for working with binary data, writing complete binary protocol implementations in Clojure, storing complex data structures in an off-heap cache, reading binary files and doing everything you would usually do with ByteBuffer.

It's very neat if your binary data is structured as you can define complex composite types and frames depending on structure types, even decode UTF.

Ivan Pierre
  • 808
  • 7
  • 9