3

What I'm trying to do is package a large file (a MIDI soundfont) in a standalone Maven repo/clojar, and then be able to pull it down programmatically and use it from a separate project. This seemingly simple task is proving to be more complicated than I expected.

What would be ideal is if there were a way to access these resources directly, or expose them as public vars, or something. This is the first thing I tried -- I did something like this:

(ns midi.soundfont.fluid-r3
  (:require [clojure.java.io :as io]))

(def sf2
  (io/file (io/resource "fluid-r3.sf2")))

However, the problem that I'm running into is that io/resource only finds resource files on the current class path. As soon as I try to require this namespace from another project (or from the REPL), I get:

java.lang.IllegalArgumentException: Not a file: jar:file:/Users/dave/.m2/repository/midi/soundfont/fluid-r3/midi.soundfont.fluid-r3/0.1.0/midi.soundfont.fluid-r3-0.1.0.jar!/fluid-r3.sf2

If it's not possible to access the resource directly, I would be happy with a solution that involves copying the file to some path in the filesystem. I did try this as well, but I ran into the same problem when trying to run the "copy the file to the filesystem" method from a different project -- io/resource still failed to locate the file because it's not on the current classpath.

I have found similar questions that have been asked on SO previously, such as:

However these solutions only seem to pertain to copying a file that is a resource in the current (running) project.

Is it possible to do one of these two things?

  1. Access a resource file from an external clojar
  2. Import the resource file into the current project, so that I can access it using io/resource
Community
  • 1
  • 1
Dave Yarwood
  • 2,866
  • 1
  • 17
  • 29
  • Copying the resource into a file is your answer. As the error message states, the resource inside a jar is not a file. It is not a classpath problem, it is the fundamental fact that a resource inside a jar is not a file. – noisesmith Jun 16 '15 at 02:17
  • "Not a file" seems to be Java's way of telling me it cannot make a file out of a nonexistent resource. Running `(io/file (io/resource "fluid-r3.sf2"))` works when running on the classpath of the `midi.soundfont.fluid-r3` project (whose resource directory contains that file), but throws the "Not a file" exception when run in any other project. – Dave Yarwood Jun 16 '15 at 02:21
  • To clarify, I am able to copy the resource to a file. The issue is doing this from a different project. Does that make sense? – Dave Yarwood Jun 16 '15 at 02:22
  • That's not how you make a file out of a resource. The argument to `io/file` must be something that can be used as a file - a resource entry inside a jar simply does not qualify. You can read from the resource and write to an actual file, as documented in the links in your post. – noisesmith Jun 16 '15 at 02:48
  • `not a file` does not mean "could not find the resource" - the resource exists, it's inside a jar. It just isn't a file. – noisesmith Jun 16 '15 at 02:50
  • https://gist.github.com/daveyarwood/357de21e85c22838d841 <-- I tried copying the resource to the filesystem via the method described [here](http://stackoverflow.com/a/28657488/2338327), but I still get the "not a file" exception when trying to run `(midi.soundfont.fluid-r3/sf2)` from a *different* project. This all hinges on `io/resource` only working for the current classpath. – Dave Yarwood Jun 16 '15 at 04:06
  • Yes, `io/resource` by definition can only find things in the classpath. If that code gets you file related errors, it is because of the destination file not the resource, not finding the resource would be a different error – Justin Smith Jun 16 '15 at 18:46
  • My question is: is it possible to access resources that are *not* on the classpath? Or, is it possible to add resources from a dependency into the current classpath? – Dave Yarwood Jun 16 '15 at 18:55
  • 1
    By definition, a resource is something on the classpath. You can slurp a resource. You can copy from a resource into a byte array or an outputstream or an actual file. But you can't simply coerce a resource to be a file, unless your URI points to a place on disk. – noisesmith Jun 17 '15 at 03:12
  • That answers my question. Thanks for your patience, everyone! – Dave Yarwood Jun 17 '15 at 04:39

3 Answers3

3

As dbasch correctly explained, io/resource returns a URL, not a file. But why you are being able to open that URL with io/file on the REPL or lein run but not from the jar? That's because the URL in the first case points to the plain file in the filesystem, while the URL when running with the jar points to the resource inside the jar, so it's not a proper file.

I made an example in this github repo. I'll copy the -main code here for reference:

(defn -main [& args]
  (let [r (io/resource "greet")]
    (println r)
    (println (slurp r))
    (with-open [rdr (io/reader r)]
      (println (clojure.string/join ", " (line-seq rdr))))
    (println (io/file r))))

Running with lein run shows:

› lein run
#<URL file:/home/nicolas/projects/clojure/resources/resources/greet>
hello
world

hello, world
#<File /home/nicolas/projects/clojure/resources/resources/greet>

Running the uberjar shows:

› java -jar target/resources-0.1.0-SNAPSHOT-standalone.jar 
#<URL jar:file:/home/nicolas/projects/clojure/resources/target/resources-0.1.0-SNAPSHOT-standalone.jar!/greet>
hello
world

hello, world
Exception in thread "main" java.lang.IllegalArgumentException: Not a file: jar:file:/home/nicolas/projects/clojure/resources/target/resources-0.1.0-SNAPSHOT-standalone.jar!/greet
        at clojure.java.io$fn__8588.invoke(io.clj:63)
        at clojure.java.io$fn__8572$G__8556__8577.invoke(io.clj:35)

See the difference between #<URL file:/home/nico... and #<URL jar:file:/home/nico..., that explains why you can't call (io/file) on it, but you can read it with slurp or create a reader with io/reader.

nberger
  • 3,659
  • 17
  • 19
1

(io/resource "fluid-r3.sf2") is a url, not a file. You can slurp it if you want it in memory all at once, or read it as a stream with the java.net.URL api (and write it to a file as you read it if you want to).

Example:

user> (type (clojure.java.io/resource "text.txt"))
java.net.URL
user> (slurp (clojure.java.io/resource "text.txt"))
"this is a text file\n"
Diego Basch
  • 12,764
  • 2
  • 29
  • 24
  • My problem isn't how to copy a file, it's how to access a resource from another project. `io/resource` fails with the "Not a file" exception if the file I'm trying to access isn't in the current project. – Dave Yarwood Jun 16 '15 at 14:52
  • minor correction: trying to do things like `io/file`, `slurp`, etc. on `(io/resource "fluid-r3.sf2")` fails with "Not a file" if that file doesn't exist on the current resource path. – Dave Yarwood Jun 16 '15 at 14:54
  • 1
    if it was not on the resource-path your error would be `IllegalArgumentException No implementation of method: :make-reader of protocol: #'clojure.java.io/IOFactory found for class: nil clojure.core/-cache-protocol-fn (core_deftype.clj:554)` - it would not mention anything about files, because nothing about slurp or resource implies a file, and `io/resource` returns nil for a resource that isn't on the path. – Justin Smith Jun 16 '15 at 18:44
  • Hmm... interesting. `io/resource` does return a URL, when trying it from a different project, but then when I try to make a file out of the resource, I get "not a file." I think I'm just misunderstanding how to make a file out of a resource... `(io/file (io/resource "path/to/file"))` has always worked for me, but only when I'm trying to access a resource that is on the resource path for the *current* project. Is there some trick to accessing a resource from an *external* project (e.g. a clojar pulled in as a dependency)? – Dave Yarwood Jun 16 '15 at 18:52
  • You can't make a file out of a URL. You can download a URL to a file. I thought my answer was clear enough. – Diego Basch Jun 16 '15 at 22:40
  • Sorry -- I was fundamentally confused about `io/resource`, which sort of led me on a wild goose chase. I appreciate your patience! – Dave Yarwood Jun 17 '15 at 04:41
1

"Access a resource file from an external clojar"

I found this solution from this article. Dmitri Sotnikov did this at his cryogen project.

source code is here. The load-plugins functions will retrieve all the plugin.edn from under every resources directory of external clojars. 

(ns cryogen-core.plugins
  (:require [clojure.edn :as edn]
            [clojure.string :as string]
            [text-decoration.core :refer :all]))

(defn load-plugin [^java.net.URL url]
  (let [{:keys [description init]} (edn/read-string (slurp url))]
    (println (green (str "loading module: " description)))
    (-> init str (string/split #"/") first symbol require)
    ((resolve init))))

(defn load-plugins []
  (let [plugins (.getResources (ClassLoader/getSystemClassLoader) "plugin.edn")]
    (doseq [plugin (enumeration-seq plugins)]
      (load-plugin (. ^java.net.URL plugin openStream)))))

If you do not like too much java interoperation, there is a resources function in pomegranate library which can retrieve all the resources with given name in external clojars.

Laurence Chen
  • 1,738
  • 1
  • 13
  • 15