5

I am about to start learning functional programming and Clojure appeals to me the most, I love its community, syntax and concept of immutable data structures. I am also interested in bio inspired ML for rich data Numenta. However, my huge concern is that Spark does not support it as yet, and Spark rocks!!! There is a Flambo Flambo Clojure ,but is it a satisfactory solution?

My course and job is in Scala. Should I defeat and enter Scala world or do you think that I should focus solely on Clojure?

elcomendante
  • 1,113
  • 1
  • 11
  • 28
  • 2
    Do both. Studying Clojure will make you a better Scala programmer (read [Paul Graham](http://www.paulgraham.com/avg.html) and [Eric Raymond](http://www.catb.org/~esr/faqs/hacker-howto.html)). And Scala will do you no harm. One of the nice things about the Clojure community is that they appreciate other languages. – Thumbnail Jun 09 '15 at 06:28

2 Answers2

10

Being the author or Sparkling (thanks to Josh Rosen for pointing that out), I can tell you that we use it at our company for ETL processing.

Here's what's good:

  • it provides a Clojuristic way of interacting with Spark, and as you can see in the presentation "Big Data with style - the Clojure/Spark way"
  • it's optimized for performance
  • it's used in production and there are others also using it

Here's what's missing:

  • There's currently no support for Spark Streaming, Spark SQL, Spark Dataframes or Spark ML. That might come in the future, I'm happy to accept pull requests, but it's currently not main focus (at the time of writing, April 2015).

I hope this helps you make up your mind on going with Clojure or starting to learn Scala.

Community
  • 1
  • 1
cbbetz
  • 136
  • 2
2

It's hard to say that something like Spark doesn't support Clojure. It would make more sense to ask if there are good libraries to use that project that are easy to use from Clojure. From googling around Flambo looks like a viable option and at the various clojure conferences I hear incidental talk of using Spark in several contexts.

I would say that there is fairly low technical risk in using spark from Clojure so you are free to make this choice based on the other constraints of your working environment and pojects. Being particularly biased toward Clojure I whole heartedly recommend at least trying it and see what parts of the language and ecosystem work well for you.

Arthur Ulfeldt
  • 90,827
  • 27
  • 201
  • 284
  • I agree. I really wanted to use Clojure for Spark jobs but had a few tricky issues with serialisation which were hard for me to debug. I would observe that shifting to Scala returned the investment in other ways, and now I would say that I have a healthy appreciation for the value of typing, even though I miss the sublime Clojure syntax. – Alister Lee Apr 10 '15 at 00:10
  • 2
    It looks like there's also [Sparkling](https://github.com/gorillalabs/sparkling) (note that I haven't tried this, as I am not a Clojure user). – Josh Rosen Apr 10 '15 at 00:12