Stack Overflow
Stack Overflow

Questions tagged [kryo]

Kryo is a fast and efficient object graph serialization framework for Java. The goals of the project are speed, efficiency, and an easy to use API. The project is useful any time objects need to be persisted, whether to a file, database, or over the network.

431 questions
38
votes
9 answers

Kryo serialization library: is it used in production?

Kryo is a very new and interesting Java serialization library, and one of the fastest in the thrift-protobuf benchmark. If you've used Kryo, has it already reached enough maturity to try it out in production code? Update (10/27/2010): We're using…
Jim Ferrans
  • 30,582
  • 12
  • 56
  • 83
33
votes
3 answers

Require kryo serialization in Spark (Scala)

I have kryo serialization turned on with this: conf.set( "spark.serializer", "org.apache.spark.serializer.KryoSerializer" ) I want to ensure that a custom class is serialized using kryo when shuffled between nodes. I can register the class with…
pheaver
  • 1,813
  • 1
  • 14
  • 23
26
votes
6 answers

How Kryo serializer allocates buffer in Spark

Please help to understand how Kryo serializer allocates memory for its buffer. My Spark app fails on a collect step when it tries to collect about 122Mb of data to a driver from workers. com.esotericsoftware.kryo.KryoException: Buffer overflow.…
vvladymyrov
  • 5,715
  • 2
  • 32
  • 50
23
votes
1 answer

Customize SparkContext using sparkConf.set(..) when using spark-shell

In Spark, there are 3 primary ways to specify the options for the SparkConf used to create the SparkContext: As properties in the conf/spark-defaults.conf e.g., the line: spark.driver.memory 4g As args to spark-shell or spark-submit e.g.,…
rake
  • 2,348
  • 3
  • 15
  • 11
19
votes
3 answers

When to use Kryo serialization in Spark?

I am already compressing RDDs using conf.set("spark.rdd.compress","true") and persist(MEMORY_AND_DISK_SER). Will using Kryo serialization make the program even more efficient, or is it not useful in this case? I know that Kryo is for sending the…
pythonic
  • 20,589
  • 43
  • 136
  • 219
18
votes
3 answers

Kryo Deserialization fails with "KryoException: Buffer underflow"

I use Kryo to write Objects into byte arrays. It works fine. But when the byte arrays are converted into the Objects, it throws, com.esotericsoftware.kryo.KryoException: Buffer underflow. exception. This is my deserialization: Kryo…
Débora
  • 5,816
  • 28
  • 99
  • 171
17
votes
3 answers

Dealing with an incompatible version change of a serialization framework

Problem description We have a Hadoop cluster on which we store data which is serialized to bytes using Kryo (a serialization framework). The Kryo version which we used to do this has been forked from the official release 2.21 to apply our own…
Michael Schmeißer
  • 3,407
  • 1
  • 19
  • 32
16
votes
2 answers

Why is Spark performing worse when using Kryo serialization?

I enabled Kryo serialization for my Spark job, enabled the setting to require registration, and ensured all my types were registered. val conf = new SparkConf() conf.set("spark.serializer",…
Leif Wickland
  • 3,693
  • 26
  • 43
14
votes
2 answers

Do you benefit from the Kryo serializer when you use Pyspark?

I read that the Kryo serializer can provide faster serialization when used in Apache Spark. However, I'm using Spark through Python. Do I still get notable benefits from switching to the Kryo serializer?
Gere
  • 12,075
  • 18
  • 62
  • 94
14
votes
1 answer

Spark Kryo: Register a custom serializer

I have a class that implements a custom Kryo serializer by implementing the read() and write() methods from com.esotericsoftware.kryo.Serializer (see example below). How can I register this custom serializer with Spark? Here is a pseudo-code example…
marios
  • 8,874
  • 3
  • 38
  • 62
11
votes
1 answer

Spark - Serializing an object with a non-serializable member

I am going to ask this question in the context of Spark, because that's what I'm facing, but this might be a plain Java issue. In our spark job, we have a Resolver which needs to be used in all of our workers (it's used in a udf). The problem is…
summerbulb
  • 5,709
  • 8
  • 37
  • 83
11
votes
4 answers

kryo.readObject cause NullPointerException with ArrayList

I get a NullPointerException when I unserialize an ArrayList object using kryo. Caused by: java.lang.NullPointerException at java.util.ArrayList.ensureExplicitCapacity(ArrayList.java:215) at…
cquezel
  • 3,859
  • 1
  • 30
  • 32
10
votes
2 answers

What are the pros and cons of java serialization vs kryo serialization?

In spark, java serialization is the default, if kryo is that efficient then why it is not set as default. Is there some cons using kryo or in what scenarios we should use kryo or java serialization?
supernatural
  • 1,107
  • 11
  • 34
10
votes
1 answer

Kryo crash EnvironmentFailureException

I'm trying to serialize some objects of custom classes and I got the following exception: com.esotericsoftware.kryo.KryoException: com.sleepycat.je.EnvironmentFailureException: (JE 5.0.73) IdentityHashMap.entrySet() should not be used. See…
raycons
  • 735
  • 12
  • 26
10
votes
4 answers

Resolving "Kryo serialization failed: Buffer overflow" Spark exception

I am trying to run Spark (Java) code and getting the error org.apache.spark.SparkException: Kryo serialization failed: Buffer overflow. Available: 0, required: 27". Other posts have suggested setting the buffer to its max value. When I tried this…
Raghavendran Pedapati
  • 101
  • 1
  • 1
  • 4
1
2 3
…
28 29