Stack Overflow
Stack Overflow

Questions tagged [avro4s]

Scala avro schema generation and serialization/deserialization.

Scala avro schema generation and serialization/deserialization. https://github.com/sksamuel/avro4s

45 questions
9
votes
1 answer

Data validation in AVRO

I am new to AVRO and please excuse me if it is a simple question. I have a use case where I am using AVRO schema for record calls. Let's say I have avro schema { "name": "abc", "namepsace": "xyz", "type": "record", "fields": [ …
user2166328
  • 199
  • 2
  • 2
  • 6
7
votes
2 answers

Creating AVRO schema from JSON Schema File

I have the JSON file & JSON Schema to be parsed into the AVRO Schema. I am little bit confused, do i have to write the manual AVRO schema using the data types defined in AVRO documentation. Or is there any automated method / function / program that…
user2857762
  • 81
  • 1
  • 4
3
votes
0 answers

Cant select data from Avro Table in presto

I have created Avro table in hive, using presto. using below query CREATE TABLE hive.psa_sbox.avro_testing ( id bigint ) WITH ( format = 'AVRO', external_location = 'my/datalake/data/folder/avro_data', avro_schema_url =…
Maqsood
  • 369
  • 4
  • 17
3
votes
0 answers

Avro4S: Error when Deserializing Abstract Data Types (Could not find implicit value for evidence parameter of type FromRecord)

I am using avro4s https://github.com/sksamuel/avro4s I wrote this code import java.io.ByteArrayOutputStream import com.sksamuel.avro4s._ sealed trait Dibble case class Dobble(str: String) extends Dibble case class Dabble(dbl: Double)…
flare
  • 344
  • 2
  • 6
3
votes
2 answers

Avro record with Any type field in Scala

Say I have a simple key value pair in Avro where the value could be a float, double, int, string etc; {"namespace": "com.namespace.kafka.event", "type": "record", "name": "RecordName", "fields": [ {"name": "key", "type": "String"}, …
NightWolf
  • 7,694
  • 9
  • 74
  • 121
2
votes
4 answers

Flink throwing com.esotericsoftware.kryo.KryoException: java.lang.UnsupportedOperationException

I am trying to deserialize kafka events in my flink stream job. This is my code: ... case class URLResponse (status: Int, domain: String, url: String, queue: String, html: String) ... val schema: Schema = AvroSchema[URLResponse] ... val stream =…
Ruslan Ksalov
  • 111
  • 1
  • 7
2
votes
1 answer

AVRO schema with optional record

Hi folks I need to create AVRO schema for the following example ; { "Car" : { "Make" : "Ford" , "Year": 1990 , "Engine" : "V8" , "VIN" : "123123123" , "Plate" : "XXTT9O", "Accident" : { "Date" :"2020/02/02" , "Location" : "NJ" , "Driver" : "Joe" }…
LSasmazel
  • 21
  • 1
  • 4
2
votes
2 answers

Schema in Avro message

I see that the Avro messages have the schema embedded, and then the data in binary format. If multiple messages are sent and new avro files are getting created for every message, is not Schema embedding an overhead? So, does that mean, it is always…
Roshan Fernando
  • 493
  • 11
  • 31
2
votes
2 answers

How to add evoltion to avro4s in binary serialized data

I have a a class like this: case class SurveyTemplate(id:UUID, name: String, version: Int, preDefinedChapter: Seq[Chapter]) { } I serialized this as binary in a DAO with…
André
  • 125
  • 1
  • 8
2
votes
1 answer

could not find implicit value for FromRecord

I'm trying to get started with avro & avro4s and I'm running into trouble getting a silly example working. I can't even get my own version of a test found in the source repo to compile -…
Jason
  • 15,915
  • 3
  • 48
  • 72
2
votes
1 answer

Is there a way to support schema evolution in avro4s

Is there a way to read bytesArray via the AvroInputStream.binary api and somehow specify the writer version ? val in = new ByteArrayInputStream(bytesFromFile) val input = AvroInputStream.binary[newClass](in)// old ? val result =…
NetanelRabinowitz
  • 1,534
  • 2
  • 14
  • 26
1
vote
1 answer

Differences between IETF standard JSON schema and Avro schema

Please help me understand the differences between IETF standard JSON schema and Avro schema. I google a lot but couldn't find anything concrete.
banjara
  • 3,800
  • 3
  • 38
  • 61
1
vote
1 answer

Do I really need avro4s when using kafka schema registry?

I noticed confluent has a kafka serializer that will let me serialize and de-serialize my case classes from my kafka topic, and it will pull the schema from the registry. If this is the case, what benefit would I get by using avro4s?
Blankman
  • 259,732
  • 324
  • 769
  • 1,199
1
vote
1 answer

Implicit object works inline but not when it is imported

I am using avro4s to help with avro serialization and deserialization. I have a case class that includes Timestamps and need those Timestamps to be converted to nicely formatted strings before I publish the records to Kafka; the default encoder is…
sparkyShorts
  • 630
  • 9
  • 28
1
vote
0 answers

Avro4s: The argument types of an anonymous function must be fully known. (SLS 8.5)

I am trying to map the fields of a case class using the avro4s FieldMapper. implicit val short: FieldMapper = { case "fieldName1" => "fieldName2" } But I'm receiving the following error: Error:(20, 39) missing parameter type for expanded…
CasualName123
  • 11
  • 1
1
2 3