Kafka Producer Error: ' Value serializer not specified and there is no default serializer defined for type ...'

Question

I just started using Kafka and hit the following rookie error:

'Value cannot be null.
Parameter name: Value serializer not specified and there is no default serializer defined for type ActMessage.'

It happens when trying to send a class object, ActMessage object, rather then the a simple string that comes with the example. The line of code that raises the erros is:

using (var p = new ProducerBuilder<Null, ActMessage>(config ).Build()

I am using the .net client.

My understanding is that i need to use one of the default serializes in the first type parameter, one that come with Kafka client, as explained here, but can't find them on this .net package. I guess i could build one but that would be a waste of time.

Here a reproducible example:

public class ActMessage  {
    public int SomeId {get;set;}
    public string SomeContent {get;set;}
}

class Tester  {

void send(){

    var config = new ProducerConfig { BootstrapServers = "localhost:9092" };

        using (var p = new ProducerBuilder<Null, ActMessage>(config).Build()) //throws error here
        {
            var dr = p.ProduceAsync("news", new Message<Null, ActMessage>
            {
                Value = new ActMessage { SomeId = 1, SomeContent="hi" },
            }
                ).Result;
        }
    }
}

Okay. You didn't set a serializer in the ProducerConfig like the error says. How would Kafka know what to do with your class to convert it to bytes? Similarly, you should at least add a toString implementation to your class — OneCricketeer, Apr 15 '20 at 17:20
I would expect some default implementation, to json, xml, or even binary. But as i say, i am starting with kafka, and try to use it in the simpler way possible. — MiguelSlv, Apr 15 '20 at 17:41
Given that C# doesn't have a default JSON or XML lib, then binary would be an option, assuming you know how to use serialization interfaces of C# — OneCricketeer, Apr 15 '20 at 17:42
I am going for json, can be useful to read the messages exchanged between services with a naked eye. I am moving away from MSMQ and kafka is quite different. — MiguelSlv, Apr 15 '20 at 17:49
Avro can also be inspected by the naked eye. Look at services like KSQL, for example. Benefit of which would be far greater throughput on your topic, and almost all external Kafka services kinda use the Confluent Schema Registry API to do really cool things — OneCricketeer, Apr 16 '20 at 09:44

OneCricketeer · Accepted Answer · 2021-07-29T16:59:28.640

7

I suggest checking out the working examples/ dir in that repo to see working code that you can copy into your own projects.

If you have your own class, you need to implement the ISerializer and IDeserializer interfaces.

Or you can use the built-in ones

However, an alternative is to use Avro

This requires writing an Avro schema file, then using avrogen to create your class, not manually write it. E.g.

dotnet tool install --global Apache.Avro.Tools

avrogen -s User.avsc .

Then you must always add some ValueSerializer in Kafka clients in order to send data

edited Jul 29 '21 at 16:59

answered Apr 15 '20 at 17:23

OneCricketeer

179,855
19
132
245

1

I was expecting that i could use some general purpose, built-in serializer, but that is ok anyway. – MiguelSlv Apr 15 '20 at 17:37
You can. Implement `ToString`, then use `StringSerializer` – OneCricketeer Apr 15 '20 at 17:41
In the examples above i see there is a string `schemaRegistryUrl`.It is provided as an argument to the `Main`. I was wondering how does that value look like. Thanks. – Bercovici Adrian Jul 29 '21 at 14:25
@BercoviciAdrian Its just the registry HTTP endpoint. `http://registry.address:8081` – OneCricketeer Jul 29 '21 at 14:27
I do not really get how i can go from a C# model to write it in `KSQL` (Kafka) as `AVRO` as vice-versa. – Bercovici Adrian Jul 29 '21 at 14:36
@BercoviciAdrian In the linked example, after `.SetValueSerializer(new AvroSerializer(schemaRegistry, avroSerializerConfig))`, where `User` is a generated class from an Avro schema [using `avrogen` tool mentioned in the repo](https://github.com/confluentinc/confluent-kafka-dotnet/tree/master/examples/AvroSpecific#avro). Then bytes are sent to Kafka.. From KSQL (or any other client), they need the corresponding deserializer. – OneCricketeer Jul 29 '21 at 16:53
I am facing the same issue yet my class implements Serializeable and has toString method but the ProducerConfig class in Confluent.Kafka does not have a definition for ValueSerializer or Serializer at all. So how to bind Serializers to ProducerConfig? – Guy_g23 Jan 31 '23 at 09:14
1

@Guy As answered, you use `SetValueSerializer`. Do not implement native Serializable interface. You instead define your own Serializer subclass for your POCO – OneCricketeer Jan 31 '23 at 15:11
@OneCricketeer this is just what I did (beware everybody of Serializeable interface). Thanks! – Guy_g23 Jan 31 '23 at 15:22
@Guy Not really "beware". Just a misunderstanding that "native serialization" is not cross-language. E.g. If someone tried to consume your C# `Serializable` class in JVM client, and had also `implements Serializable` on those class types for a consumer, then it wouldn't work. Thus, why you use cross language formats like JSON/Avro/Protobuf – OneCricketeer Jan 31 '23 at 19:00

Kafka Producer Error: ' Value serializer not specified and there is no default serializer defined for type ...'

1 Answers1