Unmarshalling stream of changing types into structs

Question

I'm currently dealing with a stream of json objects coming in to my application, and am having some difficulties figuring out what the best way of parsing them is. The stream consists of objects that have a defined type. The problem is that one of the fields in the object is of changing type. It looks like this:

[{
  "status": "closed",
  "type": "transaction",
  "transaction": {
    "TransactionType": "TypeA",
    "Account": "Some string",
    "Fee": "14",
    "date": 45325680
  },
  "validated": true
},

{
  "status": "closed",
  "type": "transaction",
  "transaction": {
    "TransactionType" : "TypeB",
    "Account" : "Some string",
    "Fee": "42",
    "Destination" : "Some string"
  },
  "validated": true
}]

You can see that the "parent" does not change, but the "transaction" does. I removed a lot of fields from the "transaction" field to make it easier to explain, but this is a type with 10-ish common fields and 10-ish changing fields that are dependent on the type. There are also 10 transaction types, this makes it pretty annoying to put everything into one struct and have a ton of optional fields. I was also thinking about one struct for every transaction type, but this does not work because then there is no way of specifying which type the field should have in the parent.

How would I parse these objects effectively? Unfortunately, I can't change the structure of the elements coming out of the stream. What would be the best way to solve this?

Possible duplicate of [Gson Parse Json with array with different object types](https://stackoverflow.com/questions/14713736/gson-parse-json-with-array-with-different-object-types) — Mike Beeler, Dec 12 '17 at 18:40
`TransactionType` is still a string. I'm guessing it's used to tell you which fields in the `transaction` struct are important. This means that there should be an authoritative list of all possible fields, with the inclusion or not of fields based on the type of transaction. What have you tried, and why didn't it work? — Marc, Dec 12 '17 at 18:40
Correct. The problem is that I can't seem to figure out (or I just don't know) what the most efficient way of unmarshalling such a structure is. I can indeed figure out how to unmarshall the transaction field with a contains on the `RawMessage` for instance, but then where would I place this result? Should the "parent" struct have a field for each type of transaction? Or should I have a field with the type `interface{}` and a string that indicates the type of the transaction on the "parent" struct? — martijn9612, Dec 12 '17 at 18:48
An interface contains a type and a value. If you use a field with interface type, then you may not need the string type in the "parent" struct. The application can type switch or type assert to detect type. The interface can contain common methods for all transaction types. — Charlie Tumahai, Dec 12 '17 at 18:54

score 0 · Answer 1 · answered Dec 12 '17 at 19:20

Perhaps keep your transaction as map[string]interface{}

Example https://play.golang.com/p/j_u_ztw04M

package main

import (
    "encoding/json"
    "fmt"
)

type Stream []struct {
    Status string `json:"status"`
    Type   string `json:"type"`

    // map instead of struct
    Transaction map[string]interface{} `json:"transaction"`

    Validated bool `json:"validated"`
}

func main() {

    stream := Stream{}

    err := json.Unmarshal(rawj, &stream)
    if err != nil {
        fmt.Println("error:", err)
        return
    }

    for i, s := range stream {
        fmt.Println("thing:", i)
        for k, v := range s.Transaction {

            // do manual stuff here, perhaps long switch on k and/or
            // as suggested by Cerise Limón type switch on v
            fmt.Printf("key: %s, val: %v, val type: %T\n", k, v, v)

        }
        fmt.Println()
    }

}

var rawj = []byte(`[{
  "status": "closed",
  "type": "transaction",
  "transaction": {
    "TransactionType": "TypeA",
    "Account": "Some string",
    "Fee": "14",
    "date": 45325680
  },
  "validated": true
},

{
  "status": "closed",
  "type": "transaction",
  "transaction": {
    "TransactionType" : "TypeB",
    "Account" : "Some string",
    "Fee": "42",
    "Destination" : "Some string"
  },
  "validated": true
}]`)

Have fun!

maerics · Answer 2 · 2017-12-12T20:54:28.633

For the "transaction" struct I would make the common keys direct types (e.g. string, int, etc.) and the optional keys pointers to their types (*string, *int, etc.), with the "omitempty" tag so they are set to nil if they aren't present in an occurrence:

type Transaction struct {
  // Common fields are direct types...
  TransactionType string
  Account         string
  Fee             string

  // Optional fields are pointers with "omitempty"...
  Date        *int    `json:",omitempty"`
  Destination *string `json:",omitempty"`
}

This way the common fields are always there but possibly the default (empty) values and optional fields are populated if present but nil if omitted:

ts[0].Date = (*int) 45325680
ts[0].Destination = nil
ts[1].Date = nil
ts[1].Destination = (*string) "Some string"

mozey · Answer 3 · 2017-12-13T16:09:39.177

Define structs for each transaction type and do the un-marshall in two steps. First use

type Stream []struct {
    ...
    TransactionRaw json.RawMessage `json:"transaction"`
    ...
}

Then use Regexp.FindStringSubmatch on TransactionRaw to determine the type for the second call to json.Unmarshal with

type Transaction struct {
    TransactionType string `json:"TransactionType"`
    Account         string `json:"Account"`
    Fee             string `json:"Fee"`
}

type TypeA struct {
    Transaction
    Date int    `json:"date"`
}

etc...

Playground example here

score 0 · Answer 4 · answered Dec 14 '17 at 00:27

0

I think using reflect is one way []map[string]interface{}

And in this answer I had gave an example about using reflect to decode changing type.

answered Dec 14 '17 at 00:27

Terry Pang

359
1
2
6

Unmarshalling stream of changing types into structs

4 Answers4

Linked