How to flatten out a nested json structure in go

Question

I am getting nested data from mongo and I want to flatten that out in a structure to store it in a csv file.

The data looks like this:

{
    "_id" : "bec7bfaa-7a47-4f61-a463-5966a2b5c8ce",
    "data" : {
        "driver" : {
            "etaToStore" : 156
        },
        "createdAt" : 1532590052,
        "_id" : "07703a33-a3c3-4ad5-9e06-d05063474d8c"
    }
} 

And the structure I want to eventually get should be something like this

type EventStruct struct {
    Id                  string      `bson:"_id"`
    DataId              string      `bson:"data._id"`
    EtaToStore          string      `bson:"data.driver.etaToStore"`
    CreatedAt           int         `bson:"data.createdAt"`
}

This doesn't work, so following some SO answers I broke it down into multiple structures:

// Creating a structure for the inner struct that I will receive from the query
type DriverStruct struct {
    EtaToStore  int     `bson:"etaToStore"`
}

type DataStruct struct {
    Id          string `bson:"_id"`
    Driver      DriverStruct `bson:"driver"`
    CreatedAt   int `bson:"createdAt"`
}
// Flattenning out the structure & getting the fields we need only
type EventStruct struct {
    Id                  string      `bson:"_id"`
    Data                DataStruct  `bson:"data"`
}

This gets all the data from the Mongo query result but it's nested:

{
  "Id": "bec7bfaa-7a47-4f61-a463-5966a2b5c8ce",
  "Data": {
     "Id": a33-a3c3-4ad5-9e06-d05063474d8c,
     "Driver": {
        "EtaToStore": 156
     },
     "CreatedAt": 1532590052
  }
}

What I want to end up with is:

{
  "Id": "bec7bfaa-7a47-4f61-a463-5966a2b5c8ce",
  "DataId": "a33-a3c3-4ad5-9e06-d05063474d8c",
  "EtaToStore": 156,
  "CreatedAt": 1532590052
}

I'm sure there's an easy way to do this but I can't figure it out, help!

Answer 1

You can implement the json.Unmarshaler interface to add a custom method to unmarshal the json. Then in that method, you can use the nested struct format, but return the flattened one at the end.

func (es *EventStruct) UnmarshalJSON(data []byte) error {
    // define private models for the data format

    type driverInner struct {
        EtaToStore int `bson:"etaToStore"`
    }

    type dataInner struct {
        ID        string      `bson:"_id" json:"_id"`
        Driver    driverInner `bson:"driver"`
        CreatedAt int         `bson:"createdAt"`
    }

    type nestedEvent struct {
        ID   string    `bson:"_id"`
        Data dataInner `bson:"data"`
    }

    var ne nestedEvent

    if err := json.Unmarshal(data, &ne); err != nil {
        return err
    }

    // create the struct in desired format
    tmp := &EventStruct{
        ID:         ne.ID,
        DataID:     ne.Data.ID,
        EtaToStore: ne.Data.Driver.EtaToStore,
        CreatedAt:  ne.Data.CreatedAt,
    }

    // reassign the method receiver pointer
    // to store the values in the struct
    *es = *tmp
    return nil
}

Runnable example: https://play.golang.org/p/83VHShfE5rI

Answer 2

This question is a year and a half old, but I ran into it today while reacting to an API update which put me in the same situation, so here's my solution (which, admittedly, I haven't tested with bson, but I'm assuming the json and bson field tag reader implementations handle them the same way)

Embedded (sometimes referred to as anonymous) fields can capture JSON, so you can compose several structs into a compound one which behaves like a single structure.

{
    "_id" : "bec7bfaa-7a47-4f61-a463-5966a2b5c8ce",
    "data" : {
        "driver" : {
            "etaToStore" : 156
        },
        "createdAt" : 1532590052,
        "_id" : "07703a33-a3c3-4ad5-9e06-d05063474d8c"
    }
} 

type DriverStruct struct {
    EtaToStore          string      `bson:"etaToStore"`

type DataStruct struct {
    DriverStruct                    `bson:"driver"`
    DataId              string      `bson:"_id"`
    CreatedAt           int         `bson:"createdAt"`
}

type EventStruct struct {
    DataStruct                      `bson:"data"`
    Id                  string      `bson:"_id"`
}

You can access the nested fields of an embedded struct as though the parent struct contained an equivalent field, so e.g. EventStructInstance.EtaToStore is a valid way to get at them.

Benefits:

• You don't have to implement the Marshaller or Unmarshaller interfaces, which is a little overkill for this problem
• Doesn't require any copying fields between intermediate structs
• Handles both marshalling and unmarshalling for free

Read more about embedded fields here.

Answer 3

You can use basically the same logic as:

package utils

// FlattenIntegers flattens nested slices of integers
func FlattenIntegers(slice []interface{}) []int {
    var flat []int

    for _, element := range slice {
        switch element.(type) {
        case []interface{}:
            flat = append(flat, FlattenIntegers(element.([]interface{}))...)
        case []int:
            flat = append(flat, element.([]int)...)
        case int:
            flat = append(flat, element.(int))
        }
    }

    return flat
}

(Source: https://gist.github.com/Ullaakut/cb1305ede48f2391090d57cde355074f)

By adapting it for what's in your JSON. If you want it to be generic, then you'll need to support all of the types it can contain.