How to FIFO Order a Map during Unmarshalling

Question

I know from reading around that Maps are intentionally unordered in Go, but they offer a lot of benefits that I would like to use for this problem I'm working on. My question is how might I order a map FIFO style? Is it even worth trying to make this happen? Specifically I am looking to make it so that I can unmarshal into a set of structures hopefully off of an interface.

I have:

type Package struct {
    Account   string
    Jobs      []*Jobs
    Libraries map[string]string
}

type Jobs struct {
// Name of the job
     JobName string `mapstructure:"name" json:"name" yaml:"name" toml:"name"`
// Type of the job. should be one of the strings outlined in the job struct (below)
     Job *Job `mapstructure:"job" json:"job" yaml:"job" toml:"job"`
// Not marshalled
     JobResult string
// For multiple values
     JobVars []*Variable
}

type Job struct {
// Sets/Resets the primary account to use
    Account *Account `mapstructure:"account" json:"account" yaml:"account" toml:"account"`
// Set an arbitrary value
    Set *Set `mapstructure:"set" json:"set" yaml:"set" toml:"set"`
// Contract compile and send to the chain functions
    Deploy *Deploy `mapstructure:"deploy" json:"deploy" yaml:"deploy" toml:"deploy"`
// Send tokens from one account to another
    Send *Send `mapstructure:"send" json:"send" yaml:"send" toml:"send"`
// Utilize eris:db's native name registry to register a name
    RegisterName *RegisterName `mapstructure:"register" json:"register" yaml:"register" toml:"register"`
// Sends a transaction which will update the permissions of an account. Must be sent from an account which
// has root permissions on the blockchain (as set by either the genesis.json or in a subsequence transaction)
    Permission *Permission `mapstructure:"permission" json:"permission" yaml:"permission" toml:"permission"`
// Sends a bond transaction
    Bond *Bond `mapstructure:"bond" json:"bond" yaml:"bond" toml:"bond"`
// Sends an unbond transaction
    Unbond *Unbond `mapstructure:"unbond" json:"unbond" yaml:"unbond" toml:"unbond"`
// Sends a rebond transaction
    Rebond *Rebond `mapstructure:"rebond" json:"rebond" yaml:"rebond" toml:"rebond"`
// Sends a transaction to a contract. Will utilize eris-abi under the hood to perform all of the heavy lifting
    Call *Call `mapstructure:"call" json:"call" yaml:"call" toml:"call"`
// Wrapper for mintdump dump. WIP
    DumpState *DumpState `mapstructure:"dump-state" json:"dump-state" yaml:"dump-state" toml:"dump-state"`
// Wrapper for mintdum restore. WIP
    RestoreState *RestoreState `mapstructure:"restore-state" json:"restore-state" yaml:"restore-state" toml:"restore-state"`
// Sends a "simulated call" to a contract. Predominantly used for accessor functions ("Getters" within contracts)
    QueryContract *QueryContract `mapstructure:"query-contract" json:"query-contract" yaml:"query-contract" toml:"query-contract"`
// Queries information from an account.
    QueryAccount *QueryAccount `mapstructure:"query-account" json:"query-account" yaml:"query-account" toml:"query-account"`
// Queries information about a name registered with eris:db's native name registry
    QueryName *QueryName `mapstructure:"query-name" json:"query-name" yaml:"query-name" toml:"query-name"`
// Queries information about the validator set
    QueryVals *QueryVals `mapstructure:"query-vals" json:"query-vals" yaml:"query-vals" toml:"query-vals"`
// Makes and assertion (useful for testing purposes)
    Assert *Assert `mapstructure:"assert" json:"assert" yaml:"assert" toml:"assert"`
}

What I would like to do is to have jobs contain a map of string to Job and eliminate the job field, while maintaining order in which they were placed in from the config file. (Currently using viper). Any and all suggestions for how to achieve this are welcome.

Related / possible duplicate of [Why can't Go iterate maps in insertion order?](http://stackoverflow.com/questions/28930416/why-cant-go-iterate-maps-in-insertion-order); and [Golang map in order range loop](http://stackoverflow.com/questions/39450120/golang-map-in-order-range-loop). — icza, Nov 04 '16 at 00:20

score 4 · Accepted Answer · edited May 23 '17 at 12:02

4

You would need to hold the keys in a separate slice and work with that.

type fifoJob struct {
    m map[string]*Job
    order []string
    result []string
    // Not sure where JobVars will go. 
}

func (str *fifoJob) Enqueue(key string, val *Job) {
    str.m[key] = val
    str.order = append(str.order, key)
}

func (str *fifoJob) Dequeue() {
    if len(str.order) > 0 {
        delete(str.m, str.order[0])
        str.order = str.order[1:]
    }
}

Anyways if you're using viper you can use something like the fifoJob struct defined above. Also note that I'm making a few assumptions here.

type Package struct {
    Account   string
    Jobs      *fifoJob
    Libraries map[string]string
}

var config Package
config.Jobs = fifoJob{}
config.Jobs.m = map[string]*Job{}

// Your config file would need to store the order in an array.
// Would've been easy if viper had a getSlice method returning []interface{}
config.Jobs.order = viper.GetStringSlice("package.jobs.order") 

for k,v := range viper.GetStringMap("package.jobs.jobmap") {

    if job, ok := v.(Job); ok {
        config.Jobs.m[k] = &job
    }
}

for

PS: You're giving too many irrelevant details in your question. I was asking for a MCVE.

edited May 23 '17 at 12:02

Community

1
1

answered Nov 03 '16 at 23:55

John S Perayil

6,001
1
32
47

Thanks for the quick response. It would appear that my hope of making something easily unmarshal-able for a json list that would enable iterable mapping by FIFO is not going to be easy and probably best to ignore atm. But thanks. This helped a lot. – Richard John Catalano Nov 04 '16 at 21:06
It would be possible if you implement the `json.Unmarshaler` interface. But how easy it would be, depends on your exact problem. – John S Perayil Nov 05 '16 at 03:44
2

Note that if your map is big, then maintaining a sorted list of keys will slow things down. I hit this problem with a hashmap on the order of 10M entries and had to rethink this approach. In the end, a combined hashmap and queue data structure worked like a charm. – johnzachary Nov 06 '16 at 21:13
right but I'm basically taking a list of jobs from a yaml/json file and trying to keep them in order and this appears to make my job more difficult, not less. – Richard John Catalano Nov 07 '16 at 19:47
1

@RichardJohnCatalano If you can share the structure, I can see if implimenting the `json.Unmarshaler` interface can solve your problem. – John S Perayil Nov 08 '16 at 02:29

score 0 · Answer 2 · answered Nov 03 '16 at 23:58

Maps are by nature unordered but you can fill up a slice instead with your keys. Then you can range over your slice and sort it however you like. You can pull out specific elements in your slice with [i].

Check out pages 170, 203, or 204 of some great examples of this:

Programming in Go

How to FIFO Order a Map during Unmarshalling

2 Answers2