Robust way check if the input is JSON or YAML

Question

In code below should accept a multiline as string input as either JSON or YAML. It firsts attempts to read the input as JSON, and if JSON failed it makes a second attempt to read it as YAML if both failed return error.

Now the problem is with yaml.Unmarshal(). I check, it never returns an error if the input is JSON string. (correct or incorrect). The main issue yaml.Unmarshal never returns an error.

Initially, I thought it error on yaml.Unmarshal implementation, but it looks to me it makes the best effort when parsing input, and structure doesn't violate yaml; it never returns an error.

func SpecsFromString(str string) (*Something, error) {
    r := strings.NewReader(str)
    return ReadTenantsSpec(r)
}

func ReadSpec(b io.Reader) (*Spec, error) {

    var spec Spec

    buffer, err := ioutil.ReadAll(b)
    if err != nil {
        return nil, err
    }

    err = json.Unmarshal(buffer, &spec)
    if err == nil {
        return &spec, nil
    }

    err = yaml.Unmarshal(buffer, &spec)
    if err == nil {
        return &spec, nil
    }

    return nil, &InvalidTenantsSpec{"unknown format"}
}

So my question how to properly do this test if the input is JSON or YAML? It looks to me that the only way to do that on JSON unmashler differentiates two error cases. The reason is JSON generally more strict in structure. When the input is not JSON at all, and when input is JSON but an error in the structure of JSON. That second case will allow me never to call the yaml parser in the first place.

Maybe someone comes up with a more neat solution?

Thank you.

Answer 1

json.Unmarshal does return SyntaxError on invalid JSON syntax and has other, different errors when the syntax is correct but unmarshaling fails, so you can use that to differentiate.

Concerning YAML, if you use yaml.v3, you can write a custom unmarshaler to access the Node representation of your input, and check whether the root node has the Style Flow set, which means JSON-like syntax. However, YAML is far more permissive even with this syntax (e.g. strings do not need to be quoted, trailing commas in sequences and mappings are allowed) and while you can check the quoting style of contained scalars, the information available will not be enough to ensure that the input is JSON-parseable (trailing commas cannot be detected via this interface).

So the proper way to check whether the input is syntactically valid JSON is to check the returned error of json.Unmarshal.

Answer 2

This is what I was referencing in my comment on the question. This is a simplistic example of a middleware reader.

1. This pattern allows you to avoid having to fully parse the text body, in case it's unreasonably large
2. It ideally has no effect on downstream operations, providing a transparent API.

From your example, you'd call something like:

b = WrapReader(b)
buffer, err := ioutil.ReadAll(b)
if err != nil {return nil, err}
if b.Writable.A > b.Writable.B {
    err = json.Unmarshal(buffer, &spec)
}
if err != nil || b.Writable.A <= b.Writable.B {
    err = yaml.Unmarshal(buffer, &spec)
}

Effectively, it doesn't change the interface you're dealing with, while gaining some control over how the process goes down. There's plenty of room for improvement, but the above API is offered by the code below:

type Line []byte
type Writable struct {
    Line
    A int
    B int
}
type Decision struct{
    io.Reader
    Writable
}
func (d *Decision) Read(b_rx []byte) (int, error) {
    n, err := d.Reader.Read(b_rx)
    if err != nil && err != io.EOF {return n, err}
    for _, b_tx := range b_rx {
        d.Writable.WriteByte(b_tx)
    }
    return n, nil       
}
func (w *Writable) WriteByte (b byte) error {
    if b == '\n' {
        pJSON, pYAML, err := w.Score()
        if err != nil {return err}
        w.A += pJSON
        w.B += pYAML
        w.Line = make(Line, 0)
    } else {
        w.Line = append(w.Line, b)
    }
    return nil
}
func (w *Writable) Score () (int, int, error) {
    //whatever scoring heuristics you can think of.
    return 0,0,nil
}
func WrapReader(b io.Reader) io.Reader {
    return Decision{b,*new(Writable)}
}

Answer 3

I came across the same problem some days ago in bash scripting: How can I detect if a file contains json, yaml or plain text?

My solution was:

process as json

• can be parsed as json without errors

process as text

• can be parsed as yaml, but type is just a yaml string

process as yaml

• can be parsed as yaml, but is not just a yaml string
• cannot be parsed as json

Bash scripting snippet

parse_as_json() {
  jq -e '.' > /dev/null 2>&1 < "$1"
}

parse_as_yaml() {
  local FILE=$1
  parse_as_json $FILE && return 1
  parse_as_text $FILE && return 1
  yq -e > /dev/null 2>&1 < $FILE || return 1
}

parse_as_text() {
  [[ $(yq 'type == "string"' 2>&1 < $1) == true ]]
}