2

Currently I am running this in a http handler:

err := mongoCollection.Find(bson.M{"name": vars["name"]}).One(&result)
data, err := json.Marshal(result)
w.Write(data)

How can I begin serving the result before the full BSON data is in?

Edit: the answer needs to go beyond the mgo extension and go into bson. mgo as far as I can see will only serve full documents if I not mistaken. I have one - possibly large - document as my code example clearly shows.

chx
  • 11,270
  • 7
  • 55
  • 129

4 Answers4

3

In order for this to be possible, you would need these things:

  1. Access to a Reader for the incoming bson stream
  2. A datatype for generic document parts
  3. A streaming decoder for bson which reads from the Reader and produces document parts
  4. A streaming encoder for json that consumes document parts and writes to a Writer

mgo does not provide number 1. encoding/json does not provide number 2 or 4. mgo/bson does not provide number 3. A bit of googling doesn't turn up any help for any of those points in Go, though there are streaming json parsers in other languages (see answers for Is there a streaming API for JSON?).

Your desire to do this is reasonable, but the support just doesn't exist yet. Fortunately, json and bson are simple enough and all the components you're using are open source, so in theory you could write the tools you need.

Community
  • 1
  • 1
Sean
  • 1,785
  • 10
  • 15
  • That's weird; isn't http://golang.org/pkg/encoding/json/#NewDecoder a streaming JSON decoder? – chx May 10 '14 at 20:13
  • @chx Yes, that is a half-streaming decoder in the sense that the input bytes are consumed in chunks. However, it does not stream its output, the resulting document, in chunks. The output is stored by `Decoder.Decode(v interface{}) error` into a result structure in one call. You need something like `bson.Decoder.NextPart() DocPart` which you can pass to `json.Encoder.WritePart(p DocPart)` one `DocPart` at a time, so the full `v interface{}` is never constructed. – Sean May 11 '14 at 01:04
1

I don't think there's anything you can do to avoid unmarshalling the whole BSON (and therefore not serving the result until the BSON has been fully delivered by mgo), short of hacking on mgo. Its API only deals in whole, unmarshalled documents, with no access to any BSON-encoded []byte or Reader that you could potentially elementwise bsondecode-then-jsonencode as data comes in.

Steve M
  • 8,246
  • 2
  • 25
  • 26
  • Erm. My edit which is days old says so "Edit: the answer needs to go beyond the mgo extension and go into bson." – chx May 07 '14 at 20:07
  • @chx I know, but the bson package can't do anything for you. It can't reach inside of the mgo package and stop it from unmarshalling the document. – Steve M May 07 '14 at 20:09
  • but can it provide the bson wire decoding necessary to talk to the server raw? – chx May 08 '14 at 00:14
0

Take a look at chanson, you can easily construct and stream json. There's an example that reads data from channels for adding elements to a list. You could probably do something similar

Gustavo Chaín
  • 903
  • 2
  • 10
  • 16
-1

Take a look at json.Encoder. It writes JSON objects to an output stream. json.Marshal produces a []byte in one shot and does not provide a stream.

On the MongoDB side take a look at mgo.Iter. In case you have a large number of documents in your result you can serialize them in batches and make your application more memory efficient.

Sample of using json.Encode:

data := map[string]int{"apple": 5, "lettuce": 7}
enc := json.NewEncoder(w)
enc.Encode(data)

Play

Sebastian
  • 16,813
  • 4
  • 49
  • 56
  • This is only helpful if you have more than one document; and even so the example wouldn't be enough to actually put it together :/ – chx May 04 '14 at 14:54
  • `json.Encode` will stream even a single document to the client. So it should solve your problem. The sample was meant to give you a pointer in the right direction and not do all the required coding. – Sebastian May 04 '14 at 15:12
  • `mgo.Iter` will give you a single document, the question really is, how to invoke `bson` to get you the resulting bytestream such that `json.NewEncoder` can stream? – chx May 04 '14 at 16:09
  • @Sebastian This answer misunderstood the question. The question is about streaming all the way from incoming bytes in bson to outgoing bytes in json without ever stalling to wait for a complete document. – Sean May 10 '14 at 15:52