What are the best practices to add metadata to a RESTful JSON response?

Question

Background

We are building a Restful API that should return data objects as JSON. In most of the cases it fine just to return the data object, but in some cases, f.ex. pagination or validation, we need to add some metadata to the response.

What we have so far

We have wrapped all json responses like this example:

{
    "metadata" :{
        "status": 200|500,
        "msg": "Some message here",
        "next": "http://api.domain.com/users/10/20"
        ...
    },
    "data" :{
        "id": 1001,
        "name": "Bob"
    }
}

Pros

• We can add helpful metadata to the response

Cons

• In most cases we don't need the metadata field, and it adds complexity to the json format
• Since it's not a data object any more, but more like a enveloped response, we can not use the response right away in f.ex backbone.js without extracting the data object.

Question

What is the best practices to add metadata to a json response?

UPDATE

What I've got so far from answers below:

• Remove the metadata.status an return the http response code in the http protocol instead (200, 500 ...)
• Add error msg to body of an http 500 repsonse
• For pagination i natural to have some metadata telling about the pagination structure, and the data nested in that structure
• Small amount of meta data can be added to http header (X-something)

Answer 1

You have several means to pass metadata in a RESTful API:

1. Http Status Code
2. Headers
3. Response Body

For the metadata.status, use the Http Status Code, that's what's for! If metadata is refers to the whole response you could add it as header fields. If metadata refers only to part of the response, you will have to embed the metadata as part of the object.DON'T wrap the whole response in an artifical envelope and split the wrapper in data and metadata.

And finally, be consistent across your API with the choices you make.

A good example is a GET on a whole collection with pagination. GET /items You could return the collection size, and current page in custom headers. And pagination links in standard Link Header:

Link: <https://api.mydomain.com/v1/items?limit=25&offset=25>; rel=next

The problem with this approach is when you need to add metadata referencing specific elements in the response. In that case just embed it in the object itself. And to have a consistent approach...add always all metadata to response. So coming back to the GET /items, imagine that each item has created and updated metadata:

{
  items:[
    {
      "id":"w67e87898dnkwu4752igd",
      "message" : "some content",
      "_created": "2014-02-14T10:07:39.574Z",
      "_updated": "2014-02-14T10:07:39.574Z"
    },
    ......
    {
      "id":"asjdfiu3748hiuqdh",
      "message" : "some other content",
      "_created": "2014-02-14T10:07:39.574Z",
      "_updated": "2014-02-14T10:07:39.574Z"
    }
  ],
  "_total" :133,
  "_links" :[
     {
        "next" :{
           href : "https://api.mydomain.com/v1/items?limit=25&offset=25"
         } 
   ]
}

Note that a collection response is an special case. If you add metadata to a collection, the collection can no longer be returned as an array, it must be an object with an array in it. Why an object? because you want to add some metadata attributes.

Compare with the metadata in the individual items. Nothing close to wrapping the entity. You just add some attributes to the resource.

One convention is to differentiate control or metadata fields. You could prefix those fields with an underscore.

Answer 2

Along the lines of @Charlie's comment: for the pagination part of your question you still need to bake the metadata into the response somhow, but the status and message attributes here are somewhat redundant, since they are already covered by the HTTP protocol itself (status 200 - model found, 404 - model not found, 403 - insufficient privs, you get the idea) (see spec). Even if your server returns an error condition you can still send the message part as the response body. These two fields will cover quite much of your metadata needs.

Personally, I have tended towards (ab)using custom HTTP headers for smaller pieces of metadata (with an X- prefix), but I guess the limit where that gets unpractical is pretty low.

I've expanded a bit about this in a question with a smaller scope, but I think the points are still valid for this question.

Answer 3

I suggest you to read this page https://www.odata.org/ You are not forced to use OData but the way they do the work is a good example of good practice with REST.

Answer 4

How about returning directly the object that you want in data, like return:

{
    "id": 1001,
    "name": "Bob"
}

And return in headers the metadata.

Option 1 (one header for all metadata JSON):

X-METADATA = '{"status": 200|500,"msg": "Some message here","next": "http://api.domain.com/users/10/20"...}'

Option 2 (one header per each metadata field):

X-METADATA-STATUS = 200|500
X-METADATA-MSG = "Some message here",
X-METADATA-NEXT = "http://api.domain.com/users/10/20"
...

Until now I was using like you, a complex JSON with two fields, one for data and one for metadata. But I'm thinking in starting using this way that I suggested, I think it will be more easy.

---

Remind that some server have size limit for HTTP headers, like this example: https://www.tutorialspoint.com/What-is-the-maximum-size-of-HTTP-header-values

Answer 5

JSON:API solves this by defining top-level meta and data properties.

Answer 6

We had the same use case, in which we needed to add pagination metadata to a JSON response. We ended up creating a collection type in Backbone that could handle this data, and a lightweight wrapper on the Rails side. This example just adds the meta data to the collection object for reference by the view.

So we created a Backbone Collection class something like this

// Example response:
// { num_pages: 4, limit_value: 25, current_page: 1, total_count: 97
//   records: [{...}, {...}] }

PageableCollection = Backbone.Collection.extend({
  parse: function(resp, xhr)  {
    this.numPages = resp.num_pages;
    this.limitValue = resp.limit_value;
    this.currentPage = resp.current_page;
    this.totalCount = resp.total_count;
    return resp.records;
  }  
});

And then we created this simple class on the Rails side, to emit the meta data when paginated with Kaminari

class PageableCollection
  def initialize (collection)
    @collection = collection
  end
  def as_json(opts = {})
    {
      :num_pages => @collection.num_pages 
      :limit_value => @collection.limit_value 
      :current_page => @collection.current_page,
      :total_count => @collection.total_count
      :records => @collection.to_a.as_json(opts)
    }
  end
end

You use it in a controller like this

class ThingsController < ApplicationController
  def index 
    @things = Thing.all.page params[:page]
    render :json => PageableCollection.new(@things)
  end
end

Enjoy. Hope you find it useful.