Big Integer in JSON getting corrupted by Angular but not by CURL?

Question

I have created a Django REST API using Django Rest Framework.

I use 3 different clients to hit this app:

Curl (for debugging purposes)
Swagger Interface (for debugging purposes)
My Angular application

I have a perplexing problem that data returned by the API is being corrupted by Swagger and Angular but not by Curl.

My Django model looks like this:

class MyModel1(CachingMixin, models.Model):
    id = models.BigIntegerField(default=make_id, primary_key=True)
    name = models.CharField(max_length=50, null=False, blank=False,)

The make_id() method referenced above is described here. I recently implemented this change from the standard Django assigned and auto-incremented primary key. As part of that change, I converted id from an IntegerField to a BigIntegerField.

I have some other Django view code (which I have not shown here) that creates an endpoint called GetMyModel1ByName. That endpoint returns a serialized instance of MyModel1. Here is the curl showing what happens when I hit that endpoint. It works perfectly:

$ curl http://localhost:4212/api/getMyModel1ByName/?name=my-name
{"id": 10150133855458395, "name": "my-name"}

Now here is what happens when I hit the same endpoint from Chrome's Developer console:

> $http = angular.element(document.body).injector().get('$http');
> $http.get("http://localhost:4212/api/getMyModel1ByName/?name=my-name").then(function(response) { console.log(JSON.stringify(response.data)) }).catch(function (error) { console.log(error.data) });
{"id":10150133855458396, "name":"my-name"}

As you can see curl reports the ID as 10150133855458395. That's correct. That's what is in the Database. However Angular reports it as 10150133855458396. The final digit is wrong. The difference is 1. It's very surprising!

This is a truly perplexing error. My Django code is so simple that I'm very confident that there is no mistake in it. Instead I feel that change the id field from IntegerField to BigIntegerField might have caused this problem. Why is it happening and what is the solution?? It's ruining the functionality of my application. I'm seeing the same corruption when I hit this endpoint through Swagger.

EDIT: The other questioner is experiencing the same problem I am. It's great to know that I'm not the only one!! However, the answers on that question are not really answers. They don't explain the cause. And they don't tell me how I should solve the issue in my Angular JS code.

Possible duplicate of [JsonResponse from Django - Incorrect values displayed](http://stackoverflow.com/questions/33271356/jsonresponse-from-django-incorrect-values-displayed) — middlestump, Jun 18 '16 at 00:27
@middlestump Thanks! That question is similar to mine. It's good to know I'm not alone. But there is no actual answer to solve my problem there. — Saqib Ali, Jun 18 '16 at 02:11
What if you simply visit `http://localhost:4212/api/getMyModel1ByName/?name=my-name` via a browser? Same error? What if you try the angular code again but take out `JSON.stringify`? — jDo, Jun 19 '16 at 22:05
@SaqibAli Ok, what about the other thing I suggested (via browser, no javascript)? — jDo, Jun 19 '16 at 22:19
Did you try in firefox and IE. the linked question states that what youa re seeing is a chrome extension error. — e4c5, Jun 20 '16 at 00:07
When I simply visit http://localhost:4212/api/getMyModel1ByName/?name=my-name via a browser (Chrome or Firefox), I get the corrupted ID: `{"id":10150133855458396, "name":"my-name"}`. Only curl produces the right answer. PS. It seems to be erratic. Sometimes this bug occurs. Other times it doesn't. I can't figure out which circumstances it occurs in and which circumstances it doesn't. — Saqib Ali, Jun 20 '16 at 07:08
@rfkortekaas this has nothing to do with python. It's purely client side. — e4c5, Jun 20 '16 at 11:10

score 5 · Answer 1 · edited Jun 20 '16 at 10:15

5

You are trying to store a number which is greater than the safe maximum javascript integer value(+/- 9007199254740991).

Your make_id generator should return values with respect to javascript limitations.

For more information read: http://blog.vjeux.com/2010/javascript/javascript-max_int-number-limits.html

edited Jun 20 '16 at 10:15

Moses Koledoye

77,341
8
133
139

answered Jun 20 '16 at 10:10

Yevgeniy Shchemelev

3,601
2
32
39

1

The OP wrote in the comments that *"When I simply visit localhost:4212/api/getMyModel1ByName/?name=my-name via a browser (Chrome or Firefox), I get the corrupted ID: `{"id":10150133855458396, "name":"my-name"}`"* Doesn't this mean that the ID is wrong even when JavaScript isn't involved? – jDo Jun 20 '16 at 10:31
2

curl is the only source you can trust. All browser representations may use javascript. – Yevgeniy Shchemelev Jun 20 '16 at 10:53
True true. Curl and wget are more trustworthy than any browser. I just wouldn't expect a browser to parse a response as JSON without actively asking for it (by coding it). Anyway, it might be what's going on here - maybe via an extension/add-on or some built-in browser magic that automatically attempts to cast numerical values and overflows. The ID should have been a string to start with. – jDo Jun 20 '16 at 11:42
Yes! See also similar question: http://stackoverflow.com/questions/1379934/large-numbers-erroneously-rounded-in-javascript – flup Jun 21 '16 at 05:27

score 4 · Accepted Answer · edited May 23 '17 at 11:55

TLDR

To cut a long story short. Have your view return the response as String.

{"id": "10150133855458395", "name": "my-name"}

The Long Story

Let's narrow this down. In fact you can and should test this without the involvement of django at all. Please save the json you get from curl http://localhost:4212/api/getMyModel1ByName/?name=my-name into the /static/ folder in your django project. Now when you fetch it django is no longer involved in the picture. You can even host this file on some other kind of server. Let's call it hello.json

Now if you type in http://localhost:4212/static/hello.json you should get.

{"id": 10150133855458395, "name": "my-name"}

Update:

If you don't see the JSON data correctly, that means you have the problem described in the Q&A in the first comment. You have a dodgy extension that messes up the JSON. Disable all extensions and try again. Enable them back one by one.

Now let's create a small angular script.

<!DOCTYPE html>
<html>
<script src="http://ajax.googleapis.com/ajax/libs/angularjs/1.4.8/angular.min.js"></script>
<body>

  <div ng-app="myApp" ng-controller="myCtrl">

    <h1>{{myjson}}</h1>

  </div>


<script>
   var app = angular.module('myApp', []);
   app.controller('myCtrl', function($scope, $http) {
   $http.get("/static/hello.json")
     .then(function(response) {
        console.log(response.data)
        $scope.myjson = response.data;
      });
   });
</script>

</body>
</html>

Then saving this as angular.html in your static folder, you can load it in your browser as http://localhost:4212/static/angular.html which shows.

{"id":10150133855458396,"name":"my-name"}

This is definitely not the correct result. However there is no BigIntegerField and there is no django here. The issue is entirely with in the browser and angular. We have managed to narrow the problem. it is entirely within angular as you can see from the response that's retrieved by the browser.

Angular is choking on large numbers. Send them as strings instead.

It's not so much angular that is choking. It's JavaScript. It does not have separate types for integers. It only has a Number type that it stores as a floating point value. — flup, Jun 21 '16 at 05:30
Thank you. I will implement this ASAP. Assuming it works, I'll mark this works correctly, I'll mark this answer correct. — Saqib Ali, Jun 21 '16 at 05:46
Its such a shame that the browser/Javascript/Angular doesn't throw an error or warning when dealing with large integers that it can't grok. — Saqib Ali, Jun 21 '16 at 06:24

Big Integer in JSON getting corrupted by Angular but not by CURL?

2 Answers2

TLDR

The Long Story