Django queryset not returning all rows

Question

Hi I have below setup.

Django 1.9, mongoDB, pymongo 2.7, mongoengine 0.9

I have written an API to store logs at backend server userwise. Below is sample of table

user  subject  registered  changed_on
abc    eng      Y          "2018-04-18T00:00:00Z"
abc    maths    N          "2018-04-18T00:10:00Z"
xyz    eng      Y          "2018-04-18T00:10:00Z"

I also have read API for this in which we give user name and timestamp for filter like below:

{
"user" : "abc",
"from_date" : "2018-04-18T00:00:00Z"
}

The line in serializers.py which is applying filter is

Logs.objects.filter(user__iexact='abc',changed_on__gte=from_date)

Now Sometimes when I add new log and retrieve it from postman, it is not working. I have to restart django server and then it gives me newly added row.

I dont understand why this is happening.

EDIT1 : Full Serializer.py

from rest_framework import serializers

class GetUserLogs(serializers.Serializer):
    user = serializers.CharField(label=_("USER"))
    token = serializers.CharField(label=_("Token"))
    from_date = serializers.CharField(label=_('From date'), default="")
    till_date = serializers.CharField(label=_('Till date'), default=datetime.datetime.now().isoformat().split(".")[0]+'Z')
    def validate(self, attrs):
        user = attrs.get('user')
        token = attrs.get('token')
        from_date = attrs.get('from_date')          
        if user:
            tokendetails = validate_token(token)
            if not tokendetails:
                msg = _('Invalid token.')
                raise serializers.ValidationError(msg)
            else:
                userdetails = tokendetails.user
                if userdetails.check_user(user):
                    rows = Logs.objects.all().filter(user__iexact=user,changed_on__gte=from_date, changed_on__lte = till_date)
                    print(len(rows))
                else:
                    msg = _('Invalid USER)
                    raise serializers.ValidationError(msg)
        else:
            msg = _('Must include "USER".')
            raise serializers.ValidationError(msg)

        attrs['rows'] = rows
        return attrs

Try to add `Logs.objects.all().filter(user__iexact='abc',changed_on__gte=from_date)` also define the amount of data. Maybe there is any limitation. — Panos Angelopoulos, Apr 18 '18 at 08:13
define amount of data how? I am putting from date as optional... but maximum we are going to keep 1 month data in db — Pratibha, Apr 18 '18 at 08:15
Let us know the result of `len(Logs.objects.all().filter(user__iexact='abc',changed_on__gte=from_date))` — Panos Angelopoulos, Apr 18 '18 at 08:18
You need to show the full context of that filter line. If you have to restart your server to see new data, that means you're doing the query at module or class level rather than inside a method. — Daniel Roseman, Apr 18 '18 at 08:26
May be caching issue, you may have used, it is getting, that's why whenever you restart server, it works. — Anup Yadav, Apr 18 '18 at 09:04
Not without showing your code. Do that and I'll explain what's wrong with it. — Daniel Roseman, Apr 18 '18 at 09:10
Then another issue could be of POSTMAN, sometimes postman goes into the cache, I would suggest use DRF's API interface. Or check through API client code. — Anup Yadav, Apr 18 '18 at 09:16
You *are* using caching, albeit inadvertently. As I said, you have something at module or class level. You need to *show your code*. — Daniel Roseman, Apr 18 '18 at 09:19
@AnupYadav, UI si giving error which we are confirming from postman. — Pratibha, Apr 18 '18 at 09:26
@DanielRoseman, what minimum code can I show you for your help. Its a secure project — Pratibha, Apr 18 '18 at 09:26
@DanielRoseman, if its ok, we can get on teamviewer session. — Pratibha, Apr 18 '18 at 09:30
@pratibha If your project is secure you can always put a fake model names and fields in your example. — T.Tokic, Apr 18 '18 at 09:39
Where is `till_date` defined in that filter? It's not referring to any field value, is there a global definition somewhere? — Daniel Roseman, Apr 18 '18 at 10:34
@DanielRoseman,from_date and til_date are filters on date column i.e. to get data between 2 dates — Pratibha, Apr 18 '18 at 10:45
No. Your filter says `changed_on__lte = till_date`. There `till_date` is a variable, which you have defined somewhere. — Daniel Roseman, Apr 18 '18 at 11:38
@DanielRoseman, Can you please help me in another question [here](https://stackoverflow.com/questions/49852738/django-app-ssl-socket-connection-to-firmware) — Pratibha, Apr 20 '18 at 17:06

score 0 · Answer 1 · answered Apr 18 '18 at 11:52

So After a lot of debugging for almost a day, I found that the default value of till_date that we are setting to current timestamp of system is not increasing as time increase. So I changed default value to empty string and added below code after reading value of till_date.

if till_date == "":
    till_date = datetime.datetime.now().isoformat().split(".")[0]+'Z'

It will reset the till_date value to current timestamp every time its value is missing.

Hope it will help someone someday.

Django queryset not returning all rows

1 Answers1