JMESPath date filtering

Question

I am trying to convert an Ansible script to a Python AWS lambda function. In my Python script I am using the jmespath library to filter by date, which is given as a string in ISO 8601 format.

My filter works in my Ansible script and also using the JMESPath website tool. I cannot figure out why my Python version does not work; Ansible is written Python so I expected it to work the same way.

This works in Ansible:

 - name: parse groups
      debug:
        msg: "{{ results_gitlabGroupsProjects | to_json | from_json  | json_query(projects_query) }}"
      vars:
        projects_query: "json[?last_activity_at > `{{gitlab_date}}`].{name: name, id: id, last_activity_at: last_activity_at }"
      register: gitlabGroupsProjects2

When I try to do the same thing in Python, I get an empty list, []:

compareTime="2020-01-15T17:55:3S.390Z"
plist2 = jmespath.search('[?last_activity_at > `str(compareTime)`]', project_data )
with open('plist2.json', 'w') as json_file:
    json.dump(plist2, json_file)

Sample JSON data:

[
  {
    "name": "test",
    "id": 16340975,
    "last_activity_at": "2020-01-15T20:12:49.775Z"
  },
  {
    "name": "test1",
    "id": 11111111,
    "last_activity_at": "2020-01-15T15:57:29.670Z"
  },
  {
    "name": "test2",
    "id": 222222,
    "last_activity_at": "2020-01-15T23:08:22.313Z"
  },
  {
    "name": "test3",
    "id": 133333,
    "last_activity_at": "2020-01-15T22:28:42.628Z"
  },
  {
    "name": "test4",
    "id": 444444,
    "last_activity_at": "2020-01-14T02:20:47.496Z"
  },
  {
    "name": "test5",
    "id": 555555,
    "last_activity_at": "2020-01-13T04:54:18.353Z"
  },
  {
    "name": "test6",
    "id": 66666666,
    "last_activity_at": "2020-01-12T07:12:05.858Z"
  },
  {
    "name": "test7",
    "id": 7777777,
    "last_activity_at": "2020-01-10T20:52:32.269Z"
  }
]

Using Ansible, and on the JMESPath website I get this output:

[
  {
    "name": "test",
    "id": 16340975,
    "last_activity_at": "2020-01-15T20:12:49.775Z"
  },
  {
    "name": "test2",
    "id": 222222,
    "last_activity_at": "2020-01-15T23:08:22.313Z"
  },
  {
    "name": "test3",
    "id": 133333,
    "last_activity_at": "2020-01-15T22:28:42.628Z"
  }
]

Martijn Pieters · Accepted Answer · 2020-01-16T21:40:32.393

You can't just put a str(compareTime) expression in as string literal and have Python understand that that's what you wanted to replace.

In the string

'[?last_activity_at > `str(compareTime)`]'

nothing is dynamic; the characters str(compareTime) do not carry special meaning to Python.

In Ansible, you used special syntax, {{gitlab_date}}, to tell Ansible to treat the string differently, and the value of the gitlab_date variable is placed into the string at that point, before using it as a JMESPath query.

In Python, you need to something similar. E.g. you could use a formatted string literal, or f-string:

plist2 = jmespath.search(f"[?last_activity_at > `{compareTime}`]", project_data)

The f before the string literal tells Python to look for any expressions between {...} curly braces, so compareTime is slotted into place at that point of the string. This is a lot like the Ansible syntax.

The above produces your expected output:

>>> jmespath.search(f"[?last_activity_at > `{compareTime}`]", project_data)
[{'name': 'test', 'id': 16340975, 'last_activity_at': '2020-01-15T20:12:49.775Z'}, {'name': 'test2', 'id': 222222, 'last_activity_at': '2020-01-15T23:08:22.313Z'}, {'name': 'test3', 'id': 133333, 'last_activity_at': '2020-01-15T22:28:42.628Z'}]
>>> from pprint import pprint
>>> pprint(_)
[{'id': 16340975,
  'last_activity_at': '2020-01-15T20:12:49.775Z',
  'name': 'test'},
 {'id': 222222,
  'last_activity_at': '2020-01-15T23:08:22.313Z',
  'name': 'test2'},
 {'id': 133333,
  'last_activity_at': '2020-01-15T22:28:42.628Z',
  'name': 'test3'}]

The f string did fix my current issue and the filter does work in this case. There is one challenge with f string when it comes to additional filtering. jmespath does uses curly brackets for advanced filter such as: [?last_activity_at > `{compareTime}`].{name: name, id: id, last_activity_at: last_activity_at } but you could break it up in two lines of code. Using the above solution then do [].{name: name, id: id, last_activity_at: last_activity_at }. — modevops, Jan 17 '20 at 15:48
@modevops three options: double literal braces (`{{` and `}}` become single braces in the final string), or use [another template option](https://realpython.com/python-string-formatting/); the [printf-style formatting option](https://docs.python.org/3/library/stdtypes.html#printf-style-string-formatting) doesn’t rely on curly braces, for example. Or use string concatenation: `(f"[?last_activity_at > \`{compareTime}\`]" ".{name: name, id: id, last_activity_at: last_activity_at}")` uses two strings, only one of which is as f-string, but produces one value. — Martijn Pieters, Jan 17 '20 at 21:42

JMESPath date filtering

1 Answers1

Linked