How to determine if some fields are missing in a dictionary with Python?

Question

I am working with a list of dictionaries and I am trying to verify which keys from those dictionaries contain empty values. The data structure that I have is like this:

filtered_data = 
[{'id': 1021972, 'Aging_Deferred_Transferred': '', 'Aging_Open_Issue': '0.94', 'Aging_Un_investigated_Issue': '0.94', 'User': 'John P.', 'ModifiedOn': '2017-09-04 21:29:59', 'Open_date':'2017-08-04 01:34:18', 'End_date': '2017-09-05 00:29:01', 'Ticket_status':'Transferred'},
 {'id': 1036722, 'Aging_Deferred_Transferred': '', 'Aging_Open_Issue': '0.12', 'Aging_Un_investigated_Issue': '0.01', 'User': 'John P.', 'ModifiedOn': '2017-09-04 21:29:59', 'Open_date':'2017-09-01 00:34:18', 'End_date': '', 'Ticket_status':'Researching'},
 {'id': 1015621, 'Aging_Deferred_Transferred': '', 'Aging_Open_Issue': '0.99', 'Aging_Un_investigated_Issue': '0.11', 'User': 'John D.', 'ModifiedOn': '2017-06-05 12:19:59', 'Open_date':'2017-01-01 00:00:18', 'End_date': '2017-09-01 20:20:57', 'Ticket_status':'Closed'}]

The mandatory keys that I want to verify if they contain data are the following:

mandatory_keys = ['id','User','Ticket_status']

And currently I have this code that reads each dictionary from the list and appends only those where the Open_date is contained between the beginning_date_format and ending_date_format.

list_data = []
for d in filtered_data:
    list_data.append({k: d[k] for k in mandatory_keys if beginning_date_format
                                          <= dateutil.parser.parse(d.get('Open_date'))
                                          < ending_date_format})

I want to add those dictionaries where the Open_date is contained between beginning and ending dates and where any of the keys from mandatory_keys contain no values " ".

How can I implement my solution? Any thoughts, comments and suggestions are welcome.

EDIT:

beginning_date_format = '2017-08-01 00:00:00' 
ending_date_format = '2017-09-05 00:29:01'

EDIT:

This is the solution I found.

list_data = []
for d in filtered_data:
    list_data.append({k: d[k] for k in mandatory_keys if beginning_date_format
                                          <= dateutil.parser.parse(d.get('Open_date'))
                                          < ending_date_format and (d.get('id')=='' 
                                 or d.get('User')=='' or d.get('Ticket_status')=='')})

Based on Return none if dictionary key is not available

I'm a bit confused by the question. What happens if you put `if d.get('Open_date') and` immediately after `for k in mandatory_fields`? — roganjosh, Sep 07 '17 at 19:03
I did some edits, please disregard `mandatory_fields`. `d.get(Open_date)` gets the value and analyzes if it is contained between `beginning_date_format` and `ending_date_format` — abautista, Sep 07 '17 at 19:05
My understanding of the question is that you want to stop an empty string of `Open_date` from crashing your code? — roganjosh, Sep 07 '17 at 19:07
I want to add those dictionaries with empty strings that contain no data in the following keys `id`,`User`, `Ticket_status`. — abautista, Sep 07 '17 at 19:10
Maybe we are misunderstanding because of language. So, you want to add `id`, `User` and `Ticket_status` IF `Open_date` is between two dates OR if `Open_date == ''`? — roganjosh, Sep 07 '17 at 19:15
I want to add `id`, `user`, `ticket_status` if `open_date` is between 2 dates and `(id=='' or user=='' or ticket_status =='')` — abautista, Sep 07 '17 at 19:17
I think I understand now. Can you give me `ending_date_format` and `beginning_date_format`? — roganjosh, Sep 07 '17 at 19:18
d.values() returns a list of all values in d, if you tes for "" in d.values it returns true if any of the values are "" and since you already test for other values to be in a specific range there should be no surprise there. — Milos Grujic, Sep 07 '17 at 19:21
`beginning_date_format: 2017-08-01 00:00:00`, `ending_date_format: 2017-09-05 00:00:00` — abautista, Sep 07 '17 at 19:22
What exactly is wrong with the solution you already have in your question? — ekhumoro, Sep 07 '17 at 20:22
I've got this error `TypeError: 'builtin_function_or_method' object is not subscriptable` — abautista, Sep 07 '17 at 20:23
@ekhumoro see latest updated. I found the solution to my problem. — abautista, Sep 07 '17 at 20:25

score 0 · Answer 1 · answered Sep 07 '17 at 19:38

I've struggled a bit to understand the requirement from your question and our dialogue in comments. As far as I can see, this does what you are asking for.

import datetime as dt

filtered_data = [{'id': 1021972, 'Aging_Deferred_Transferred': '', 'Aging_Open_Issue': '0.94', 'Aging_Un_investigated_Issue': '0.94', 'User': 'John P.', 'ModifiedOn': '2017-09-04 21:29:59', 'Open_date':'2017-08-04 01:34:18', 'End_date': '2017-09-05 00:29:01', 'Ticket_status':'Transferred'},
 {'id': 1036722, 'Aging_Deferred_Transferred': '', 'Aging_Open_Issue': '0.12', 'Aging_Un_investigated_Issue': '0.01', 'User': 'John P.', 'ModifiedOn': '2017-09-04 21:29:59', 'Open_date':'2017-09-01 00:34:18', 'End_date': '', 'Ticket_status':'Researching'},
 {'id': 1015621, 'Aging_Deferred_Transferred': '', 'Aging_Open_Issue': '0.99', 'Aging_Un_investigated_Issue': '0.11', 'User': 'John D.', 'ModifiedOn': '2017-06-05 12:19:59', 'Open_date':'2017-01-01 00:00:18', 'End_date': '2017-09-01 20:20:57', 'Ticket_status':'Closed'}]

beginning_date_format = dt.datetime.strptime('2017-08-01 00:00:00',
                                             '%Y-%m-%d %H:%M:%S')
ending_date_format = dt.datetime.strptime('2017-09-05 00:29:01',
                                          '%Y-%m-%d %H:%M:%S')

list_data = []
for d in filtered_data:
    if not d.get('Open_date'): # I'm unsure about this. Remove if not desired.
        list_data.append([d['id'], d['User'], d['Ticket_status']])
    elif (beginning_date_format
          < dt.datetime.strptime(d['Open_date'], '%Y-%m-%d %H:%M:%S')
          < ending_date_format):
        list_data.append([d['id'], d['User'], d['Ticket_status']])

EDIT: This question has been edited that many times in such a short space of time that I think this answer can only serve as an illustration. It still checks for null fields.

Serge · Answer 2 · 2017-09-07T19:53:34.663

bad_data = []
for d in filtered_data:
    try:

        dt = d['Open_date']
        if dt and not all([ d[k] for k in mandatory]) and ( beginning_date_format
                                      <= dt 
                                      < ending_date_format):
            bad_data.append(d))
    except Exception, e:
        logging.exception('some problem', exc_info=True)

Note. Exception catching like that is bad practice just want you get a list even if open date key is absent, or just use your get. But mind that comparing to None fail in Python 3, workaraound could be .get('Open_date', '1940....')

score 0 · Accepted Answer · answered Sep 07 '17 at 20:41

This is the solution that solved my problem:

list_data = []
for d in filtered_data:
    list_data.append({k: d[k] for k in mandatory_keys if beginning_date_format <= 
    dateutil.parser.parse(d.get('Open_date')) < ending_date_format and (d.get('id')=='' 
    or d.get('User')=='' or d.get('Ticket_status')=='')})

How to determine if some fields are missing in a dictionary with Python?

3 Answers3