validation check - dictionary value types

Question

After converting my csv to dictionary with pandas, a sample of the dictionary will look like this:

[{'Name': '1234', 'Age': 20},
 {'Name': 'Alice', 'Age': 30.1},
 {'Name': '5678', 'Age': 41.0},
 {'Name': 'Bob 1', 'Age': 14},
 {'Name': '!@#$%', 'Age': 65}]

My goal is to do a validation check if the columns are in string. I'm trying to use pandera or schema libs to achieve it as the csv may contain a million rows. Therefore, I am trying to convert the dict to as follows.

[{'Name': 1234, 'Age': 20},
 {'Name': 'Alice', 'Age': 30.1},
 {'Name': 5678, 'Age': 41.0},
 {'Name': 'Bob 1', 'Age': 14},
 {'Name': '!@#$%', 'Age': 65}]

After converting the csv data to dict, I use the following code to check if Name is string.

import pandas as pd
from schema import Schema, And, Use, Optional, SchemaError

schema = Schema([{'Name': str,
                  'Age':  float}])

validated = schema.validate(dict)

Is it possible?

So how do you process the data, one by one or doing some bulk operation on it ? — Sandeep Rawat, Jul 09 '22 at 06:35
@SandeepRawat hi, i updated my ques! basically, bulk would be better as doing it one by one (row by row) might increase the time it takes to run the code — H_SOCIAL MEDIA, Jul 09 '22 at 06:48
Why don't you just remove rows which have row["Name"].isdigit()? — FLAK-ZOSO, Jul 09 '22 at 07:05
The answer by @FLAK-ZOSO make sense. Moreover you can use multiprocessing here if you want to improve time complexity, only if its a huge data. — Sandeep Rawat, Jul 09 '22 at 07:26

FLAK-ZOSO · Answer 1 · 2022-07-09T07:10:57.003

1

Is it possible?

For sure. You can use the int constructor to convert that strings to integers if possible.

for element in list_:
    try:
        element["Name"] = int(element["Name"])
    except ValueError:
        pass

A faster way for doing it would be using isdigit method of class str.

for element in list_:
    if element["Name"].isdigit(): # Otherwise no need to convert
        element["Name"] = int(element["Name"])

So that you don't have to enter that try/except block.

edited Jul 09 '22 at 07:10

answered Jul 09 '22 at 06:21

FLAK-ZOSO

3,873
4
8
28

hi, thanks for your answer! some of my datasets might consist a million rows so this method might be inefficient due to the time complexity... – H_SOCIAL MEDIA Jul 09 '22 at 06:24
Ok, let me edit... – FLAK-ZOSO Jul 09 '22 at 06:52
1

the second code you showed us doesn't work, there are floats and ints too, so it'll give an AttributeError – XxJames07- Jul 09 '22 at 06:58
‍♀️sorry, the first one doesn't work either, just need to remove the second for loop, thanks – FLAK-ZOSO Jul 09 '22 at 07:01
loop through element.items() – XxJames07- Jul 09 '22 at 07:02
I think only the first column needs to be converted. At least that's what I see in the question as the expected result – FLAK-ZOSO Jul 09 '22 at 07:03
@FLAK-ZOSO if the column name includes different types like float and boolean, how do I work around it? – H_SOCIAL MEDIA Jul 09 '22 at 10:57
1

Try/except all the types that you want to convert. Notice that bool should be tried as the last one because it doesn't raise ValueError on non-boolean values. – FLAK-ZOSO Jul 09 '22 at 11:51

validation check - dictionary value types

1 Answers1