Convert list with duplicated values into unique value over loop

Question

I've this issue and i've been doing quite a few researches and doing some testing without success.

I've a Dataframe which one of the variables content looks like this:

var1 = ['QALIF',
 ['PRD', 'PRD', 'PRD', 'PRD', 'PRD'],
 ['DV', 'DV', 'DV', 'DV', 'DV', 'DV', 'DV']]

I don't know how represent this in a dataframe, when I convert into pd.DataFrame(var1) the Data Frame splits into several columns which is not the case. The values are type text for Single values or liste of values with different lengths.

I need to get unique values for each list inside my Dataframe, in other words, iterate over my variable "var1" and convert the "List" into Single value

I've been trying to get some inspiration thought this link below, some test did work fine but when I try to apply to entire dataset it doesn't return good results.

Get unique values from a list in python

I've build a (several) functions :

for lst in df[var1]:
     df[var1].apply(lambda x: list(dict.fromkeys(lst)))

But I get an error

List object is not callable

And I've tried so many other options but none of them seem to work.

Any help is welcome

do you want to [flatten](https://stackoverflow.com/q/952914/10197418) the list of lists? — FObersteiner, Jun 08 '21 at 13:49
i) I think you can take the first value of the lists: `df[var1].str[0]`. ii) error reason: you probably assigned a list to a variable named `list` previously. — Mustafa Aydın, Jun 08 '21 at 13:53
@MustafaAydın yes I did that mystake but I've restarted the kernel after it, and when I run the function again I don't get the error but the result is not good, the text is splitted into [P,R,D] [D,V] ... and so on — Carlos Carvalho, Jun 08 '21 at 14:16
@CarlosCarvalho I see, what about `df[var1].str[0]`; does it work at all? — Mustafa Aydın, Jun 08 '21 at 14:16

score 0 · Answer 1 · answered Jun 08 '21 at 14:15

0

var1 = ['QALIF',
 ['PRD', 'PRD', 'PRD', 'PRD', 'PRD'],
 ['DV', 'DV', 'DV', 'DV', 'DV', 'DV', 'DV']]
import pandas as pd
var2=[]
def frame(var1):
    if type(var1)==list and len(var1)>1:
        return var1[0]
    return var1



df=pd.DataFrame(var1)
      
df[0]=df[0].apply(frame)

print(df)

answered Jun 08 '21 at 14:15

Chinu

11
1

@Chimu Var1 is an column from a Dataframe, and the values from this column are text, but some others are lists inside brackets [ ] , so df['var1'] it's a series – Carlos Carvalho Jun 08 '21 at 14:26

score 0 · Answer 2 · answered Jun 08 '21 at 14:29

0

If what you want is a Seriescontaining all the unique items inside the lists you can use the explode method (Documentation):

var1 = ['QALIF', ['PRD', 'PRD', 'PRD', 'PRD', 'PRD'], ['DV', 'DV', 'DV', 'DV', 'DV', 'DV', 'DV']]

df = pd.DataFrame(var1).explode(0).drop_duplicates()

print(df)

output:

       0
0  QALIF
1    PRD
2     DV

answered Jun 08 '21 at 14:29

99_m4n

1,239
3
17

No I don't a series, I want to get unique values from one variable of my dataframe which some of the values are lists with repeated values. – Carlos Carvalho Jun 08 '21 at 14:45
can you edit the question showing an example of your expected output? – 99_m4n Jun 08 '21 at 14:56

Convert list with duplicated values into unique value over loop

2 Answers2