Pandas error: Reindexing only valid with uniquely valued Index

Question

I am trying to concatenate multiple files, but I get the following error:

raise ValueError("Reindexing only valid with uniquely valued Index objects")
ValueError: Reindexing only valid with uniquely valued Index objects

My code:

import pandas as pd
import glob
from functools import reduce
path = r'/path/to'
all_files = glob.glob(path + "/*.txt")
my_dfs = [pd.read_csv(file, sep='\t').set_index(['POS', 'CHROM', 'REF', 'ALT']) for file in all_files]
df = pd.concat(my_dfs, axis=1)
df.reset_index(inplace=True)
result_df = df.fillna('.')
result_df.to_csv('out.txt', sep="\t", index=False)

It this because I have duplictes? how to osolve this problem?

If you already know that you have duplicates, you just have to remove them. or choose the one to keep. `df = df.loc[~df.index.duplicated(keep='first')]` — FugitiveMemories, Mar 12 '21 at 17:08
but I have a problem during this step df = pd.concat(my_dfs, axis=1) , so Icannot generate a df. Also this command line generate a very strange output. — user3224522, Mar 12 '21 at 17:10
I dont know how to iterate as I do not use pandas often.. any link or something like this as an example? — user3224522, Mar 12 '21 at 17:27
new_dfs = [df.drop_duplicates(subset=['ALT']) for df in my_dfs) somethins like this? — user3224522, Mar 12 '21 at 17:32
[Here](https://stackoverflow.com/questions/13035764/remove-pandas-rows-with-duplicate-indices) you go. — FugitiveMemories, Mar 12 '21 at 17:44
new_df = [df.loc[~df.index.duplicated(keep='first')] for df in my_dfs] OK, I think this is working. — user3224522, Mar 12 '21 at 17:46
This is because you have to reset the index post concatenation. Hence `df.reset_index(inplace=True, drop=True)` — FugitiveMemories, Mar 12 '21 at 17:51
Let us [continue this discussion in chat](https://chat.stackoverflow.com/rooms/229849/discussion-between-kenpachi-zaraki-and-user3224522). — FugitiveMemories, Mar 13 '21 at 06:02

Pandas error: Reindexing only valid with uniquely valued Index

0 Answers0