pandas : appending data frames with addtional column names

Question

I've extracted data via a API and the source system has created a new field. I'm trying to concatenate the 2 items below. There are more fields than this but this gets the point across. How do i merge the follow data sets?

The field names will always be the same but there might be additional columns and columns removed in the future.

Audit_ID    Start Time           End Time   
1           02/09/2019 05:00     02/09/2019 10:45

new data

Audit_ID    Start Time           End Time            Shift
2           03/09/2019 03:00     03/09/2019 10:45    Afters

This is what i want it to look like :

Audit_ID    Start Time           End Time            Shift
1           02/09/2019 05:00     02/09/2019 10:45    
2           03/09/2019 03:00     03/09/2019 10:45    Afters

When i run the code :

joined_rows = pd.concat(data1 , data2], axis=0

This gives error :

joined_rows = pd.concat([data1, data2])

    AssertionError                            Traceback (most recent call last)
<ipython-input-46-469b3f9d61b5> in <module>()
----> 1 joined_rows = pd.concat(data1 , data2], axis=0)
      2 joined_rows

C:\Users\user\AppData\Local\Continuum\anaconda3\lib\site-packages\pandas\core\reshape\concat.py in concat(objs, axis, join, join_axes, ignore_index, keys, levels, names, verify_integrity, sort, copy)
    256     )
    257 
--> 258     return op.get_result()
    259 
    260 

C:\Users\user\AppData\Local\Continuum\anaconda3\lib\site-packages\pandas\core\reshape\concat.py in get_result(self)
    471 
    472             new_data = concatenate_block_managers(
--> 473                 mgrs_indexers, self.new_axes, concat_axis=self.axis, copy=self.copy
    474             )
    475             if not self.copy:

C:\Users\user\AppData\Local\Continuum\anaconda3\lib\site-packages\pandas\core\internals\managers.py in concatenate_block_managers(mgrs_indexers, axes, concat_axis, copy)
   2057         blocks.append(b)
   2058 
-> 2059     return BlockManager(blocks, axes)

C:\Users\user\AppData\Local\Continuum\anaconda3\lib\site-packages\pandas\core\internals\managers.py in __init__(self, blocks, axes, do_integrity_check)
    141 
    142         if do_integrity_check:
--> 143             self._verify_integrity()
    144 
    145         self._consolidate_check()

C:\Users\user\AppData\Local\Continuum\anaconda3\lib\site-packages\pandas\core\internals\managers.py in _verify_integrity(self)
    348                 "Number of manager items must equal union of "
    349                 "block items\n# manager items: {0}, # "
--> 350                 "tot_items: {1}".format(len(self.items), tot_items)
    351             )
    352 

AssertionError: Number of manager items must equal union of block items
# manager items: 44, # tot_items: 48

Any help appreciated

Add an empty column named Shift in data1 then apply this concatenation. because both the dataframes should have same number of columns — Subbu VidyaSekar, Oct 18 '19 at 10:49
you could use `data=data.reindex(columns=data2.columns)`and then use `pd.concat` ....... but anyway I think you shouldn't get that error — ansev, Oct 18 '19 at 11:12
solved from kmader answer on https://stackoverflow.com/questions/35137952/pandas-concat-failing — resolver101, Oct 18 '19 at 19:25

Subbu VidyaSekar · Answer 1 · 2019-10-18T11:21:32.160

0

sample code:

dict1 = {'Audit_ID':['1'],'Start_time':['02/09/2019 05:00'],'End_Time':['02/09/2019 10:45']}
dict2 = {'Audit_ID':['2'],'Start_time':['03/09/2019 05:00'],'End_Time':['03/09/2019 10:45'], 'shift':['Afters']}
df1 = pd.DataFrame.from_dict(dict1)
df2 = pd.DataFrame.from_dict(dict2)
# above code is to create your dataset

res = pd.concat([df1, df2], axis=0)
print(res)

Result :

  Audit_ID          End_Time        Start_time   shift
0        1  02/09/2019 10:45  02/09/2019 05:00     NaN
0        2  03/09/2019 10:45  03/09/2019 05:00  Afters

edited Oct 18 '19 at 11:21

answered Oct 18 '19 at 11:01

Subbu VidyaSekar

2,503
3
21
39

It is same like `res = pd.concat([df1, df2])`, so your solution not help – jezrael Oct 18 '19 at 11:27
based on the result, I came to this conclusion, kindly explain what went wrong here – Subbu VidyaSekar Oct 18 '19 at 11:29
Check dupe, there is explanation – jezrael Oct 18 '19 at 11:29
the code you gave worked so obviously something else is going on to what i thought. On my dataset ive used the same code and it gives the same error. I've included more detail in the error log to see if this helps – resolver101 Oct 18 '19 at 11:50

pandas : appending data frames with addtional column names

1 Answers1