Merge 2 lists and remove duplicates in Python

Question

I have 2 lists, looking like:

temp_data:

{
  "id": 1,
  "name": "test (replaced)",
  "code": "test",
  "last_update": "2020-01-01",
  "online": false,
  "data": {
    "temperature": [
      {
        "date": "2019-12-17",
        "value": 23.652905748126333
      },
      ...
    ]}

hum_data:

{
  "id": 1,
  "name": "test (replaced)",
  "code": "test",
  "last_update": "2020-01-01",
  "online": false,
  "data": {
    "humidity": [
      {
        "date": "2019-12-17",
        "value": 23.652905748126333
      },
      ...
    ]}

I need to merge the 2 lists to 1 without duplicating data. What is the easiest/efficient way? After merging, I want something like this:

{
  "id": 1,
  "name": "test",
  "code": "test",
  "last_update": "2020-01-01",
  "online": false,
  "data": {
    "temperature": [
      {
        "date": "2019-12-17",
        "value": 23.652905748126333
      },
      ...
    ],
    "humidity": [
      {
        "date": "2019-12-17",
        "value": 23.652905748126333
      },
      ...

Thanks for helping.

Do you want something like `data :{humidity:{[...]}, temperature:{[...]}}` or `data{[...]}`? — Gabriel Avendaño, Feb 04 '20 at 09:34
@GabrielAvendaño I edited the question with a clear answer to your question :) — sg_sg94, Feb 04 '20 at 09:38
I just had the error they are ReturnDict but my debugger says they are lists. They are in order — sg_sg94, Feb 04 '20 at 09:43
Does this answer your question? [Append dictionary to a dictionary?](https://stackoverflow.com/questions/8930915/append-dictionary-to-a-dictionary) — Jalo, Feb 04 '20 at 09:57

score 0 · Answer 1 · answered Feb 04 '20 at 10:17

If your lists hum_data and temp_data are not sorted then first sort them and then concatenate the dictionaries pair-wise.

# To make comparisons for sorting
compare_function = lambda value : value['id']

# sort arrays before to make later concatenation easier
temp_data.sort(key=compare_function)
hum_data.sort(key=compare_function)


combined_data = temp_data.copy()

# concatenate the dictionries using the update function
for hum_row, combined_row in zip(hum_data, combined_data):
    combined_row['data'].update(hum_row['data'])

# combined hum_data and temp_data
combined_data

If the lists are already sorted then you just need to concatenate dictionary by dictionary.

combined_data = temp_data.copy()

# concatenate the dictionries using the update function
for hum_row, combined_row in zip(hum_data, combined_data):
    combined_row['data'].update(hum_row['data'])

# combined hum_data and temp_data
combined_data

With that code I got the following result:

[
    {
    'id': 1,
    'name': 'test (replaced)',
    'code': 'test',
    'last_update': '2020-01-01',
    'online': False,
    'data': {
        'temperature': [{'date': '2019-12-17', 'value': 1}],
        'humidity': [{'date': '2019-12-17', 'value': 1}]}
    },
    {
    'id': 2,
    'name': 'test (replaced)',
    'code': 'test',
    'last_update': '2020-01-01',
    'online': False,
    'data': {
        'temperature': [{'date': '2019-12-17', 'value': 2}],
        'humidity': [{'date': '2019-12-17', 'value': 2}]}
    }
]

Merge 2 lists and remove duplicates in Python

1 Answers1