1

I need help with processing an unsorted dataset. Sry, if I am a complete noob. I never did anything like that before. So as you can see, each conversation is identified by a dialogueID which consists of multiple rows of "from" & "to", as well as text messages. I would like to concatenate the text messages from the same sender of a dialogueID to one column and from the receiver to another column. This way, I could have a new csv-file with just [dialogueID, sender, receiver].

dataset the new dataset should look like this new dataset

I watched multiple tutorials and really struggle to figure out how to do it. I read in this 9-year-old post that iterating through data frames are not a good idea. Could someone help me out with a code snippet or give me a hint on how to properly do it without overcomplicating things? I thought something like this pseudo code below, but the performance with 1 million rows is not great, right?

while !endOfFile
  for dialogueID in range (0, 1038324)
    if dialogueID+1 == dialogueID and toValue.isnull()
      concatenate textFromPrevRow + " " + textFromCurrentRow
      add new string to table column sender
    else
      add text to column receiver
CodingStudent
  • 776
  • 1
  • 5
  • 10

2 Answers2

1

Edit 1

According to your clarification, this is what I believe you're looking for.

Create an aggregation function which basically concats your string values with a line-break character. Then group by dialogueID and apply your aggregation.

d = {}
d['from'] = '\n'.join
d['to'] = '\n'.join
new_df = dialogue_dataframe.groupby('dialogueID', as_index=False).agg(d)

After that rename the columns as you'd like:

df.rename(columns={"from": "sender", "to": "receiver"})

Original answer

Not quite sure I understood what you try to achieve, but maybe this will give some insights. Maybe write a couple of rows of the table you expect to get, for better clarification

Nazar Nintendo
  • 209
  • 5
  • Hi, I added a new picture of how I meant it. Sry, I am bad with words. I hope this is more clear. Also thanks for the link. – CodingStudent Nov 16 '22 at 23:46
  • Yep, I understood you correctly. Please see the accepted answer by the link I gave. I suppose you can use something like `d = dict.fromkeys(dialogue_dataframe.columns.difference(['dialogueID']), 'first') d['from'] = ', '.join d['to'] = ', '.join new_df= dialogue_dataframe.groupby('dialogueID', as_index=False).agg(d)` After that just rename the columns `from` and `to` to `sender` and `receiver` – Nazar Nintendo Nov 16 '22 at 23:58
0

While the exact structure of the data (and thus your task) is not completely clear, maybe DataFrame.apply or rather DataFrame.aggregate can help you speed things up. Also, I would aggregate into either a dictionary or dataframe indexed by dialogue id. This way you can easily check if a given dialogue / sender already exists.

Yuri Feldman
  • 2,354
  • 1
  • 20
  • 23