Pandas ValueError Arrays Must be All Same Length

Question

Iterates over a big list of .mp3 links to get the metadata tags and save it to an Excel file. Results in this error. I appreciate any help. Thanks.

    #print is_connected();

    # Create a Pandas dataframe from the data.
df = pd.DataFrame({'Links' : lines ,'Titles' : titles , 'Singers': finalsingers , 'Albums':finalalbums , 'Years' : years})


    # Create a Pandas Excel writer using XlsxWriter as the engine.
writer = pd.ExcelWriter(xlspath, engine='xlsxwriter')

    # Convert the dataframe to an XlsxWriter Excel object.
df.to_excel(writer, sheet_name='Sheet1')
    #df.to_excel(writer, sheet_name='Sheet1')


    # Close the Pandas Excel writer and output the Excel file.
writer.save()

Traceback (most recent call last):
  File "mp.py", line 87, in <module>
    df = pd.DataFrame({'Links' : lines ,'Titles' : titles , 'Singers': finalsingers , 'Albums':finalalbums , 'Years' : years})
  File "C:\Python27\lib\site-packages\pandas\core\frame.py", line 266, in __init__
    mgr = self._init_dict(data, index, columns, dtype=dtype)
  File "C:\Python27\lib\site-packages\pandas\core\frame.py", line 402, in _init_dict
    return _arrays_to_mgr(arrays, data_names, index, columns, dtype=dtype)
  File "C:\Python27\lib\site-packages\pandas\core\frame.py", line 5409, in _arrays_to_mgr
    index = extract_index(arrays)
  File "C:\Python27\lib\site-packages\pandas\core\frame.py", line 5457, in extract_index
    raise ValueError('arrays must all be same length')
ValueError: arrays must all be same length

score 142 · Answer 1 · edited Feb 21 '22 at 09:14

142

you can do this to avoid that error

a = {'Links' : lines ,'Titles' : titles , 'Singers': finalsingers , 'Albums':finalalbums , 'Years' : years}
df = pd.DataFrame.from_dict(a, orient='index')
df = df.transpose()

Explanation:

This creates the DataFrame as each key (e.g. 'Links') was a row and like this the missing values are actually missing columns which is no problem for pandas (only missing rows lead to ValueError during creation) After that you transpose the DataFrame (flip the axis) and make the rows to columns, which results the DataFrame you initially wanted.

edited Feb 21 '22 at 09:14

gustavz

2,964
3
25
47

answered Nov 05 '16 at 19:04

Vivek Srinivasan

2,687
3
17
17

1

It doesn't work for me. It adds the index as the first row, and split the rows apparently randomly – robertspierre Jun 25 '19 at 12:35
1

try this answer from a different question, which worked for me: https://stackoverflow.com/a/45052003/4682460 – lobi Nov 14 '19 at 15:14
17

What exactly does this do? – Richard Nov 23 '19 at 11:19
I too am curious about how exactly this works. Meanwhile, I'm happy that it does work. Values in `a` shorter than the longest list in `a` get padded with `None`, which is what one would expect in that case. – rodrigo-silveira Dec 02 '20 at 07:38
@Richard This works as it creates the DataFrame as each key (e.g. 'Links') was a row and like this the missing values are actually missing columns which is no problem for pandas (only missing rows are during creation) and then you transpose the dataframe (flip the axis) and make the rows to columns, which results you the dataframe you initially wanted. Did that help? – gustavz Feb 21 '22 at 09:10

score 14 · Answer 2 · answered Nov 05 '16 at 19:04

It's telling you that the arrays (lines, titles, finalsingers, etc...) are not of the same length. You can test this by

print(len(lines), len(titles), len(finalsingers)) # Print all of them out here

This will show you which data is malformed and then you'll need to do some investigating into what the right way to correct this is.

score 14 · Answer 3 · edited Feb 02 '23 at 14:42

14

You can pad the shortest lists with empty elements:

def pad_dict_list(dict_list, padel):
    lmax = 0
    for lname in dict_list.keys():
        lmax = max(lmax, len(dict_list[lname]))
    for lname in dict_list.keys():
        ll = len(dict_list[lname])
        if  ll < lmax:
            dict_list[lname] += [padel] * (lmax - ll)
    return dict_list


dict_list = {'Links': [1, 2, 3], 'Titles': [1, 2, 3, 4], 'Singers': [1, 2], 'Albums': [1, 2, 3], 'Years': [1, 2, 3, 4]}
dict_list = pad_dict_list(dict_list, 0)
print(dict_list)

Output

{'Links': [1, 2, 3, 0], 'Titles': [1, 2, 3, 4], 'Singers': [1, 2, 0, 0], 'Albums': [1, 2, 3, 0], 'Years': [1, 2, 3, 4]}

edited Feb 02 '23 at 14:42

hrdom

83
1
7

answered Jun 25 '19 at 15:51

robertspierre

3,218
2
31
46

what's `padel`? – loretoparisi Mar 15 '21 at 16:20
2

@loretoparisi it's something that you would like to use as an element for filling the dictionary values – R. Kulebyakin Mar 15 '21 at 18:22

score 5 · Answer 4 · answered Oct 25 '19 at 21:52

5

Duplicate variable names caused this problem for me

answered Oct 25 '19 at 21:52

cubeloid

149
1
6

score 3 · Answer 5 · edited Nov 23 '20 at 00:35

3

I have come across the same error while reading JSON file to the pandas frame. adding linesbool, default False parameter solved the issue.

StringData = StringIO(obj.get()['Body'].read().decode('utf-8'))
                mydata = pdf.read_json(StringData, lines=True)

edited Nov 23 '20 at 00:35

Lumos

570
1
11
24

answered Oct 15 '20 at 00:27

Jaliya Sumanadasa

131
1
7

Pandas ValueError Arrays Must be All Same Length

5 Answers5

Linked

Related