Error message when appending data to pandas dataframe

Question

Can someone give me a hand with this:

I created a loop to append successive intervals of historical price data from Coinbase.

My loop iterates successfully a few times then crashes.

Error message (under data_temp code line):

"ValueError: If using all scalar values, you must pass an index"

days = 10
end = datetime.now().replace(microsecond=0)
start = end - timedelta(days=days)
data_price = pd.DataFrame()

for i in range(1,50):
    print(start)
    print(end)
    data_temp = pd.DataFrame(public_client.get_product_historic_rates(product_id='BTC-USD', granularity=3600, start=start, end=end))
    data_price = data_price.append(data_temp)
    end = start
    start = end - timedelta(days=days)

Would love to understand how to fix this and why this is happening in the first place.

Thank you!

Here's the full trace:

Traceback (most recent call last): File "\coinbase_bot.py", line 46, in data_temp = pd.DataFrame(public_client.get_product_historic_rates(product_id='BTC-USD', granularity=3600, start=start, end=end)) File "D:\Program Files\Python37\lib\site-packages\pandas\core\frame.py", line 411, in init mgr = init_dict(data, index, columns, dtype=dtype) File "D:\Program Files\Python37\lib\site-packages\pandas\core\internals\construction.py", line 257, in init_dict return arrays_to_mgr(arrays, data_names, index, columns, dtype=dtype) File "D:\Program Files\Python37\lib\site-packages\pandas\core\internals\construction.py", line 77, in arrays_to_mgr index = extract_index(arrays) File "D:\Program Files\Python37\lib\site-packages\pandas\core\internals\construction.py", line 358, in extract_index raise ValueError("If using all scalar values, you must pass an index") ValueError: If using all scalar values, you must pass an index

Here's json returned via simple url call:

[[1454716800,370.05,384.54,384.44,375.44,6276.66473729],[1454630400,382.99,389.36,387.99,384.5,7443.92933224],[1454544000,368.74,390.63,368.87,387.99,8887.7572324],[1454457600,365.63,373.01,372.93,368.87,7147.95657328],[1454371200,371.17,374.41,371.33,372.93,6856.21815799],[1454284800,366.26,379,367.89,371.33,7931.22922922],[1454198400,365,382.5,378.46,367.95,5506.77681302]]

Very similar to this user's issue but cannot put my finger on it: When attempting to merge multiple dataframes, how to resolve "ValueError: If using all scalar values, you must pass an index"

Hi please add some example data and a the full trace for that error — eva-vw, Oct 02 '19 at 19:56
updated the post with the full error trace and sample json data obtained from url call. i'm using an python lib which is wrapping the url calls. — , Oct 02 '19 at 20:26

Joe Ernst · Accepted Answer · 2019-10-02T20:46:16.930

0

-- Hi DashOfProgramming,

Your problem is that the data_temp is initialised with only a single row and pandas requires you to provide it with an index for that.

The following snippet should resolve this. I replaced your API call with a simple dictionary that resembles what I would expect the API to return and used i as index for the dataframe (this has the advantage that you can keep track as well):

import pandas as pd
from datetime import datetime, timedelta

days = 10
end = datetime.now().replace(microsecond=0)
start = end - timedelta(days=days)
data_price = pd.DataFrame()

temp_dict = {'start': '2019-09-30', 'end': '2019-10-01', 'price': '-111.0928', 
'currency': 'USD'}

for i in range(1,50):
  print(start)
  print(end)
  data_temp = pd.DataFrame(temp_dict, index=[i])
  data_price = data_price.append(data_temp)
  end = start
  start = end - timedelta(days=days)

print(data_price)

EDIT

Just saw that your API output is a nested list. pd.DataFrame() thinks the list is only one row, because it's nested. I suggest you store your columns in a separate variable and then do this:

cols = ['ts', 'low', 'high', 'open', 'close', 'sth_else']

v = [[...], [...], [...]] # your list of lists

data_temp = pd.DataFrame.from_records(v, columns=cols)

edited Oct 02 '19 at 20:46

answered Oct 02 '19 at 20:39

Joe Ernst

179
3

What's confusing me with all of this that if I use a range of 5 instead of 50 when I start the for loop everything works great; I get panda frame with the expected number of rows and columns. I'll try your EDITED solution and report back shortly. Thanks. – Oct 02 '19 at 20:58
ok i just noticed that i'm retrieving empty data frames later on ... that's prob what's causing all these issues – Oct 02 '19 at 21:07
just add an additional check if len(values) > 0 add dataframe else continue – Joe Ernst Oct 02 '19 at 21:13
Thx for your help Joe. I was calling the API too quickly. Added a one sec sleep between each loop and that resolved the issue. – Oct 02 '19 at 21:15
@DashOfProgramming no problem! Could have actually thought of that myself too.. built sth similar before. Although with a PUSH API and asyncio listeners. Good Luck! – Joe Ernst Oct 02 '19 at 21:50

Error message when appending data to pandas dataframe

1 Answers1

Linked