0

I am using VM instance in google cloud and i want to use bigquery as well. I am trying to append the newly generated report in while loop to last row of the bigquery table every 10 minutes with below script.

 position_size = np.zeros([24, 24], dtype=float)

... some codes here
... some codes here
... some codes here
... some codes here

 while 1:
 current_time = datetime.datetime.now()
 if current_time.minute % 10 == 0:
    try:
        report = pd.DataFrame(position_size[12:24], columns=('pos'+str(x) for x in range(0, 24)))
        report.insert(loc=0, column="timestamp", datetime.datetime.now())
        ... some codes here
        pandas_gbq.to_gbq(report, "reports.report", if_exists = 'append')
    except ccxt.BaseError as Error:
        print("[ERROR] ", Error)
        continue

But as you can see below screenshot it does not append in an order. How can i solve this issue? Thank you in advance

enter image description here

euphrates85
  • 87
  • 1
  • 8

1 Answers1

1

How are you reading results? In most databases (BigQuery included), the row order is indeterminate in the absence of an ordering expression. You likely need an ORDER BY clause in your SELECT statement.

shollyman
  • 4,216
  • 19
  • 17
  • I was thinking that when a new report is generated it will add it to the bottom line. Can i solve this issue within python script? – euphrates85 May 20 '22 at 17:38
  • 1
    Your screenshot indicates you're viewing a table preview, so it's unordered. Wherever you actually consume the data, run a query on the table with the ordering clause. https://cloud.google.com/bigquery/docs/samples/bigquery-simple-app-all has an example of running a query and printing results with python. – shollyman May 20 '22 at 19:06