convert ceilometer output to python dataframe

Question

I'm a little new to Python and openstack ceilometer. I'm reading ceilometer data using the following code:

import ceilometerclient.client

cclient = ceilometerclient.client.get_client(2, os_username="Ceilometeradmin", os_password="blahblah", os_tenant_name="blahblah", os_auth_url="http://xxx.xx.xx.x:5000/v2.0")

query = [dict(field='resource_id', op='eq', value='dd893564-85e5-43f8-a384-086417f1d82c')]

ls = cclient.meters.list(q=query)

(Please see picture of output attached)

Does anyone know how i could convert this into a dataframe?

I tried : ls2 = pandas.DataFrame(ls, columns=["user_id", "name", "resource_id", "source", "meter_id", "project_id", "type", "unit"])

but get the following error:

Traceback (most recent call last):
File "", line 1, in File "/usr/lib/python2.7/dist-packages/pandas/core/frame.py", line 250, in init copy=copy)
File "/usr/lib/python2.7/dist-packages/pandas/core/frame.py", line 363, in _init_ndarray return create_block_manager_from_blocks([values.T], [columns, index])
File "/usr/lib/python2.7/dist-packages/pandas/core/internals.py", line 3750, in create_block_manager_from_blocks construction_error(tot_items, blocks[0].shape[1:], axes, e)
File "/usr/lib/python2.7/dist-packages/pandas/core/internals.py", line 3732, in construction_error passed,implied))
ValueError: Shape of passed values is (1, 2), indices imply (8, 2)

if someone could help it would really be much much appreciated..

Thank you

Best Wishes T

Also, what happens when you do ls2 = pandas.DataFrame(ls)? (This will give you an insight as to why you got that ValueError) — jgloves, Oct 29 '15 at 17:34
it's a list type; it returns this when i try to convert to pandas.dataframe: 0 0 — tezzaaa, Oct 29 '15 at 17:39
I'm a bit busy atm but it's possible to use `DataFrame.from_items` with a generator. Will post an answer soon — Lim H., Oct 29 '15 at 17:49

score 1 · Answer 1 · edited Nov 14 '15 at 14:42

With the help of a colleague we managed to find the solution. I'm posting it in case it's useful to anyone trying to convert ceilometer data into a dataframe.

ls2 = str(ls)[8:-2]
tmp = ls2.replace("u'","'")
tmp = tmp.replace("<Meter","")
tmp = tmp.replace("}>","}")
tmp= "["+tmp+"]"
tmp = tmp.replace("'", "\"")
parsed = json.loads(tmp)
ls3 = pandas.DataFrame(parsed)

Replaced characters which make it as invalid dictionary and converted into dataframe

score 0 · Answer 2 · answered Nov 14 '15 at 13:59

meters = c.meters.list()
fields = ['user_id', 'name', 'resource_id', 'source', 'meter_id', 'project_id', 'type', 'unit']
converted_meters = [dict([(key, getattr(m, key, None))]) for m in meters for key in fields]
data_frame = pandas.DataFrame(converted_meters)

convert ceilometer output to python dataframe

2 Answers2