Reading Elastic cluster data into python data frame

Question

I am pretty new with elasticsearch. so, please forgive if i am asking a very simple question.

In my workplace we have a proper setup of ELK.

Due to the very large volume of data we are just storing 14 days of data and my question is how can i read the data in Python and later store my analysis in some NOSQL.

As of now my primary goal is to read the raw data into python in the form of data frame or any format from the elastic cluster.

I want to get it for different time intervals like 1 day, 1 week, 1 month etc..

I am struggling for the last 1 week.

Answer 1

you can use the below code to achieve that

# Create a DataFrame object
from pandasticsearch import DataFrame
df = DataFrame.from_es(url='http://localhost:9200', index='indexname')

To get the schema of your index:-

 df.print_schema()

After that you can perform general dataframe operation on the df.

If you want to parse the result then do the below :-

from elasticsearch import Elasticsearch
es = Elasticsearch('http://localhost:9200')
result_dict = es.search(index="indexname", body={"query": {"match_all": {}}})

and then finally everything into your final dataframe:-

from pandasticsearch import Select
pandas_df = Select.from_dict(result_dict).to_pandas()

I hope it helps..

Answer 2

It depends on how you want to read the data from the Elasticsearch. Is it incremental reading i.e. reading new data that comes to you every day or is it like a bulk reading. For the latter, you need to use the bulk API of Elasticsearch in python and for the former, you can restrict yourself to a simple range query.

Schematic code for reading bulk data: https://gist.github.com/dpkshrma/04be6092eda6ae108bfc1ed820621130

How to use bulk API of ES:

How to use Bulk API to store the keywords in ES by using Python

https://elasticsearch-py.readthedocs.io/en/master/helpers.html#elasticsearch.helpers.bulk

How to use the range query for incremental inserts:

https://martinapugliese.github.io/python-for-(some)-elasticsearch-queries/

How to have Range and Match query in one elastic search query using python?

Since you want your data to be inserted in different intervals, you will require to perform date aggregations as well.

https://www.elastic.co/guide/en/elasticsearch/reference/current/search-aggregations-bucket-datehistogram-aggregation.html

How to perform multiple aggregation on an object in Elasticsearch using Python?

Once you issue your Elasticsearch query, your data will be collected in a temporary variable, you can use the python library over NOSQL database such as PyMongo to insert into Elasticsearch data into it.