10

I use spark to read from elasticsearch.Like 

select col from index limit 10;

The problem is that the index is very large, it contains 100 billion rows.And spark generate thousands of tasks to finish the job.
All I need is 10 rows, even 1 tasks returns 10 rows that can finish the job.I don't need so many tasks.
Limit is very slow even limit 1.
Code：

sql = select col from index limit 10
sqlExecListener.sparkSession.sql(sql).createOrReplaceTempView(tempTable)
no123ff
  • 307
  • 5
  • 16
  • Have you tried to set the partition size explicitly? – Justin Pihony Nov 30 '17 at 02:56
  • @JustinPihony yes,I set es_input_max_docs_per_partition=5000,it seems that total_rows_es_contains = es_input_max_docs_per_partition * num_of_partitions – no123ff Nov 30 '17 at 02:58
  • If you are using push.down=True make sure that double.filtering=False since it might prevent the limit to be pushed down. Check your physical plan by calling df.explain(True), and make sure there is no filtering between elastic search and the limit – itaifrenkel Jul 11 '18 at 10:34

1 Answers1

8

The source code of limit shows that it will take the first limit elements for every partition, and then it will scan all partitions.

To speed up the query you can specify one value of the partition key. Suppose that you are using day as the partition key, the following query will be much faster

select col from index where day = '2018-07-10' limit 10;
secfree
  • 4,357
  • 2
  • 28
  • 36