1

There is a specific requirement to get the accurate no of rows from database for showing progress while processing the data. The application supports a lot of databases, So looking for a generic approach.

The current approach is using count(*)/count(INDEXED_COLUMN) but that is taking enormous amount of time. The time taken for count is significant to the actual processing which seems inappropriate.

Apart from that I have also tried using info from system tables but that is not 100% accurate and for updating the stats it's again taking a lot of time.

I want to reduce the count query time. Also, I want the accurate count from the query.

Javed Hussain
  • 11
  • 2
  • Maybe you can reduce the frequency in which you calculate and show the progress. I.e. don't show it for every iteration but only for every *n*-th, like the 100th or 100th... – sticky bit Apr 08 '19 at 11:26
  • 1
    Maybe instead you should try to avoid the need for getting an accurate number. In almost all cases needing the count is something that is actually not necessary (apart from requirements that were added for no good reason). – Mark Rotteveel Apr 08 '19 at 11:32
  • `count(*)/count(INDEXED_COLUMN)` can be simplified to `count(*)` (actually your expression could [result in an error](https://rextester.com/QGLAM75302) if `indexed_column` can be null). But yes, using `select count(*) from ...` i s the only way to get an _accurate_ result. –  Apr 08 '19 at 13:11
  • @Mark Rotteveel we are only using the total count to show the progress how much we have processed. Is there a better way to do it? – Javed Hussain Apr 09 '19 at 03:57
  • If you have billions of records to process then accuracy of that number is not really important and you can either estimate (although that is not always easy to do), or you just show a spinner and maybe update number of records processed to show that there is actually something happening. Other than that, no one will really care if you are processing 4 billion or 5 billion records (the requirements may say that someone care, but in reality, probably not). – Mark Rotteveel Apr 09 '19 at 16:58

0 Answers0