0

I made a Django application (runs on Apache mod-wsgi server) which reads contents of a file(with million entries in it) and displays it after performing some operations on it (operation time very less compared to read time).

Actually the file stores the metadata of objects in cloud. So instead of fetching the data from cloud everytime a user opens the application, the metadata is cached on the server which is used. This cached data is synced regularly with cloud to store the latest metadata.

Now the problem with this is that the data in cloud scales exponentially due to which the cached metadata in the file also increases thereby increasing the number of entries which in turn slows down the application.

What should I do to reduce the read time of file so that the application reads quickly and in turn load time decreases?

Sample data in File (separator used is '<=>' instead of comma) File

Same data displayed on web application enter image description here

zester
  • 165
  • 3
  • 12
  • What format is the data in, can you show an extract? You should consider storing the data somewhere where accessing it doesn't involve reading a huge file like a DB or something like redis – Iain Shelvington Apr 13 '22 at 17:04
  • You should use a database or something similar, as the comment above suggests. If you can't, have you considered splitting your large file into smaller chunks? – Kacperito Apr 13 '22 at 21:29
  • @IainShelvington The data is stored in CSV format. I thought of moving it to database but the main problem was how to organize data there because I didn't want to load everything into a single table. – zester Apr 14 '22 at 04:09
  • @KacperFloriański Splitting the file would complicate things and would not be the most efficient solution because that would create too many unwanted small files which could be a pain for debugging. – zester Apr 14 '22 at 05:02
  • I have added screenshot of sample file data – zester Apr 14 '22 at 05:26
  • 1
    What's wrong with having a single table? Access should be fast if you define your schema correctly. For example, see https://stackoverflow.com/questions/1108/how-does-database-indexing-work. – Kacperito Apr 14 '22 at 09:09

0 Answers0