-4

What is the best approach for saving statistical data on a file using spring framework? is there any available library that offers reading and updating the data on a file? or should I build my own IO code?

I already have a relational database, but don't like the approach of creating an additional table to save the calculated values in different multiple tables with joins, also don't want to add more complexity to the project by using an additional database for just one task like MongoDB.

To understand the complexity of this report, Imagine you are drawing a chart with a total number of daily transactions for a full year with billions of records at any time with a lot of extra information like( total and average with different currencies on different rates).

So, my approach was to generate those data in a file on a regular basis, so later I don't need to generate them again once requested, only accumulate the new dates if available to the file

Is this approach fine? and what is the best library to do that in an efficient way?

Update

I found this answer useful for why sometimes people prefer using flat files rather than the relational or non-relational one

Is it faster to access data from files or a database server?

Hany Sakr
  • 2,591
  • 28
  • 27
  • You should use a suitable database. – Kayaman Aug 26 '18 at 08:32
  • Please explain why you want to write to a file, instead of writing to a database that is able to persist data on disk, such as PostgreSQL or MongoDB for example. – Andreas Vogl Aug 26 '18 at 08:34
  • I don't want to add an extra database or tables for statistical data that are temporary, and it is better performance to load a file than loading a lot of tables with queries. – Hany Sakr Aug 26 '18 at 09:16

2 Answers2

0

I would preferet to use MongoDB for such purposes, but if you need simple approach, you can write your data to csv\excel file. Just using I\O 

    List<String> data = new ArrayList<>();
    data.add("head1;head2;head3");
    data.add("a;b;c");
    data.add("e;f;g");
    data.add("9;h;i");

    Files.write(Paths.get("my.csv"), data);

That is all) How to convert your own object, to such string 'filed1;field2' I think you know. Also you can use apache-poi csv library, but I think this is way much faster.

    Files.write(Paths.get("my.csv"), data, StandardOpenOption.APPEND);

If you want to append data to existed file, there are many different options in StandardOpenOption.

For reading you should use Files.readAllLines(Paths.get("my.csv")); it will return you list of strings. Also you can read lines in range.

But if you need to retrieve one column, or update two columns where, and so on. You should read about MongoDB or other not relational databases. It is difficult write about MongoDB here, you should read documentation.

Enjoy)

lord_hokage
  • 189
  • 7
  • Thanks for your reply, I like your idea of using a CSV file, but I don't want to add a completely new database for such one task. I found a Jackson library that can be used with CSV and can map directly to objects. – Hany Sakr Aug 26 '18 at 10:37
0

I found a library that can be used to write/read CSV files easily and can be mapped to objects as well Jackson data formats

Find an example with spring

Hany Sakr
  • 2,591
  • 28
  • 27