0

We are using Django with its ORM in connection with an underlying PostgreSQL database and want to extend the data model and technology stack to store massive amounts of time series data (~5 million entries per day onwards).

The closest questions I found were this and this which propose to combine Django with databases such as TimescaleDB or InfluxDB. But his creates parallel structures to Django's builtin ORM and thus does not seem to be straightforward.

How can we handle large amounts of time series data while preserving or staying really close to Django's ORM?

Any hints on proven technology stacks and implementation patterns are welcome!

Cord Kaldemeyer
  • 6,405
  • 8
  • 51
  • 81

2 Answers2

1

Your best option is to keep your relational data in Postgres and your time series data in a separate database, and combining them when needed in your code.

With InfluxDB you can do this join with a Flux script by passing it the SQL that Django's ORM would execute, along with your database connection info. This will return your data in InfluxDB's format though, not Django models.

mhall119
  • 522
  • 2
  • 3
  • But that would be a separate system if I understand you right. For us, it would be nice to have a way to handle the data close to Django. – Cord Kaldemeyer May 04 '22 at 09:18
  • Whether you keep your data in PostgreSQL or InfluxDB, it's going to be in a separate system from Django. The Django ORM only works with relational database (hence the name), so you can't use it to access InfluxDB. The InfluxDB Python client library will perform much the same function, but won't map your database data into Django model objects. – mhall119 Jul 07 '22 at 16:45
0

why not using in parallel to your existing postgres a timescaledb for the time series data, and use this django integration for the latter one: https://pypi.org/project/django-timescaledb/.

Using multiple databases in django is possible, also I not did it by myself so far. Have a look here to do it in a convenient way (reroute certain Models to another db instead of default postgres one) Using Multiple Databases with django

rodo
  • 1
  • 1