3

I have some difficulties to understand the pros and the cons of using JuliaDB as a main backend database for a production website.

https://juliadb.org/

My use case is a collaborative data sciences platform. The client request 1 million unique visitors and 100 000 writes per day. Well... I wish so.

Implementing a SQL database mean that I need to "translate" the data science dataframes used for the calculus into SQL and backwards.

In the other hands JuliaDB is an end to end solution.

Regarding the different criterions for a website production database:

Julia natively hase concurrency:

Julia supports three main categories of features for concurrent and parallel programming:

Asynchronous "tasks", or coroutines Multi-threading Distributed computing Julia Tasks allow suspending and resuming computations for I/O, event handling, producer-consumer processes, and similar patterns. Tasks can synchronize through operations like wait and fetch, and communicate via Channels.

Multi-threading functionality builds on tasks by allowing them to run simultaneously on more than one thread or CPU core, sharing memory.

Finally, distributed computing runs multiple processes with separate memory spaces, potentially on different machines. This functionality is provided by the Distributed standard library as well as external packages like MPI.jl and DistributedArrays.jl.

In the other hand, JuliaDB doc tells that they support parallel computing but does not give much details.

Can JuliaDB handle parallel connections and asynchronous operation, making it performant for lots of users using it in parrallel?

Morgane
  • 155
  • 8

1 Answers1

1

From your question it looks that what you need is a massively parallel data ingestion mechanism. You a software architecture that allows to concurrently collect data for huge amount of users. Perhaps you should look at one of noSQL databases that provide horizontal scaling capability good example could be MongoDB (or perhaps a cloud equivalent such as DynamoDB).

If your data volume and parallelism is even higher you should consider streaming solution such as Apache Kafka.

On the other hand JuliaDB is totally on the other site of processing workflow. Once your massive data gets collected it ends up in an analytic process. In the recent years the most popular tool was the Hadoop stack with Apache Spark used for processing.

JuliaDB brings a new paradigm on the analytics step of data workflow. With this tool you can massively parallelize processing of huge data and hence you should consider JuliaDB as a nice alternative to Spark.

Przemyslaw Szufel
  • 40,002
  • 3
  • 32
  • 62