22

I am very interested in using monetdb as a datamart, holding some huge data tables for querying and reporting

However, after some searching, I am unable to find any online posts / blogs regarding their use of Monetdb in any kind of production capacity.

Also, there seems to be little or next to no activity online regarding Monetdb.

Is this a bad sign for the future of Monetdb ?

user1320615
  • 1,247
  • 1
  • 10
  • 5

6 Answers6

13

I am very interested in using monetdb as a datamart, holding some huge data tables for >querying and reporting

My boss is also interested in MonetDB and I had the same reaction as you. No one is writing about MonetDB... is no one using MonetDB?

Regardless, I have been running performance tests on datasets of 500,000 to 1,000,000 records comparing MonetDB (column-oriented dbms) vs. MySQL (row-oriented dbms) and MonetDB beats MySQL in all regards- even in bulk inserts... which hypothetically it should not be as good at.

I can't speculate as to what all this means for MonetDB's future, but while it's around you might want to check it out because it performs well.

(I run Windows 7 and am communicating with each database using PHP)

Floyd
  • 445
  • 5
  • 9
11

I react a bit late to this post, but I'd like to add my voice to the ones using MonetDB in a production environment. We use it as the back-end of Spinque, a framework for designing complex search solutions. I've been using MonetDB for about 10 years, but only in the past 3 years in a production environment. Clearly, it has pros and cons and bugs like all other products, but it is being developed and improved very actively (I don't understand the low-activity signs that you refer to). If you want a DB that allows you to be ahead of the market standards, it's a good choice. Otherwise, just go for MS SQL ;)

cornuz
  • 2,678
  • 18
  • 35
  • They are different beasts, with different aims. See [this](http://db-engines.com/en/system/Cassandra%3BMonetDB%3BRedis) for a quick feature comparison. Obviously, if you try to use MonetDB as a key-value store, or Redis for OLAP, you won't be very happy. – cornuz Oct 14 '14 at 08:43
  • Hi, I'm evaluating monetDB for an accommodation availability search engine. I've been able to work with hundred million rows in my local computer with amazing performance but my worries are about concurrent connections because in live it will be beat by ~1k req/sec. Do you think Spinque+monetDB will work for this? Thanks – GBrian Jan 03 '15 at 06:30
  • 1
    @GBrian, it's hard to give a real answer without defining a "request" and without knowing the hardware specs, but in principle - sure, why not? I feel that going into details may be off-topic here and I don't want to turn this discussion into commercial propositions. I recommend to drop an email to [Spinque](http://www.spinque.com) and/or [MonetDB Solutions](http://www.monetdbsolutions.com) teams to learn more about what is possible. – cornuz Jan 05 '15 at 12:16
  • Thanks @cornuz, I've opened a new question regarding updating data. Can you please take a look? How many records do you use to manage and how often are they updated/added/deleted? Thanks. http://stackoverflow.com/questions/28122178/monetdb-refresh-data-in-background-best-strategy-with-active-connections-making – GBrian Jan 24 '15 at 04:15
10

I've been evaluating it lately for a client so I've had some time with it. My impression at this point is that it is just finishing "growing up" from being an academic experimental playground. It clearly has yet to be really discovered, though it does have some rough edges which might hinder certain applications.

As I write, I'm in the process of trying to load over 100 million rows into an instance (at 27mil presently). So far, it performs startlingly well in some areas (aggregates), but is oddly sluggish in others (most joins I've tried so far); that said, I've not yet run the recommended sampling process yet and I'm forcing it to live in just a single service with 32GB RAM.

I've found a few little glitches and one thing that caused a full service crash (obscure and reported), but I'm thinking that for many applications MonetDB could be just the ticket. Columnar storage (rather than NoSQL) seems to be the future IMO.

I'll update this if I find anything particularly interesting.

N8allan
  • 2,138
  • 19
  • 32
  • 1
    What is the recommended sampling process? – rsilva4 Sep 17 '13 at 20:55
  • I'm evaluating MonetDB for production environment. Main table that will be queried against is about 30G and 250 mil records with simple joins on integer column. Includes several filters on bools, ints and char fields (zip code). So far tests yield several to tens of seconds. For our use case it is acceptable. I've been running tests on EC2 with 15GB RAM and attached 250GB EBS. – marcin_koss Mar 02 '15 at 04:49
7

MonetDB is first and for all a research system, but has progressed far beyond the level of the average research prototype. It is the (only) relational column-store platform in open source that I know of that supports full SQL. I have used it myself at CWI in many research projects that are not core DB research, but do need advanced DB technology.

You can see on the user's mailing list that deployments happen in many different organisations. As Roberto Cornacchia stated in a different answer, it is the backend of all Spinque deployments and we are happy MonetDB users. MonetDB is also used at a variety of non-profit projects like open streetmap and open kvk.

More and more commercial parties deploy MonetDB for analytics. (They do not always like to advertise that their analyses depend on an open source system.) Recently, MonetDB Solutions has started to provide dedicated commercial support for these deployments.

Arjen P. De Vries
  • 988
  • 7
  • 10
2

We have been using MonetDB in our business. We analyse very large data sets with many millions of rows. Traditional methods of data warehousing on SQL databases became so slow. The problem we were facing was that the data was only going to get bigger! The only way forward was to go columnar.

The results have been amazing. When you have very few joins it is staggeringly quick. Even with joins on the data sets we are looking at it is still frightening how fast it comes back.

Having seen some of the commercial partnerships I think MonetDB is going to boom over the next few years. I believe some of the major BI suppliers are using Monet under their hood to perform the large data work.

1zebedee23
  • 110
  • 7
  • Just tried Monetdb . WOW .. its lightening fast .. same aggregation query on Mysql on same machine takes 190 secs .. which monetdb takes only 5 secs ..just wondering that does it consumes all the available memory .. and how often are you inserting new data .. in my case i might need to insert around 50K recs every 1 hour.. will that be ok with monetdb ? – UberNeo Jan 16 '15 at 00:47
  • Hi @UberNeo, I'm looking for same answer: http://stackoverflow.com/questions/28122178/monetdb-refresh-data-in-background-best-strategy-with-active-connections-making . Please let me know if you have more details or news on that. Thanks – GBrian Jan 24 '15 at 04:18
0

I am thinking of implementing monetdb, I have read a lot about its responsiveness, I came to it because I was looking for a sybase iq type columnar database, but with a free license, visualizing an architecture for a hybrid lakehouse, with monetdb for the warehouse, redis for the ODS populated by event-driven services and a file system like s3 or hadoop for advanced analytics processed with pyspark, all integrated via etl nifi towards the monetdb warehouse, but also looking for success stories to support the decision . I have seen bad comparisons of relational databases like postgres v/s Monet, but they are not comparable since one is transactional and they are ready for applications and the other is columnar oriented to load large amounts of data, data analysis and business intelligence, and analytical dashboards

user2932523
  • 1
  • 1
  • 2
  • Your answer could be improved with additional supporting information. Please [edit] to add further details, such as citations or documentation, so that others can confirm that your answer is correct. You can find more information on how to write good answers [in the help center](/help/how-to-answer). – Community May 23 '23 at 12:03