Java framework to manage BLOB data outside of database

Question

I want to store my blobs outside of the database in files, however they are just random blobs of data and aren't directly linked to a file.

So for example I have a table called Data with the following columns:

id
name
comments
...

I can't just include a column called fileLink or something like that because the blob is just raw data. I do however want to store it outside of the database. I would love to create a file called 3.dat where 3 is the id number for that row entry. The only thing with this setup is that the main folder will quickly start to have a large number of files as the id is a flat folder structure and there will be OS file issues. And no the data is not grouped or structured, it's one massive list.

Is there a Java framework or library that will allow me to store and manage the blobs so that I can just do something like MyBlobAPI.saveBlob(id, data); and then do MyBlobAPI.getBlob(id) and so on? In other words something where all the File IO is handled for me?

Many apps do it just like this. Why specifically do you want this? — Ondra Žižka, May 28 '16 at 21:57
I would like the data files to be stored on a separate server which I can just call as needed, and not have to worry about how they are stored or managed, synchronization issues, etc. — Stephane Grenier, May 30 '16 at 18:42

score 1 · Answer 1 · answered May 28 '16 at 22:02

1

Simply use an appropriate database which implements blobs as you described, and use JDBC. You really are not looking for another API but a specific implementation. It's up to the DB to take care of effective storing of blobs.

answered May 28 '16 at 22:02

Ondra Žižka

43,948
41
217
277

It's not always a good thing to store blobs in the database: http://stackoverflow.com/questions/3748/storing-images-in-db-yea-or-nay#3756 – Stephane Grenier May 28 '16 at 22:18
The DB may take care of storing it as files. I recall Oracle has a plugin for this - you may access the blob using a file system path using a certain directory structure. – Ondra Žižka Jun 06 '16 at 13:47

Robert Moskal · Answer 2 · 2016-05-28T22:34:24.027

0

I think a home rolled solution will include something like a fileLink column in your table and your api will create files on the first save and then write that file on update.

I don't know of any code base that will do this for you. There are a bunch that provide an in memory file system for java. But it's only a few lines of code to write something that writes and reads java objects to a file.

You'll have to handle any file system limitations yourself. Though I doubt you'll ever burn through the limitations of modern file systems like btrfs or zfs. FAT32 is limited to 65K files per directory. But even last generation file systems support something on the order of 4 billion files per directory.

So by all means, write a class with two functions. One to serialize an object to a file; given it a unique key as a name. And another to deserialize the object by that key. If you are using a modern file system, you'll never run out of resources.

edited May 28 '16 at 22:34

answered May 28 '16 at 21:28

Robert Moskal

21,737
8
62
86

The reason for not using blobs is because the blobs will be too large, and it's never a good idea to store large amounts of blob data in the database. As for why, what happens if you have a million rows? You don't want to store a million files in a single directory ;) – Stephane Grenier May 28 '16 at 21:41
Using a database to store blobs is not an easy yes or no answer: http://programmers.stackexchange.com/questions/150669/is-it-a-bad-practice-to-store-large-files-10-mb-in-a-database – Stephane Grenier May 28 '16 at 22:04
Here's another Stackoverflow answer suggestion it's bad to store blobs in the database: http://stackoverflow.com/questions/3748/storing-images-in-db-yea-or-nay#3756 – Stephane Grenier May 28 '16 at 22:11

score 0 · Answer 3 · edited May 23 '17 at 12:10

As far as I can tell there is no framework for this. The closest I could find was Hadoop's HDFS.

That being said the advice of just putting the BLOB's into the database as per the answers below is not always advisable. Sometimes it's good and sometimes it's not, it really depends on your situation. Here are a few links to such discussions:

I did find some addition really good links but I can't remember them offhand. There was one in particular on StackOverFlow but I can't find it. If you believe you know the link please add it in the comments so that I can confirm it's the right one.

Java framework to manage BLOB data outside of database

3 Answers3