Can YDN-DB handle large entries at once?

Question

Im having troubles trying to retrieve a large amount of data using Ydn-db.

The problem is:

-I have a large JSon file that I want to store in my app to use it offline

-I query the file and then save the data successfully using db.put

-Then, if I "print" the stored data using db.values, I only get a portion of the text I previously saved.

You can also test it usign the Todo list demo given as an example here: http://dev.yathit.com/demo/ydn-db/todo.html

If you enter, lets say a 1MB text in the input, you wont get the whole text saved but just a fraction of it.

Is there a way to get around this issue?

Thanks!

EDIT: Here is a working example of what Im talking about http://flatic.com/test.html

EDIT 2: Ok, I think I found a temporary solution, it looks like YDN-DB can't store more than 100 Json objects so instead of saving my Json data directly like:

db.put('table',largeJsonData);

I first put the largeJsonData as string inside a simple Json Array, something like this:

var data = {
"json":largeJsonData
};

db.put('table',data);

Now I can read the data doing:

db.values('table').done(function(items) {
console.log(items[0].json);
});

But you wont be able to make index search or get a specific value by any given Id

I think you will find a limit based on this subsystem: http://stackoverflow.com/questions/2989284/what-is-the-max-size-of-localstorage-values — Jason Sperske, Mar 19 '14 at 23:53

Kyaw Tun · Accepted Answer · 2014-03-21T00:32:09.540

1

Please prove your claim by a simple test. I don't think browser limit to 1mb per record. If fail, it fail whole. Partial commit is just impossible.

If websql, may be lost during JSON serialization. Possible. Haven't tested.

EDIT after question update.

Your data is not text, you parse it - and becomes array of JSON objects. db.put will insert them individually as a separate record. db.values has 100 records limit. If you put a limit on db.values('todo', null, 100000), you should get all.

100 limit is common surprise for user. I hate surprise on API. But still I feel there should be a limit on that method.

edited Mar 21 '14 at 00:32

answered Mar 19 '14 at 23:58

Kyaw Tun

12,447
10
56
83

Curiously I'm able to replicate the text truncation at 4092 bytes. Here's a 1mb string I used on the Todo demo: https://gist.github.com/editor/f85257b29f51ac5a3280 Could it be the demo and not the library? – buley Mar 20 '14 at 01:33
The demo well represents library. Is that copy/paste problem? – Kyaw Tun Mar 20 '14 at 01:53
Seems like a solid guess – buley Mar 20 '14 at 02:05
I don't think its a copy paste issue nor a browser limitation or input problem. I have the same problem when I put the large text inside a variable. I tested it with Firefox, Chrome, Safari and I always get a truncated string. – Crash Override Mar 20 '14 at 12:02
Thanks Kyaw, that solved the problem! And thanks for this amazing API and great support also. – Crash Override Mar 21 '14 at 11:28
Hi again, I'm sorry to bother you guys again with the same thing but after I increased the size of my Json file I got stuck with the same problem, even if I increase the limit in `db.values('todo', null, 100000)` I have 1647 Json objects but after using `db.values` I only get 1623. Here is a working example [link](http://flatic.com/test.html) (I print the result in the console btw) – Crash Override Mar 27 '14 at 21:06
1

I think I found the problem, its like db.values doesn't accept similar objects inside the same array, for example if you make a Json array with 5 objects with the same content it will take the array as if it only has 1 object. I see that my Json array has two similar objects that differ in 1 number only and thats creating a problem here. – Crash Override Mar 27 '14 at 23:39
1

Ok.. I changed the keypath to a random auto-incremented one so they are always unique. Now its finally working! The problem was that the Keypath weren't unique. – Crash Override Mar 27 '14 at 23:56

buley · Answer 2 · 2014-03-20T02:28:03.323

On the demo, it appears the browser is truncating the input field.

String.length gives you ~bytesize in English. I used copy/pasted .87 mb string of "stackoverflow.com|": https://gist.github.com/editor/f85257b29f51ac5a3280

You can see here I take my big_string and assign it to the input field value.

enter image description here

When I read it back, it's 397308 bytes smaller.

enter image description here

It's true that IDB is bound by it's implemention technologies (LevelDB in Chrome, SQLLite in Firefox). For example I guarantee writes are going to be low with large data objects. But if Kyaw says there are no library limitations as the author of the library I would trust him as far as YDN is concerned.

Can YDN-DB handle large entries at once?

2 Answers2