1

So I've got this server application running on Google AppEngine that I want to migrate to node.js.

My main concern is building an API similar to the powerfull GAE's blobstore/imageservice combinaison to host and serve the users uploaded pictures.

I'm really new to node.js and I've only read some documentations, viewed some video and played with node so far. I'd like to have some insight on what would be the best solutions to:

  • host the pictures
  • serve them through an efficient cache system

At the moment, I'm leaning toward redis to store and cache the pictures, I haven't looked at all the node modules yet (there are quite a lot!).

What would be your architecture of choice for such an application?

plus-
  • 45,453
  • 15
  • 60
  • 73
  • This is too big and too vague a question, and can't be reasonably answered as is. You're basically asking "how do i create a web site using stuff i've never used before". – cHao Dec 26 '11 at 05:36
  • I think there is way more vague/difficult questions than this one on stackoverflow to be honest. I don't see the difficulty for someone experienced with node to answer to those 2 or at least give feedbacks on their experience towards those problematics. – plus- Dec 26 '11 at 07:24
  • There are a lot of open questions here: are you using a cloud like Amazon AWS or Windows Azure? What are your scaling needs? Is there a reason to cache the images instead of rolling with cloud storage / CDN? The answers to those questions would help shape a better direction for you. – Justin Beckwith Dec 26 '11 at 08:15
  • @JustinBeckwith Well I plan to host it on my own box, but yes I'd like to have the door to scale if needed on a cloud service. The reason for the cache itself is because of the application requirements, so won't expend on this much. Thanks – plus- Dec 26 '11 at 08:23
  • Just because you don't know the answer, doesn't mean it's not a real question. – Michael Cole Aug 29 '15 at 16:05

1 Answers1

3

Regarding the node.js infrastructure, you will probably want to use the express framework to build the application. I don't know if you also require some image processing capabilities, but there is a node module to wrap the imagick library. It looks like an on-going development though - not sure it is already usable. Anyway, you absolutely need to avoid running any image processing code in the node single-threaded event loop.

For the upload part of your application, I would start by looking at node-formidable or this question.

Now for the storage itself, Redis is very efficient to store plenty of small objects, but not really designed to store large objects. Recent Redis versions are based on jemalloc which is a good general purpose memory allocator, but if you store large objects with Redis, it will generate some internal fragmentation. There is no memcached-like slab allocator in Redis.

So I would not store the images themselves in Redis, but only metadata and their associated indexes (file path, owner, size, tags, etc ...). Images will be better stored directly on the filesystem IMO. There is an excellent node-redis module to access Redis from node.js

I'm not sure a cache is really required. If the images are stored on the filesystem, perhaps you can rely on the filesystem cache to avoid the I/Os. Node is quite good at delegating file operations to the libeio thread pool to avoid blocking the main event loop. I'm not really convinced caching image content with Redis would bring any benefit here. I would try the filesystem cache first, and investigate more complex caching only if needed.

Community
  • 1
  • 1
Didier Spezia
  • 70,911
  • 12
  • 189
  • 154
  • Thanks, I've tried formidable and node-redis already and they fit my needs nicely. Regarding the cache, I will follow your words, it seems that I get some really good benchmark already using the 'fs' directly. It's already on par with the appengine imageservice, which I wouldn't suspect at all. – plus- Dec 26 '11 at 19:35
  • thanks Didier, was considering a ram based image cache for a project. I'll have to dig a little deeper to find a better fit. The file based solution is a little too slow for per frame for read-> process-> writes. – Mark Essel May 22 '12 at 16:44