0

I need to sync user's phone contacts to my app with a rails API backend. The frontend sends the dump of all contacts and the backend needs to parse, validate and create ActiveRecords in an async job. I'm having trouble evaluating and choosing the temporary storage from which the async process can consume the input. I have been considering Redis but would like to know if there are more options and what are the parameters to think of while evaluating them. Some of the parameters I could think of are 1. Speed of writing the data to be dumped 2. Storing the data securely, like escaping before storing

Any suggestions or references would be helpful. Thanks!

shala
  • 1
  • how many contacts could be sent in one request ? is it a time consuming operation so it must be added to a background job? – Abdullah Fadhel Apr 27 '20 at 07:23
  • @AbdullahFadhel An average case should not have more than 1000 - 2000 contacts but there may be cases where the number is larger and the experience might be affected. Say 6000+ contacts? It seemed like the kind of thing that should be handled asynchronously, open to suggestions otherwise – shala Apr 27 '20 at 09:44
  • convert the contacts to a string then passed to the worker like this – Abdullah Fadhel Apr 27 '20 at 09:49
  • https://stackoverflow.com/questions/49831276/is-it-a-good-idea-to-pass-a-huge-string-as-an-argument-to-sidekiq-worker check this if it will help you. the last answer seems good – Abdullah Fadhel Apr 27 '20 at 09:52
  • so you can store them in a temp table for later processing by your worker. or use some external storage like S3 and fetch them later through the url so you can use them. I find the temp table is good option. since storing to s3 could be a hassle since you want to make a worker to upload to s3, which uses another worker which will process the url later. – Abdullah Fadhel Apr 27 '20 at 09:55
  • if you could use a nosql db like mongo for temporarily storing the contacts for later processing that would be a nice idea in my view. – Abdullah Fadhel Apr 27 '20 at 10:00
  • Ok, this helps. I will try the temp table approach. Thanks! @AbdullahFadhel – shala Apr 27 '20 at 10:00

1 Answers1

0

If you use Sidekiq as your background job processor you can safely just pass the whole payload as an argument to the job. It will be stored in Redis and then easily available in your background job (no fetching from any store, Sidekiq does it for you out of the box).

Redis is pretty good at handling large payloads, a single object in Redis can take up to 512mb so don't be afraid to store here even 100k contacts.

P. Boro
  • 716
  • 3
  • 12