1

I have a set of database documents (a few thousand) for which I would like to run some data migration activities. Each of these documents holds a reference to a different file in a Windows filesystem. These files are stored in a file server that is accessed through a shared folder. What I would like to do is to move the files to a different location in the filesystem, rearranging them based on properties in the documents.

I have thought of the following design:

  1. Have one actor that queries the database and that spawns one actor per document
  2. Each of these actors would be responsible for copying their related file to its new location, and of handling any exceptions
  3. When each of these actors finishes its task, it will inform a central actor of the outcome (success / failure)

Is this a naïve design? Should I throttle how many actors are created, or does Akka regulate the assigned resources based on processor availability? Could there be any problems on the filesystem side, having too many requests active simultaneously?

Qantas 94 Heavy
  • 15,750
  • 31
  • 68
  • 83
Eduardo
  • 8,362
  • 6
  • 38
  • 72
  • offtopic: I guess in this particular case you will be better to go with parallel collections. – om-nom-nom Oct 30 '13 at 12:18
  • @om-nom-nom: interesting. do you mean having a collection with (sourcePath, destinationPath) elements, and then mapping over it with a function that returns success / failure based on the outcome of the copy operation? – Eduardo Oct 30 '13 at 12:26
  • yep, more or less: `val xs = Seq(src -> dst, ...); xs.par.map { Try(...) }` + [parallelism hint](http://stackoverflow.com/q/14207762/298389) -- Should be simplier than horde of actors – om-nom-nom Oct 30 '13 at 12:45

1 Answers1

2

The default thread pool/execution context/dispatcher of akka will have one something like one thread per core, moving files is probably a blocking operation so you will only have as many files moved at any given moment as you have cores and during that time no other actor will be invoked.

You could configure the dispatcher to use more threads to get more parallell activity, or put the file moving actors to a specific dispatcher that does not interfer with the other actors.

Maybe a better idea would be to see the documents as the messages passing through your system, in one end you enumerate each document as a message, send it to an actor that parses the properties from the files and forward that to an actor that does the actual moving. This makes it possible to decide how many parser-actor instances and how many mover-actor instances depending on the throughput of your disks etc.

johanandren
  • 11,249
  • 1
  • 25
  • 30