Passing data from C to R

Question

I am calling a C function from R using .C. This is a simulation which will run for a few minutes, and every few iterations, I would like to send some information of the progress to R. That is, I do not want to wait till the C function finishes to send all the info at once to R.

Note: I do not want to PRINT in R (Rprintf does that). But I want to pass such info to R. Also using error passes the error to R if R.h is included, but I am not interested in exception handling.

My first direction: I use futile.logger in R to log such stuff. Preferably, if such information can be passed to the same logger that the calling R function is using, would be great. But I could not find any examples on the web.

Alternative direction: I am also using redis to write information in cache, which are then consumed by others that connect to the redis db. But I don't find any C interface to redis. I do NOT want to use Lua. The closest I have found is Writing a Custom Redis Command In C - Part 2.

But my needs are far simpler in my opinion. Any ideas?

Update: This is how I am hoping this will work ideally.

# PART 1: webserver calls R function

# the R call
res = .C("montecarlo_sampler.c", as.matrix(inputData), as.matrix(ouputData), as.integer(iterations))


// PART 2: the C function

void montecarlo_sampler( double *inputData, double *outputData, int *iterations){

  // do some preprocessing
  int iter =1;
  while(iter<1000000){

    if(iter % 1000 == 0) {
      // summarize output from last 1000 iterations
      // dump summary data to a logger or redis
    }

    // do usual sampling stuff in C
  }
}

PART 3:
// listening on the django side
// polls redis every few seconds to see if update has reached.
// sends summary output for visualization to client

What kind of progress info do you need ? If you want to consume data, probably the easiest way is to add an index argument to your C function, that will only compute a part of the problem and ouput it to R, along with an index that allow to compute the next part of the problem. — Karl Forner, Sep 23 '13 at 08:47
Maybe http://stackoverflow.com/questions/6658168/passing-a-data-frame-from-to-r-and-c-using-call is useful for you. — Tim, Sep 23 '13 at 08:48
@KarlForner - I am doing simulation, so I want to output estimated parameter values every say 100,000 iterations. These parameter values are used by another engine as and when they arrive intermittently. — user1971988, Sep 23 '13 at 10:20
@Tim - I have to wait for the function to end to return based on the idea in the link. I want to avoid that. Hence my insistence on logging related frameworks. — user1971988, Sep 23 '13 at 10:21
are your other engine processing in parallel ? It could be as simple as: calling the C function to compute the next estimations, feeding to your other engine (possibly in parallel), calling again... — Karl Forner, Sep 23 '13 at 14:14

score 0 · Answer 1 · answered Sep 23 '13 at 12:37

You're really asking to do parallel computation in a short pipeline. The "easiest" way to do this might be with socket connections, following the example "Two R processes communicating via non-blocking sockets" on ?socketConnection, in conjunction with the suggestion of @KarlForner. Process 1 would

repeat {
    ## get input from process 2
    ## do R calculations
}

The other process would

repeat {
    ## call C for a chunk of results
    ## forward result to process 1
}

This could also be done with the parallel package on non-Windows, using mcparallel to fork the child process and communicating with mccollect. There are not many examples of this kind of pipeline implementation; I think the processes would have to establish some third-party communication, like a socket(!) or redis (communicate with redis at the R level, rather than C). It might also be possible to fork the calculation in C, using e.g., openMP directives but with a similar scheme -- the R process calling in to C to poll for results in some agreed-upon location. It would be interesting to see your solution, e.g., as an update to your question; it might help to revise the title of your post to reflect the challenge you're working on.

Thanks @Martin - I am now trying to call the futile.logger function flag.info in R using the code structure in call_R. — user1971988, Oct 04 '13 at 12:25
The other option is credis which I will explore later. Both C and Redis can then get the information from redis via predefined keys. I will put my results here when I am done. Thanks for your help. — user1971988, Oct 04 '13 at 12:35

Passing data from C to R

1 Answers1