0

I am trying to make a bunch of web calls in the background, and am basically doing:

  class AmazingMultiThreadedWebCallMachine
    module Worker
      extend self

      def perform_task(row)
        response = `curl -H "Content-type: application/json" localhost:3000/fancy/path/#{row['id']}`
        ...do something with response...
      end
    end
  end

  def pool
    @pool = Concurrent::FixedThreadPool.new(10)
  end

  def execute
    rows_from_a_csv_file.each do |row|
      pool.post do
        Worker.perform_task(row)
      end
    end
  end

When I run this, I see:

- - : - - : - -   - - : - - : - -   - - : - - : - -           0  Dload  Upload   Total   Spent    Left  Speed
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0
100    20    0    20    0     0     16      0 --:--:--  0:00:01 --:--:--    17
curl: (23) Failed writing body
 -    0     0  0:  -  -     0    0     0  0      0    0      0      0    0         0  00          0  0   - - :    - 0-0 --::---:--  0:00:01 --:--:--     0-  0:00: 01 --:--:--     0     0 --:--:--  0:00:01 --:--:-- 100    20    0    20    0     0     15      0 --:--:--  0:00:01 --:--:--    15l
curl: (23) Failed writing body
100    20    0    20    0     0     14      0 --:--:--  0:00:01 --:--:--    14
curl: (23) Failed writing body
100    20    0    20    0     0     12      0 --:--:--  0:00:01 --:--:--    12
curl: (23) Failed writing body
100    20    0    20    0     0     10      0 --:--:--  0:00:01 --:--:--    10
curl: (23) Failed writing body
100    20    0    20    0     0      9      0 --:--:--  0:00:02 --:--:--    10
curl: (23) Failed writing body
100    20    0    20    0     0      9      0 --:--:--  0:00:02 --:--:--     9
curl: (23) Failed writing body
100    20    0    20    0     0      7      0 --:--:--  0:00:02 --:--:--     7
curl: (23) Failed writing body
100    20    0    20    0     0      7      0 --:--:--  0:00:02 --:--:--     7
curl: (23) Failed writing body

If I do this outside of the pool, it all works fine (except each call is blocking which is my problem)...

Does curl does not like making web calls in background threads? Is there a way to make this work?

patrick
  • 9,290
  • 13
  • 61
  • 112

0 Answers0