I am trying to make a bunch of web calls in the background, and am basically doing:
class AmazingMultiThreadedWebCallMachine
module Worker
extend self
def perform_task(row)
response = `curl -H "Content-type: application/json" localhost:3000/fancy/path/#{row['id']}`
...do something with response...
end
end
end
def pool
@pool = Concurrent::FixedThreadPool.new(10)
end
def execute
rows_from_a_csv_file.each do |row|
pool.post do
Worker.perform_task(row)
end
end
end
When I run this, I see:
- - : - - : - - - - : - - : - - - - : - - : - - 0 Dload Upload Total Spent Left Speed
0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 % Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 % Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0
100 20 0 20 0 0 16 0 --:--:-- 0:00:01 --:--:-- 17
curl: (23) Failed writing body
- 0 0 0: - - 0 0 0 0 0 0 0 0 0 0 00 0 0 - - : - 0-0 --::---:-- 0:00:01 --:--:-- 0- 0:00: 01 --:--:-- 0 0 --:--:-- 0:00:01 --:--:-- 100 20 0 20 0 0 15 0 --:--:-- 0:00:01 --:--:-- 15l
curl: (23) Failed writing body
100 20 0 20 0 0 14 0 --:--:-- 0:00:01 --:--:-- 14
curl: (23) Failed writing body
100 20 0 20 0 0 12 0 --:--:-- 0:00:01 --:--:-- 12
curl: (23) Failed writing body
100 20 0 20 0 0 10 0 --:--:-- 0:00:01 --:--:-- 10
curl: (23) Failed writing body
100 20 0 20 0 0 9 0 --:--:-- 0:00:02 --:--:-- 10
curl: (23) Failed writing body
100 20 0 20 0 0 9 0 --:--:-- 0:00:02 --:--:-- 9
curl: (23) Failed writing body
100 20 0 20 0 0 7 0 --:--:-- 0:00:02 --:--:-- 7
curl: (23) Failed writing body
100 20 0 20 0 0 7 0 --:--:-- 0:00:02 --:--:-- 7
curl: (23) Failed writing body
If I do this outside of the pool, it all works fine (except each call is blocking which is my problem)...
Does curl does not like making web calls in background threads? Is there a way to make this work?