HTTP 1.1 requests are pipelined by default. If you don't "Connection: Close", it assumes "Connection: Keep-Alive", and then you have to wait for the connection to time out (since you never explicitly closed it) before the next loop will start.
Answered by TML -> file_get_contents() with context to use http/1.1 significantly slow download speeds
To avoid the risk of getting slowed down to get the response using HTTP 1.1, you must pass header connection close in wp_remote_get :
$response = wp_remote_get( 'http://www.example.com/index.php?action=foo',
array(
'timeout' => 120,
'httpversion' => '1.1',
'headers' => 'Connection: close'
)
);