0

I am trying to set up Curl connection for Gnip Streaming. My objective is to download all incoming data in text and then upload the same in the database. But the curl request is getting dropped after some time. Is there any method to set up the process for streaming data.

curl --compressed -v -u <login>:<password> "https://gnip-stream.twitter.com/stream/sample10/accounts/UpSwell/publishers/twitter/prod.json?partition=1" --connect-time 900 -o output.txt

After sometime I am getting "closing connection".

pankaj jha
  • 299
  • 5
  • 15

1 Answers1

2

It looks like your curl command is fine but the issue is that you are not keeping up with the volume on the stream. This causes a data buffer on the server side to completely fill, which triggers a 'forced disconnect' from the server.

Fundamental recommendation is to make sure you have the bandwidth to stream that data volume. Often these forced disconnects are caused by processing bottlenecks in the client application. However, you are using curl, which is extremely efficient... curl is typically recommended to take the client side app out of the loop and to test the underlying bandwidth...

Seems that you were on a trial of the decahose product... reach out to account manager to discuss next steps there.

Thanks!

Jim Moffitt
  • 46
  • 1