Transfer large data from one server to another

Question

I m currently trying to transfer large data from one server to another using php cURL (posting the data). In some cases the remote server is getting incomplete data(corrupted).

Is there any other way to achieve this reliably

EDIT - 1

Using FTP seems good idea, anybody would like to say that it is bad or i should avoid it for any reason (Suggestions - @Ed Heal, @Neo)

In what way is it being corrupted (end of line problem perhaps)? Is it truncated? — Ed Heal, Jan 02 '12 at 15:09
Related: [$POST doesn't contain all data](http://stackoverflow.com/questions/8700163/post-doesnt-contain-all-data) — hakre, Jan 02 '12 at 15:13
@clyfe - I have checks in place, but looking for solution how to send it. — Sudesh, Jan 02 '12 at 15:16
@hakre - How should override this so that data is completely transferred (Transferring with mime-type JSON/Text would work) — Sudesh, Jan 02 '12 at 15:18
What is the size of data you receive? Did you try changing `post_max_size` in `php.ini`? — Shiplu Mokaddim, Jan 02 '12 at 15:20

score 1 · Answer 1 · edited May 23 '17 at 11:48

1

I would guess your php session is timing out. See How to increase the execution timeout in php?

Or you could get curl to run in it's own thread. Call it from a bash script maybe.

edited May 23 '17 at 11:48

Community

1
1

answered Jan 02 '12 at 15:13

Kerry Woolgar

11
2

score 0 · Answer 2 · answered Jan 02 '12 at 15:11

0

Posting large files is not what http is for. Ftp is for transferring files. Hence the name.

But if you are stuck on using http, you can take a look at the WebDAV extensions to http. There is a php library called SabreDAV that you should take a look at:

http://code.google.com/p/sabredav/

answered Jan 02 '12 at 15:11

Anders Marzi Tornblad

18,896
9
51
66

1

Eeek, anything but using FTP. See http://stackoverflow.com/questions/125893/is-there-an-alternative-to-ftp for alternatives. HTTP is ok. – FlappySocks Jan 02 '12 at 15:26

score 0 · Answer 3 · answered Jan 02 '12 at 15:14

0

You can even use scp to do so, so that the data transfer is secure as well. You would be able to find libraries to do so. Also basic function in php can be useful: http://php.net/manual/en/function.ssh2-sftp.php

answered Jan 02 '12 at 15:14

Neo

6,753
1
19
25

I could use FTP, but using cURL as i want to process this data. Using FTP i would have to first transfer file then call another script to process it (that would be the flow), also at maintenance end i will have to keep a record of FTP account(In my case accounts) in PHP file – Sudesh Jan 02 '12 at 15:23
would it be possible for you divide the content in chunks and have checksum with each chuck checked on other end? – Neo Jan 02 '12 at 15:33
I will have to calculate the sizes of chunks, break them and send then join them hmmm..., also the size of post would vary from server to server. FTP seems better has some maintenance work but thats fine. Do you see any FTP issues – Sudesh Jan 02 '12 at 15:40
FTP will be fine, but better use SFTP. – Neo Jan 02 '12 at 16:30

score 0 · Answer 4 · answered Jan 02 '12 at 15:14

0

As you say that it is truncated, I would imaging that the server has a file limitation size - i.e. to prevent abuse and denial of service attacks.

I would stick to FTP and perhaps compressing the files.

answered Jan 02 '12 at 15:14

Ed Heal

59,252
17
87
127

Transfer large data from one server to another

4 Answers4