2

The problem: I am trying to create a local mirror of a public FTP site. When I use lftp to do the job it creates a mirror without a problem, but when I try to update the mirror a few days later it becomes very slow due to getting stuck on several files.

Running lftp -d I can see that lftp makes several requests to RETR the file, but these requests result in several **** Timeout - reconnecting messages and after about 2-3 minutes I see

<--- 150 Opening BINARY mode data connection for {filename removed}.`

After this last command the file successfully downloads and lftp proceeds further.

From the manual I understand that BINARY is the default mode for lftp, but somehow it doesn't seem to work for the early requests. Can someone suggest how I can force lftp to always open BINARY mode data connection to download all files?

Here's a MWE:

``lftp -d -u anonymous,anonymous -c "open {url}; get {file}"``

And response from lftp -d:

---- Connecting to {url} ({IP}) port 21
<--- 220 (vsFTPd 3.0.3)                          
---> FEAT
<--- 211-Features:                                     
<---  EPRT
<---  EPSV
<---  MDTM
<---  PASV
<---  REST STREAM
<---  SIZE
<---  TVFS
<--- 211 End
---> USER anonymous
<--- 331 Please specify the password.            
---> PASS anonymous
<--- 230 Login successful.                                      
---> TYPE I
<--- 200 Switching to Binary mode.                         
---> SIZE {file}
<--- 213 3321                                              
---> MDTM {file}
--- 213 20160318190446                                         
---> PASV
<--- 227 Entering Passive Mode ({IP}).        
---- Connecting data socket to ({IP}) port 55380
---- Data connection established                                  
---> RETR {file}
**** Timeout - reconnecting                                     
---- Closing data socket
---- Closing control socket
econ
  • 547
  • 7
  • 22

2 Answers2

3

lftp uses binary mode by default for all file transfers and ascii mode for directory listings. So the binary mode should not be a problem here.

Maybe you have a subtle connectivity problem, sometimes setting net:socket-maxseg to a lower value (e.g. 500) helps.

lav
  • 1,351
  • 9
  • 17
  • I did not try the setting you've suggested, because using `--ignore-time` resolved my issue. I accepted your answer since you've made it clear that binary mode is always used. – econ Apr 18 '16 at 21:39
1

The ftp command for binary mode is bin so use that command before you get the file.

cd /direc/tory; bin; get file.xml

The problem is that I see

---> TYPE I
<--- 200 Switching to Binary mode.

in your output, so you're already in binary mode. I wonder if you have a different problem? I also see that you're using passive mode (PASV), and that's good because passive works around firewalls and NATs [1], so we need another reason why you see those timeouts.

Do you have any other clues, maybe from ping or netstat?

Community
  • 1
  • 1
nethope
  • 19
  • 3
  • I am starting to suspect that I am being throttled by the server... the reason for that is that opening any files on the ftp server (e.g. via lynx) becomes veeery slow. – econ Apr 17 '16 at 02:47
  • However, oddly, if I cancel and restart lftp, it begins from the very first folder and re-downloads files that it already updated from the cancelled session (without any errors or delays)... so I'm not quite sure what to make of it. – econ Apr 17 '16 at 02:48