Encoding::UndefinedConversionError: "\xA8" from ASCII-8BIT to UTF-8 (SFTP)

Question

Using the Net-SFTP gem, Ruby 2 and Rails 4

I wrote code that was working in pure ruby, but copied my code over to rails and now I get:

Encoding::UndefinedConversionError: "\xA8" from ASCII-8BIT to UTF-8

What can I change in my code to get this working?

def self.get_recent_file(ftp_file, local_file)
    Net::SFTP.start(Config::A_FTP[:domain], Config::A_FTP[:username], :password => Config::A_FTP[:password]) do |sftp|
      sftp.download!(ftp_file, local_file)
    end
  end

Log

Encoding::UndefinedConversionError: "\xA8" from ASCII-8BIT to UTF-8
    from /usr/local/opt/rbenv/versions/2.1.0-rc1/lib/ruby/gems/2.1.0/gems/net-sftp-2.1.2/lib/net/sftp/operations/download.rb:339:in `write'
    from /usr/local/opt/rbenv/versions/2.1.0-rc1/lib/ruby/gems/2.1.0/gems/net-sftp-2.1.2/lib/net/sftp/operations/download.rb:339:in `write'
    from /usr/local/opt/rbenv/versions/2.1.0-rc1/lib/ruby/gems/2.1.0/gems/net-sftp-2.1.2/lib/net/sftp/operations/download.rb:339:in `on_read'
    from /usr/local/opt/rbenv/versions/2.1.0-rc1/lib/ruby/gems/2.1.0/gems/net-sftp-2.1.2/lib/net/sftp/request.rb:87:in `call'
    from /usr/local/opt/rbenv/versions/2.1.0-rc1/lib/ruby/gems/2.1.0/gems/net-sftp-2.1.2/lib/net/sftp/request.rb:87:in `respond_to'
    from /usr/local/opt/rbenv/versions/2.1.0-rc1/lib/ruby/gems/2.1.0/gems/net-sftp-2.1.2/lib/net/sftp/session.rb:948:in `dispatch_request'
    from /usr/local/opt/rbenv/versions/2.1.0-rc1/lib/ruby/gems/2.1.0/gems/net-sftp-2.1.2/lib/net/sftp/session.rb:911:in `when_channel_polled'
    from /usr/local/opt/rbenv/versions/2.1.0-rc1/lib/ruby/gems/2.1.0/gems/net-ssh-2.8.0/lib/net/ssh/connection/channel.rb:311:in `call'
    from /usr/local/opt/rbenv/versions/2.1.0-rc1/lib/ruby/gems/2.1.0/gems/net-ssh-2.8.0/lib/net/ssh/connection/channel.rb:311:in `process'
    from /usr/local/opt/rbenv/versions/2.1.0-rc1/lib/ruby/gems/2.1.0/gems/net-ssh-2.8.0/lib/net/ssh/connection/session.rb:222:in `block in preprocess'
    from /usr/local/opt/rbenv/versions/2.1.0-rc1/lib/ruby/gems/2.1.0/gems/net-ssh-2.8.0/lib/net/ssh/connection/session.rb:222:in `each'
    from /usr/local/opt/rbenv/versions/2.1.0-rc1/lib/ruby/gems/2.1.0/gems/net-ssh-2.8.0/lib/net/ssh/connection/session.rb:222:in `preprocess'
    from /usr/local/opt/rbenv/versions/2.1.0-rc1/lib/ruby/gems/2.1.0/gems/net-ssh-2.8.0/lib/net/ssh/connection/session.rb:205:in `process'
    from /usr/local/opt/rbenv/versions/2.1.0-rc1/lib/ruby/gems/2.1.0/gems/net-ssh-2.8.0/lib/net/ssh/connection/session.rb:169:in `block in loop'
    from /usr/local/opt/rbenv/versions/2.1.0-rc1/lib/ruby/gems/2.1.0/gems/net-ssh-2.8.0/lib/net/ssh/connection/session.rb:169:in `loop'
    from /usr/local/opt/rbenv/versions/2.1.0-rc1/lib/ruby/gems/2.1.0/gems/net-ssh-2.8.0/lib/net/ssh/connection/session.rb:169:in `loop'
... 13 levels...
    from /usr/local/opt/rbenv/versions/2.1.0-rc1/lib/ruby/gems/2.1.0/gems/net-ssh-2.8.0/lib/net/ssh/connection/session.rb:222:in `preprocess'
    from /usr/local/opt/rbenv/versions/2.1.0-rc1/lib/ruby/gems/2.1.0/gems/net-ssh-2.8.0/lib/net/ssh/connection/session.rb:205:in `process'
    from /usr/local/opt/rbenv/versions/2.1.0-rc1/lib/ruby/gems/2.1.0/gems/net-ssh-2.8.0/lib/net/ssh/connection/session.rb:169:in `block in loop'
    from /usr/local/opt/rbenv/versions/2.1.0-rc1/lib/ruby/gems/2.1.0/gems/net-ssh-2.8.0/lib/net/ssh/connection/session.rb:169:in `loop'
    from /usr/local/opt/rbenv/versions/2.1.0-rc1/lib/ruby/gems/2.1.0/gems/net-ssh-2.8.0/lib/net/ssh/connection/session.rb:169:in `loop'
    from /usr/local/opt/rbenv/versions/2.1.0-rc1/lib/ruby/gems/2.1.0/gems/net-sftp-2.1.2/lib/net/sftp/session.rb:802:in `loop'
    from /usr/local/opt/rbenv/versions/2.1.0-rc1/lib/ruby/gems/2.1.0/gems/net-sftp-2.1.2/lib/net/sftp/session.rb:787:in `connect!'
    from /usr/local/opt/rbenv/versions/2.1.0-rc1/lib/ruby/gems/2.1.0/gems/net-sftp-2.1.2/lib/net/sftp.rb:32:in `start'

Code referenced in log from GEM:

https://github.com/net-ssh/net-sftp/blob/master/lib/net/sftp/operations/download.rb#L339

# Called when a read from a file finishes. If the read was successful
  # and returned data, this will call #download_next_chunk to read the
  # next bit from the file. Otherwise the file will be closed.
  def on_read(response)
    entry = response.request[:entry]

    if response.eof?
      update_progress(:close, entry)
      entry.sink.close
      request = sftp.close(entry.handle, &method(:on_close))
      request[:entry] = entry
    elsif !response.ok?
      raise "read #{entry.remote}: #{response}"
    else
      entry.offset += response[:data].bytesize
      update_progress(:get, entry, response.request[:offset], response[:data])
      entry.sink.write(response[:data]) # <~~ Line#339
      download_next_chunk(entry)
    end
  end

what was written */download.rb:339:i* in line#339? I need to see it. — Arup Rakshit, Feb 12 '14 at 20:48
@ArupRakshit I edited the question and added what you requested — user2012677, Feb 12 '14 at 21:02
@ArupRakshit https://github.com/net-ssh/net-sftp/blob/master/lib/net/sftp/operations/download.rb#L339 — user2012677, Feb 12 '14 at 21:12

score 6 · Answer 1 · answered Apr 28 '14 at 08:01

6

This helps me:

def self.get_recent_file(ftp_file, local_file)
  local_io = File.new(local_file, mode: 'w', encoding: 'ASCII-8BIT')
  Net::SFTP.start(Config::A_FTP[:domain], Config::A_FTP[:username], :password => Config::A_FTP[:password]) do |sftp|
    sftp.download!(ftp_file, local_io)
  end
  local_io.close
end

answered Apr 28 '14 at 08:01

user72136

61
1
2

The `local_io.close` was unnecessary for me. – warder57 May 14 '16 at 13:52

score 5 · Answer 2 · edited May 23 '17 at 12:16

5

A combination of user72136's answer and the answer to this question worked for me (my remote file wasn't even ASCII):

def self.get_recent_file(ftp_file, local_file)
  local_io = File.new(local_file, mode: 'wb')
  Net::SFTP.start(Config::A_FTP[:domain], Config::A_FTP[:username], :password => Config::A_FTP[:password]) do |sftp|
    sftp.download!(ftp_file, local_io)
  end
  local_io.close
end

edited May 23 '17 at 12:16

Community

1
1

answered Oct 13 '15 at 03:02

Dimitar Koparov

51
1
3

This is the correct answer. The remote files are not ASCII and the accepted answer would convert them to ASCII, which can cause more issues. – KPheasey Jul 19 '17 at 18:35

score 1 · Answer 3 · answered Feb 12 '14 at 21:23

1

As line#339 is showing

entry.sink.write(response[:data])

Fix it as :

entry.sink.write(response[:data].force_encoding('ASCII-8BIT').encode('UTF-8'))

answered Feb 12 '14 at 21:23

Arup Rakshit

116,827
30
260
317

I was trying to avoid editing the gem, any other ideas? – user2012677 Feb 12 '14 at 21:26
@user2012677 No idea dear! – Arup Rakshit Feb 12 '14 at 21:28

score 0 · Answer 4 · answered Feb 12 '14 at 21:02

0

Change the line -

sftp.download!(ftp_file, local_file)

to say

sftp.download!(ftp_file, local_file).to_s.encode('UTF-8', {:invalid => :replace, :undef => :replace, :replace => '?'})

answered Feb 12 '14 at 21:02

Noah Davis

1,054
1
15
28

your recommendation does not work because it's saving the file to a drive. Your solution could work if it was just a string. thanks – user2012677 Feb 12 '14 at 21:16

score 0 · Answer 5 · answered Oct 01 '18 at 20:55

This problem is being produced by how Ruby opens text-files by default after Ruby 2.0 version with UTF-8 encoding. Where you open your file you can put:

local_file = Tempfile.new(encoding: 'ascii-8bit')
#or another thing to do is to switch to binary-mode
local_file = Tempfile.new
local_file.binmode

You can also open a binary-file like this:

local_file = File.open('/tmp/local_file', 'wb')

Another solution you can do is to pass to the gem-code the filepath, instead of an open file:

def self.get_recent_file(ftp_file, local_file)
  Net::SFTP.start(Config::A_FTP[:domain], Config::A_FTP[:username], :password => Config::A_FTP[:password]) do |sftp|
    sftp.download!(ftp_file, local_file.path)
  end
end

Encoding::UndefinedConversionError: "\xA8" from ASCII-8BIT to UTF-8 (SFTP)

5 Answers5