Why won't this Ruby script for text replacement work?

Question

I have some simple text processing scripts I use all the time, and I want to translate those scripts into Ruby, to get familiar with the language.

This is the first script I can't get to run:

#!/usr/bin/env ruby

@text = ARGF.read

@replacements = [{:from=>"—", :to=>". "}, {:from=>"ﬄ", :to=>"ffl"}, {:from=>"ﬃ", :to=>"ffi"}, {:from=>"ﬁ", :to=>"fi"}, {:from=>"ﬂ", :to=>"fl"}, {:from=>"ﬀ", :to=>"ff"}, {:from=>"æ", :to=>"ae"}, {:from=>"é", :to=>"e"}, {:from=>"Ç", :to=>"s"}, {:from=>"ü", :to=>"u"}, {:from=>"â", :to=>"a"}, {:from=>"ä", :to=>"a"}, {:from=>"à", :to=>"a"}, {:from=>"å", :to=>"a"}, {:from=>"ç", :to=>"s"}, {:from=>"ê", :to=>"e"}, {:from=>"ë", :to=>"e"}, {:from=>"è", :to=>"e"}, {:from=>"ï", :to=>"i"}, {:from=>"î", :to=>"i"}, {:from=>"ì", :to=>"i"}, {:from=>"Ä", :to=>"a"}, {:from=>"Å", :to=>"a"}, {:from=>"É", :to=>"e"}, {:from=>"ô", :to=>"oh"}, {:from=>"ö", :to=>"oe"}, {:from=>"ò", :to=>"o"}, {:from=>"û", :to=>"uu"}, {:from=>"ù", :to=>"u"}, {:from=>"ÿ", :to=>"o"}, {:from=>"Ö", :to=>"o"}, {:from=>"Ü", :to=>"u"}, {:from=>"á", :to=>"ah"}, {:from=>"í", :to=>"ee"}, {:from=>"ó", :to=>"oh"}, {:from=>"ú", :to=>"uu"}, {:from=>"ñ", :to=>"ny"}, {:from=>"Ñ", :to=>"ny"}] 

@replacements.each do |pair|
  @text.gsub!(/#{pair[:from]}/, pair[:to])
end

puts @text

And here's the error I get:

/home/alec/.bei/under-boac:5: invalid multibyte char (US-ASCII)
/home/alec/.bei/under-boac:5: invalid multibyte char (US-ASCII)
/home/alec/.bei/under-boac:5: syntax error, unexpected $end, expecting '}'
@replacements = [{:from=>"—", :to=>". "}, {:from=>"ﬄ"...
                            ^

I based part of it on "Best practices with STDIN in Ruby?".

What version of Ruby? Looks like you need to allow UTF-8; are you using 1.8? — Dave Newton, Jan 09 '13 at 20:00
Hmm, worked fine for me; `ruby 1.9.3p194 (2012-04-20 revision 35410) [x86_64-darwin11.4.0]` — Dave Newton, Jan 09 '13 at 20:12
It is a bad practice to have such a long line in the code unless for minifying. — sawa, Jan 10 '13 at 07:13

score 4 · Accepted Answer · answered Jan 09 '13 at 20:34

This is your base code, reformatted for readability:

@replacements = [
  { :from => "—", :to => ". "  },
  { :from => "ﬄ", :to => "ffl" },
  { :from => "ﬃ", :to => "ffi" },
  { :from => "ﬁ", :to => "fi"  },
  { :from => "ﬂ", :to => "fl"  },
  { :from => "ﬀ", :to => "ff"  },
  { :from => "æ", :to => "ae"  },
  { :from => "é", :to => "e"   },
  { :from => "Ç", :to => "s"   },
  { :from => "ü", :to => "u"   },
  { :from => "â", :to => "a"   },
  { :from => "ä", :to => "a"   },
  { :from => "à", :to => "a"   },
  { :from => "å", :to => "a"   },
  { :from => "ç", :to => "s"   },
  { :from => "ê", :to => "e"   },
  { :from => "ë", :to => "e"   },
  { :from => "è", :to => "e"   },
  { :from => "ï", :to => "i"   },
  { :from => "î", :to => "i"   },
  { :from => "ì", :to => "i"   },
  { :from => "Ä", :to => "a"   },
  { :from => "Å", :to => "a"   },
  { :from => "É", :to => "e"   },
  { :from => "ô", :to => "oh"  },
  { :from => "ö", :to => "oe"  },
  { :from => "ò", :to => "o"   },
  { :from => "û", :to => "uu"  },
  { :from => "ù", :to => "u"   },
  { :from => "ÿ", :to => "o"   },
  { :from => "Ö", :to => "o"   },
  { :from => "Ü", :to => "u"   },
  { :from => "á", :to => "ah"  },
  { :from => "í", :to => "ee"  },
  { :from => "ó", :to => "oh"  },
  { :from => "ú", :to => "uu"  },
  { :from => "ñ", :to => "ny"  },
  { :from => "Ñ", :to => "ny"  }
] 

@replacements.each do |pair|
  @text.gsub!( /#{ pair[:from] }/, pair[:to] )
end

That can be simplified, and the hashes combined to be one big one, and, incidently, be used as a hash should be used:

# encoding: utf-8

@replacements = {
  "—" => ". "  ,
  "ﬄ" => "ffl" ,
  "ﬃ" => "ffi" ,
  "ﬁ" => "fi"  ,
  "ﬂ" => "fl"  ,
  "ﬀ" => "ff"  ,
  "æ" => "ae"  ,
  "é" => "e"   ,
  "Ç" => "s"   ,
  "ü" => "u"   ,
  "â" => "a"   ,
  "ä" => "a"   ,
  "à" => "a"   ,
  "å" => "a"   ,
  "ç" => "s"   ,
  "ê" => "e"   ,
  "ë" => "e"   ,
  "è" => "e"   ,
  "ï" => "i"   ,
  "î" => "i"   ,
  "ì" => "i"   ,
  "Ä" => "a"   ,
  "Å" => "a"   ,
  "É" => "e"   ,
  "ô" => "oh"  ,
  "ö" => "oe"  ,
  "ò" => "o"   ,
  "û" => "uu"  ,
  "ù" => "u"   ,
  "ÿ" => "o"   ,
  "Ö" => "o"   ,
  "Ü" => "u"   ,
  "á" => "ah"  ,
  "í" => "ee"  ,
  "ó" => "oh"  ,
  "ú" => "uu"  ,
  "ñ" => "ny"  ,
  "Ñ" => "ny"  
}
@replacements.each do |k,v|
  @text.gsub!(k, v)
end

Note: use the "encoding comment" to help Ruby understand the encoding of the characters.

But using the same hash, that loop can be reduced down to this, which runs extremely fast:

@text.gsub!(Regexp.union(@replacements.keys), @replacements)

score 2 · Answer 2 · answered Jan 09 '13 at 20:17

If you're using non 7-bit ASCII in your source file, you need to add a header to identify what character set is employed. Example:

#!/usr/bin/env ruby
# encoding: utf-8

Further, you're doing something here that's exceedingly inefficient, requiring O(N) passes on the file when you can do this in O(1) if you construct the proper regular expression:

replace_map = Hash[@replacements.collect { |r| [ r[:from], r[:to] ] }]
replace_regex = Regexp.new("(#{replace_map.keys.collect { |r| Regexp.escape(r) }.join('|')})")

@text.gsub!(replace_regexp) do |s|
  replace_map[s[1]]
end

It would be a lot easier to work with a key/value mapping instead of the strange :from / :to pairing you have there.

score 2 · Answer 3 · answered Jan 09 '13 at 20:26

#encoding: utf-8

@text = "öbñ"
@replacements = [{:from=>"—", :to=>". "}, {:from=>"ﬄ", :to=>"ffl"}, {:from=>"ﬃ", :to=>"ffi"}, {:from=>"ﬁ", :to=>"fi"}, {:from=>"ﬂ", :to=>"fl"}, {:from=>"ﬀ", :to=>"ff"}, {:from=>"æ", :to=>"ae"}, {:from=>"é", :to=>"e"}, {:from=>"Ç", :to=>"s"}, {:from=>"ü", :to=>"u"}, {:from=>"â", :to=>"a"}, {:from=>"ä", :to=>"a"}, {:from=>"à", :to=>"a"}, {:from=>"å", :to=>"a"}, {:from=>"ç", :to=>"s"}, {:from=>"ê", :to=>"e"}, {:from=>"ë", :to=>"e"}, {:from=>"è", :to=>"e"}, {:from=>"ï", :to=>"i"}, {:from=>"î", :to=>"i"}, {:from=>"ì", :to=>"i"}, {:from=>"Ä", :to=>"a"}, {:from=>"Å", :to=>"a"}, {:from=>"É", :to=>"e"}, {:from=>"ô", :to=>"oh"}, {:from=>"ö", :to=>"oe"}, {:from=>"ò", :to=>"o"}, {:from=>"û", :to=>"uu"}, {:from=>"ù", :to=>"u"}, {:from=>"ÿ", :to=>"o"}, {:from=>"Ö", :to=>"o"}, {:from=>"Ü", :to=>"u"}, {:from=>"á", :to=>"ah"}, {:from=>"í", :to=>"ee"}, {:from=>"ó", :to=>"oh"}, {:from=>"ú", :to=>"uu"}, {:from=>"ñ", :to=>"ny"}, {:from=>"Ñ", :to=>"ny"}] 

# Hammer the @replacements into one Hash like {"—"=>". ", "ﬄ"=>"ffl"}:
from_to = Hash[@replacements.map{|h| h.values}]
# Generate one Regular expression to catch all keys:
re = Regexp.union(from_to.keys)
# Let gsub do the work in one pass:
@text.gsub!(re, from_to)

This does about the same as @tadman's code. The #encoding: utf-8 should solve your problem; the rest of the lines prevent scanning the text 38 times.

score 0 · Answer 4 · answered Jul 09 '14 at 16:24

You even can simplify:

@replacements=Hash[*%w[— .  ﬄ ffl ﬃ ffi ﬁ fi ﬂ fl ﬀ ff æ ae é e Ç s ü u â a ä a à a å a ç s ê e ë e è e ï i î i ì i Ä a Å a É e ô oh ö oe ò o û uu ù u ÿ o Ö o Ü u á ah í ee ó oh ú uu ñ ny Ñ ny]]

Why won't this Ruby script for text replacement work?

4 Answers4