How to URL encode a string in Ruby

Question

How do I URI::encode a string like:

\x12\x34\x56\x78\x9a\xbc\xde\xf1\x23\x45\x67\x89\xab\xcd\xef\x12\x34\x56\x78\x9a

to get it in a format like:

%124Vx%9A%BC%DE%F1%23Eg%89%AB%CD%EF%124Vx%9A

as per RFC 1738?

Here's what I tried:

irb(main):123:0> URI::encode "\x12\x34\x56\x78\x9a\xbc\xde\xf1\x23\x45\x67\x89\xab\xcd\xef\x12\x34\x56\x78\x9a"
ArgumentError: invalid byte sequence in UTF-8
    from /usr/local/lib/ruby/1.9.1/uri/common.rb:219:in `gsub'
    from /usr/local/lib/ruby/1.9.1/uri/common.rb:219:in `escape'
    from /usr/local/lib/ruby/1.9.1/uri/common.rb:505:in `escape'
    from (irb):123
    from /usr/local/bin/irb:12:in `<main>'

Also:

irb(main):126:0> CGI::escape "\x12\x34\x56\x78\x9a\xbc\xde\xf1\x23\x45\x67\x89\xab\xcd\xef\x12\x34\x56\x78\x9a"
ArgumentError: invalid byte sequence in UTF-8
    from /usr/local/lib/ruby/1.9.1/cgi/util.rb:7:in `gsub'
    from /usr/local/lib/ruby/1.9.1/cgi/util.rb:7:in `escape'
    from (irb):126
    from /usr/local/bin/irb:12:in `<main>'

I looked all about the internet and haven't found a way to do this, although I am almost positive that the other day I did this without any trouble at all.

Maybe useful if using Ruby 1.9: http://yehudakatz.com/2010/05/05/ruby-1-9-encodings-a-primer-and-the-solution-for-rails/ — apneadiving, Jul 15 '11 at 23:38

score 203 · Accepted Answer · edited Apr 26 '20 at 02:03

203

str = "\x12\x34\x56\x78\x9a\xbc\xde\xf1\x23\x45\x67\x89\xab\xcd\xef\x12\x34\x56\x78\x9a".force_encoding('ASCII-8BIT')
puts CGI.escape str


=> "%124Vx%9A%BC%DE%F1%23Eg%89%AB%CD%EF%124Vx%9A"

edited Apr 26 '20 at 02:03

Matthias Winkelmann

15,870
7
64
76

answered Jul 15 '11 at 23:43

kain

5,510
3
27
36

2

`force_encoding('binary')` might be a more self-documenting choice. – mu is too short Jul 15 '11 at 23:53
68

They deprecated that method, use * `CGI.escape` * instead. -> [http://www.ruby-forum.com/topic/207489#903709](http://www.ruby-forum.com/topic/207489#903709). You should also be able to use `URI.www_form_encode` * `URI.www_form_encode_component` *, but I have never used those – J-Rou Jul 06 '12 at 14:36
2

No need to `require 'open-uri'` here. Did you mean `require 'uri'`? – pje Sep 26 '13 at 20:59
1

@J-Rou, CGI.escape can escape whole URL, it does not selectively escapes query parameters, for instance, if you pass `'a=&!@&b=&$^'` to CGI.escape it will escape whole thing with query separators `&` so this could be used only to query values. I suggest using `addressable` gem , it is more intellectual working with urls. – Alexander.Iljushkin Oct 13 '15 at 19:46
I needed to access files on remote server. Encoding with CGI didn't work, but URI.encode did the work just fine. – Tashows Mar 19 '19 at 14:34

score 111 · Answer 2 · edited Jun 21 '22 at 22:44

Nowadays, you should use ERB::Util.url_encode or CGI.escape. The primary difference between them is their handling of spaces:

>> ERB::Util.url_encode("foo/bar? baz&")
=> "foo%2Fbar%3F%20baz%26"

>> CGI.escape("foo/bar? baz&")
=> "foo%2Fbar%3F+baz%26"

CGI.escape follows the CGI/HTML forms spec and gives you an application/x-www-form-urlencoded string, which requires spaces be escaped to +, whereas ERB::Util.url_encode follows RFC 3986, which requires them to be encoded as %20.

See "What's the difference between URI.escape and CGI.escape?" for more discussion.

score 73 · Answer 3 · answered Jan 30 '14 at 20:33

73

str = "\x12\x34\x56\x78\x9a\xbc\xde\xf1\x23\x45\x67\x89\xab\xcd\xef\x12\x34\x56\x78\x9a"
require 'cgi'
CGI.escape(str)
# => "%124Vx%9A%BC%DE%F1%23Eg%89%AB%CD%EF%124Vx%9A"

Taken from @J-Rou's comment

answered Jan 30 '14 at 20:33

Jared Beck

16,796
9
72
97

score 13 · Answer 4 · edited Feb 06 '20 at 00:21

13

I was originally trying to escape special characters in a file name only, not on the path, from a full URL string.

ERB::Util.url_encode didn't work for my use:

helper.send(:url_encode, "http://example.com/?a=\11\15")
# => "http%3A%2F%2Fexample.com%2F%3Fa%3D%09%0D"

Based on two answers in "Why is URI.escape() marked as obsolete and where is this REGEXP::UNSAFE constant?", it looks like URI::RFC2396_Parser#escape is better than using URI::Escape#escape. However, they both are behaving the same to me:

URI.escape("http://example.com/?a=\11\15")
# => "http://example.com/?a=%09%0D"
URI::Parser.new.escape("http://example.com/?a=\11\15")
# => "http://example.com/?a=%09%0D"

edited Feb 06 '20 at 00:21

the Tin Man

158,662
42
215
303

answered Mar 24 '16 at 00:25

kangkyu

5,252
3
34
36

The only actual answer that I could find. Thank you. – akostadinov Mar 08 '21 at 12:09
This whole issue is a mess! Thanks for shedding some real light on it. I wasted at least a day chasing my tail before finding this! – Russell Fulton Apr 28 '21 at 00:14

score 12 · Answer 5 · edited Feb 06 '20 at 00:16

You can use Addressable::URI gem for that:

require 'addressable/uri'   
string = '\x12\x34\x56\x78\x9a\xbc\xde\xf1\x23\x45\x67\x89\xab\xcd\xef\x12\x34\x56\x78\x9a'
Addressable::URI.encode_component(string, Addressable::URI::CharacterClasses::QUERY)
# "%5Cx12%5Cx34%5Cx56%5Cx78%5Cx9a%5Cxbc%5Cxde%5Cxf1%5Cx23%5Cx45%5Cx67%5Cx89%5Cxab%5Cxcd%5Cxef%5Cx12%5Cx34%5Cx56%5Cx78%5Cx9a"

It uses more modern format, than CGI.escape, for example, it properly encodes space as %20 and not as + sign, you can read more in "The application/x-www-form-urlencoded type" on Wikipedia.

2.1.2 :008 > CGI.escape('Hello, this is me')
 => "Hello%2C+this+is+me" 
2.1.2 :009 > Addressable::URI.encode_component('Hello, this is me', Addressable::URI::CharacterClasses::QUERY)
 => "Hello,%20this%20is%20me"

Also can do like this: `CGI.escape('Hello, this is me').gsub("+", "%20")` `=> Hello%2C%20this%20is%20me"` if don't want to use any gems — Raccoon, Oct 29 '18 at 11:57

score 8 · Answer 6 · answered Jul 30 '19 at 19:10

8

Code:

str = "http://localhost/with spaces and spaces"
encoded = URI::encode(str)
puts encoded

Result:

http://localhost/with%20spaces%20and%20spaces

answered Jul 30 '19 at 19:10

Thiago Falcao

4,463
39
34

If the receiving server is old, it might not respond well to CGI.escape. This is still a valid alternative. – cesartalves Jul 14 '20 at 14:30

score 6 · Answer 7 · edited Feb 06 '20 at 00:18

I created a gem to make URI encoding stuff cleaner to use in your code. It takes care of binary encoding for you.

Run gem install uri-handler, then use:

require 'uri-handler'

str = "\x12\x34\x56\x78\x9a\xbc\xde\xf1\x23\x45\x67\x89\xab\xcd\xef\x12\x34\x56\x78\x9a".to_uri
# => "%124Vx%9A%BC%DE%F1%23Eg%89%AB%CD%EF%124Vx%9A"

It adds the URI conversion functionality into the String class. You can also pass it an argument with the optional encoding string you would like to use. By default it sets to encoding 'binary' if the straight UTF-8 encoding fails.

score 2 · Answer 8 · edited Feb 06 '20 at 00:22

2

If you want to "encode" a full URL without having to think about manually splitting it into its different parts, I found the following worked in the same way that I used to use URI.encode:

URI.parse(my_url).to_s

edited Feb 06 '20 at 00:22

the Tin Man

158,662
42
215
303

answered Mar 01 '19 at 04:42

Glenn 'devalias' Grant

1,928
1
21
33

How to URL encode a string in Ruby

8 Answers8

Linked

Related