Strange \n in base64 encoded string in Ruby

Question

The inbuilt Base64 library in Ruby is adding some '\n's. I'm unable to find out the reason. For this special example:

irb(main):001:0> require 'rubygems'
=> true
irb(main):002:0> require 'base64'
=> true
irb(main):003:0> str =  "1110--ad6ca0b06e1fbeb7e6518a0418a73a6e04a67054"
=> "1110--ad6ca0b06e1fbeb7e6518a0418a73a6e04a67054"
irb(main):004:0> Base64.encode64(str)
=> "MTExMC0tYWQ2Y2EwYjA2ZTFmYmViN2U2NTE4YTA0MThhNzNhNmUwNGE2NzA1\nNA==\n"

The \n's are at the last and 6th position from end. The decoder (Base64.decode64) returns back the old string perfectly. Strange thing is, these \n's don't add any value to the encoded string. When I remove the newlines from the output string, the decoder decodes it again perfectly.

irb(main):005:0> Base64.decode64(Base64.encode64(str).gsub("\n", '')) == str
=> true

More of this, I used an another JS library to produce the base64 encoded output of the same input string, the output comes without the \n's.

Is this a bug or anything else? Has anybody faced this issue before?

FYI,

$ ruby -v
ruby 1.8.7 (2008-08-11 patchlevel 72) [i486-linux]

score 279 · Accepted Answer · edited Feb 09 '21 at 07:44

279

Edit: Since I wrote this answer Base64.strict_encode64() was added, which does not add newlines.

The docs are somewhat confusing, the b64encode method is supposed to add a newline for every 60th character, and the example for the encode64 method is actually using the b64encode method.

It seems the pack("m") method for the Array class used by encode64 also adds the newlines. I would consider it a design bug that this is not optional.

You could either remove the newlines yourself, or if you're using rails, there's ActiveSupport::CoreExtensions::Base64::Encoding with the encode64s method.

edited Feb 09 '21 at 07:44

lulalala

17,572
15
110
169

answered Apr 12 '10 at 09:38

Christoffer Hammarström

27,242
4
49
58

48

Seems that since i wrote this answer there is now `strict_encode64()` which ostensibly does not add newlines. – Christoffer Hammarström Jan 11 '11 at 14:03
22

So dumb! Why you do this to me, Ruby? – Josh M. Feb 03 '14 at 22:21
2

This is an assinie default, but as @ChristofferHammarström mentions, there's a way to get a true b64 encoding now. – Dan Mar 21 '15 at 17:36
1

Apparently this exists to maintain backwards compat with software that cannot handle long lines. https://stackoverflow.com/a/20065991/5749914 – Elliott Beach Aug 21 '17 at 23:57

score 121 · Answer 2 · answered Jan 13 '11 at 12:32

121

In ruby-1.9.2 you have Base64.strict_encode64 which doesn't add that \n (newline) at the end.

answered Jan 13 '11 at 12:32

ghtn

2,233
3
15
7

That's a much easier solution! – Avishai Sep 19 '13 at 00:21
I have checked Base64.strict_encode64 is not properly encoded three character string. like Base64.strict_encode64('abc') -> YWJj . the above example is not encoded correctly. – VijayGupta Sep 17 '19 at 11:01
@CodeMaker I thought `YWJj` is correct. What's wrong with that? – lulalala Feb 09 '21 at 08:09

score 13 · Answer 3 · answered Feb 01 '18 at 13:19

13

Use strict_encode64 method. encode64 adds \n every 60 symbols

answered Feb 01 '18 at 13:19

Александр Тихонович

199
1
3

can we change it to add after 76 symbols ? – sonic Jul 25 '18 at 13:07

score 9 · Answer 4 · answered Apr 12 '10 at 09:43

9

Yeah, this is quite normal. The doc gives an example demonstrating the line-splitting. base64 does the same thing in other languages too (eg. Python).

The reason content-free newlines are added at the encode stage is because base64 was originally devised as an encoding mechanism for sending binary content in e-mail, where the line length is limited. Feel free to replace them away if you don't need them.

answered Apr 12 '10 at 09:43

bobince

528,062
107
651
834

1

This just happened in my Android App (Java's Base64 library). I was totally confused with this strange occuring. Took me literally 1 hour to figure out what's wrong and then searched for the error. This comment is helping understand legacy issues even after 6 years. – burglarhobbit Oct 14 '16 at 14:42

score 6 · Answer 5 · answered Feb 22 '13 at 15:28

6

Seems they've got to be stripped/ignored, like:

Base64.encode64(str).gsub(/\n/, '')

answered Feb 22 '13 at 15:28

meesern

2,518
27
29

This solution is dirty.. anythein else? – Arnold Roa Jul 10 '14 at 15:51
1

@yaauie ([via suggested edit](http://stackoverflow.com/review/suggested-edits/6210102)): Edits that make a substantial change should generally not be done. I'd recommend posting that as a separate answer. – Pokechu22 Nov 12 '14 at 01:07

user1519240 · Answer 6 · 2016-12-12T18:59:04.780

2

The \n added when using Base64#encode64 is correct, check this post out: https://glaucocustodio.github.io/2014/09/27/a-reminder-about-base64encode64-in-ruby/

edited Dec 12 '16 at 18:59

answered Oct 29 '15 at 12:10

user1519240

2,186
1
22
20

Strange \n in base64 encoded string in Ruby

6 Answers6

Linked