If you only write ASCII characters to a file, it will be possible to open it in Notepad or any other text editor just fine:
local out = io.open("file.bin", "wb")
local str = string.char(72,101,108,108,111,10) -- "Hello\n"
out:write(str)
out:close()
The resulting file will contain:
Hello
On the other hand if you write real binary data (for instance random bytes) you will get garbage:
local out = io.open("file.bin", "wb")
local t = {}
for i=1,1000 do t[i] = math.random(0,255) end
local str = string.char(unpack(t))
out:write(str)
out:close()
This is similar to those video game save files you have seen.
If you still don't get it, try to write all possible octets to a file:
local out = io.open("file.bin", "wb")
local t = {}
for i=0,255 do t[i+1] = i end
local str = string.char(unpack(t))
out:write(str)
out:close()
and then open it with a hexadecimal editor (here I used Hex Fiend on Mac OS) to see the correspondences:

Here, on the left, you have the bytes in hexadecimal and on the right you have their text representation. I have selected uppercase H which, as you can see on the left, corresponds to 0x48. 0x48 is 4*16 + 8 = 72 in base 10 (look at the bottom bar of the screenshot which tells you that).
Now look at my first code example and guess what the code for lowercase e is...
And finally look at the 4 last rows of the screenshot (bytes 128 to 255). This is the garbage you were seeing.