0

I want to send a json string with base64 encoded file in it from a client, basically it looks like this:

{  
   "data":"aGVscA==",
   "filename":"file.txt"
}

And I wrote this struct:

type StoredFile struct {
    Data     []byte `json:"data"`
    Filename string `json:"filename"`
}

Then I decode the json into the struct:

decoder := json.NewDecoder(request.Body)
storedFile := StoredFile{}
err := decoder.Decode(&storedFile)

And save it with gorm:

db.Create(&storedFile)

My question is:

  1. How do json package decode the base64 string to byte array? Does it treat it like plain text, because I know that the data size will be increased by 33% when it is encoded to base64 and if it is treated like text the 33% increase will still be there right?
  2. What kind of encoding does gorm use when it stores []byte to PostgreSQL database? Is it UTF-8? If not, how do I set the encoding to UTF-8?
Jonathan Hall
  • 75,165
  • 16
  • 143
  • 189
William Wino
  • 3,599
  • 7
  • 38
  • 61

1 Answers1

1
  1. Yes, it considers your b64 string like plain text, because there's no way it can know that it's a base 64 string. So yes again, your size increase will still be there.

  2. The encoding depends on the table. If gorm is simply writing into an already existing table, it will use the encoding that is specified in the structure. For example:

CREATE TABLE `users` (
  `id` int(10) unsigned NOT NULL AUTO_INCREMENT,
  `email` varchar(255) NOT NULL,
  `password` varchar(255) NOT NULL,
  `token_user_id` varchar(255) NOT NULL,
  `is_admin` tinyint(4) NOT NULL DEFAULT '1',
  PRIMARY KEY (`id`),
  UNIQUE KEY (email)

) ENGINE=InnoDB DEFAULT CHARSET=utf8mb4 COLLATE=utf8mb4_0900_ai_ci;

For this table, the CHARSET is utf8mb4 so it would store using an UTF-8 encoding.

If gorm is also responsible for creating the table however, I believe it uses utf8 by default as the encoding, but I can't seem to find any source to back it up.

Ullaakut
  • 3,554
  • 2
  • 19
  • 34
  • If I store non-base64encoded bytes in the database, how does the json library convert it to json? Is it using a base64 encoding? – William Wino Sep 12 '18 at 05:58
  • 1
    It stores a `StoredFile` structure, right? So when you put it in your database, it's just inserting an entry to your `StoredFiles` table that contains two strings. Since your database doesn't know the difference between a slice of bytes and a string, it will just be inserted as two strings. There is no conversion or encoding made into base 64, no. – Ullaakut Sep 12 '18 at 06:03
  • Alright, thanks for the answer. What is your suggestion on this? Should I just override the MarshalJSON and UnmarshalJSON? Or there is a tag for this? – William Wino Sep 12 '18 at 07:21
  • there is someone who says that it is encoded with base64 encoding: https://stackoverflow.com/questions/36465065/marshaling-json-byte-as-strings-in-go – William Wino Sep 12 '18 at 07:59
  • If you wanted your data to be base64 encoded when in JSON format, and binary encoded when it's in the database, the best way is to create your own type (based on a `[]byte`), which implements the `driver.Valuer`, `sql.Scanner`, `json.Marshaler` and `json.Unmarshaler` interfaces. – Ezequiel Muns Sep 26 '18 at 10:12