How can I turn off escaping while converting a byte array to string?

Question

This porblem already surfaced many times:

Examples:

UTF-8 Encoding a string with double Quotes in C#

Invalid JSON after calling Encoding.UTF8.GetString()

I am trying to get the JSON representation of bytes and calling the following seems to be the right solution:

Encoding.UTF8.GetString(bytes)

This results in the following:

bytes = []
string = "[]"

bytes = {"a": 1}
string = "{\"a\": 1}"

Is there a way to stop this behaviour without using a JSON library?

The complete workflow:

Data on S3:

> awsc s3 cp s3://depoxy-dev/table-size-stats/current.json - | jq
[
  {
    "DatabaseName": "default",
    "Tables": [],
    "DatabaseSize": 0
  },
  {
    "DatabaseName": "dwh",
    "Tables": [],
    "DatabaseSize": 0
  },
  {
    "DatabaseName": "site_traffic",
    "Tables": [
      {
        "TableName": "dev_l1x_be",
        "FileCount": 47,
        "TableSize": 41019220
      }
    ],
    "DatabaseSize": 41019220
  },
  {
    "DatabaseName": "test_database",
    "Tables": [],
    "DatabaseSize": 0
  }
]

I read it from S3 as bytes.

let readAllBytes (stream: Stream) : byte array =
    use s = stream
    use ms = new MemoryStream()
    s.CopyTo(ms)
    ms.ToArray()

This is what I try to convert to string.

Encoding.UTF8.GetString(bytes)

Results (without JQ):

"[{\u0022DatabaseName\u0022: \u0022default\u0022, \u0022Tables\u0022: [], \u0022DatabaseSize\u0022: 0}, {\u0022DatabaseName\u0022: \u0022dwh\u0022, \u0022Tables\u0022: [], \u0022DatabaseSize\u0022: 0}, {\u0022DatabaseName\u0022: \u0022site_traffic\u0022, \u0022Tables\u0022: [{\u0022TableName\u0022: \u0022dev_l1x_be\u0022, \u0022FileCount\u0022: 47, \u0022TableSize\u0022: 41019220}], \u0022DatabaseSize\u0022: 41019220}, {\u0022DatabaseName\u0022: \u0022test_database\u0022, \u0022Tables\u0022: [], \u0022DatabaseSize\u0022: 0}]"

With JQ:

"[{\"DatabaseName\": \"default\", \"Tables\": [], \"DatabaseSize\": 0}, {\"DatabaseName\": \"dwh\", \"Tables\": [], \"DatabaseSize\": 0}, {\"DatabaseName\": \"site_traffic\", \"Tables\": [{\"TableName\": \"dev_l1x_be\", \"FileCount\": 47, \"TableSize\": 41019220}], \"DatabaseSize\": 41019220}, {\"DatabaseName\": \"test_database\", \"Tables\": [], \"DatabaseSize\": 0}]"

The solution was to return the http answer as raw string and not as json.

I think you're looking at the contents of the string using the debugger, which is adding the escapes for you. — canton7, Dec 09 '21 at 13:13
What does a "json representation of bytes" mean? You have a byte array, you're running it through a serializer, and looking at the generated output? — Caius Jard, Dec 09 '21 at 13:13
I have a JSON file on disk. I read it with a reader that reads it as bytes. I want to convert the bytes to string and return it to the client. — Istvan, Dec 09 '21 at 13:15
Show us a screenshot with your complaint highlighted; in windows click start, type SNIP, open snipping tool, capture the region (if you need to capture a tooltip set a delay of 5 seconds, hit new, then cause the tooltip and wait for the snapshot), then circle the problem with a highlight, press ctrl-c, come to StackOverflow, edit your question and press ctrl-v — Caius Jard, Dec 09 '21 at 13:15
Having read your updated comment, I think canton7 might be correct, and you're eg looking at this string in the tooltip of the debugger, which does add `\"` (it shows you the c# that would generate the string, not the actual string. If you want the atual string, point to the variable, wait for the tooltip and then click the magnifying glass next to the value - it opens the Text Visualizer which shows the string verbatim — Caius Jard, Dec 09 '21 at 13:17
That, or view the string using e.g. `theString,nq` in the watch window (with the `,nq` suffix) to stop it adding quotes and escapes. [See here](https://learn.microsoft.com/en-us/visualstudio/debugger/format-specifiers-in-csharp?view=vs-2022) — canton7, Dec 09 '21 at 13:20
How are you viewing the contents of the string? Include a screenshot of what you're seeing (one of the few times where an image would actually be helpful!) — canton7, Dec 09 '21 at 13:21
I use the command curl that calls into the API that returns the string. — Istvan, Dec 09 '21 at 13:21
You said "This results in the following: ... `string = "{\"a\": 1}"`". Include a screenshot of where you're seeing `string = "{\"a\": 1}"`. Whatever you're seeing that makes you think the ``\``'s have been added, show us — canton7, Dec 09 '21 at 13:22
I don't follow how you've got from the output of `curl` / `jq` to where `readAllBytes` is called — canton7, Dec 09 '21 at 13:35
JSON on s3 -> readAsBytes() -> convertToString() -> returnToCaller() — Istvan, Dec 09 '21 at 13:38
The two characters `->` don't really help me. Could you please post a full [mcve] which we can run? — canton7, Dec 09 '21 at 13:41

score 1 · Answer 1 · answered Dec 09 '21 at 14:11

1

The problem was that I kept the reponse as JSON instead of a string that was causing the string to be double encoded.

answered Dec 09 '21 at 14:11

Istvan

7,500
9
59
109

score 0 · Answer 2 · answered Dec 09 '21 at 13:35

0

If you use ASP.Net Core and return a string from a controller, that string is re-encoded as JSON. You can return e.g. Content(someString, “application/json”). Other frameworks provide similar capabilities.

answered Dec 09 '21 at 13:35

CaringDev

8,391
1
24
43

I do that, in fact everything we return is "application/json". – Istvan Dec 09 '21 at 13:37
Do you set that one the response or is it (auto) negotiated by the client? If you (on the server) return a string and the client asks for JSON, the escaping (double encode) happens. So you need to tell your framework that the string is JSON already. Also, if you look at it in e.g. the console, the output of the tool you use to fetch the resource might be escaped again, that’s why others ask for screenshots. – CaringDev Dec 09 '21 at 14:11

How can I turn off escaping while converting a byte array to string?

2 Answers2