Scala API with encode special character not working for client

Question

We have this function which takes care of special character. For example this function will convert He shouldn’t be allowed to He shouldn�t be allowed.

import io.circe._, io.circe.generic.auto._, 
io.circe.syntax._, io.circe.parser._, io.circe.optics.JsonPath._

private def bytesToJsonCirce(value: Array[Byte]): Json = {
    parse(encodeCharacters(value.map(_.toChar).mkString)) match {
      case Right(x: Json) => x
      case Left(err: ParsingFailure) =>
        logger.error(err.getLocalizedMessage)
        Json.Null
    }
  }  

private def encodeCharacters(x: String): String = {
        val encodeChar = '�'
        (x.toCharArray map {
          case y if y.isControl => encodeChar
          case y => y
        }).mkString
      }

Now we are trying to consume this API response at client side(python) getting error UnicodeEncodeError: 'ascii' codec can't encode character u'\ufffd' in position 11: ordinal not in range(128) . If we apply .encode("utf8") at client its giving back He shouldn�t be allowed but we are loosing the original form He shouldn’t be allowed.

Also this UnicodeEncodeError is not desired without .encode("utf8"). How it can be achieved at scala front .

I am a bit confused, what side is in Scala - server or client? — michaJlS, Sep 01 '21 at 19:38
Does you API produce JSON responses? Where does the `value: Array[Byte]` come from, which I understand is a JSON? — michaJlS, Sep 01 '21 at 19:45
If `value: Array[Byte]` is a UTF-8 encoded string then treating each element of the array as separate character is wrong because some characters can take more than a byte. — michaJlS, Sep 01 '21 at 19:54
`value: Array[Byte]` will come like this `Array(110, 117, 108, 108)` byte representation of each character of a string . We are mapping each character to `string` and checking against each character of that string `isControl` or not . — curiousguy, Sep 02 '21 at 04:24
`’` is not a control character so the example you give will be unchanged — Tim, Sep 02 '21 at 06:19

score 1 · Answer 1 · answered Sep 02 '21 at 06:18

1

The error message says that \ufffd is not a valid ASCII character:

'ascii' codec can't encode character u'\ufffd'

\ufffd is the Unicode value of � so the problem is that the replacement value encodeChar is not a valid ASCII value. Try changing encodeChar to a valid ASCII character.

answered Sep 02 '21 at 06:18

Tim

26,753
2
16
29

I was more looking for instead of replacing by same encoder `�` , can I have some escape mechanism that can help to retain the actual value. – curiousguy Sep 02 '21 at 06:50

Scala API with encode special character not working for client

1 Answers1