curl command output has wrong encoding

Question

when i execute

curl "http://weather.yahooapis.com/forecastrss?w=1225955&u=c"

it returns me response with incorrect encoding:

khan@khan-P55A-UD3P:~$ curl "http://weather.yahooapis.com/forecastrss?w=1225955&u=c" 
���ǆ��ud@3��v(
����$j$��~����4(���Xy����wH�o�9<q��,�s\��e"�tA�\݌h�ʄ���
                                                             �����h��M���{���J=�m93W
                                                                                      �S�)�e�[sv,�҉eAKM�z{ǔ��g��:���*�����(n�m��&�Jꟈ��Mg�,yn?F�&��_��
ik6                                                                                                                                                      >��0�e&X��簺
sQ~�:�Z;*9�.a"ߕ|��EO[�5"�׫[�k�����1ӆ�n?}r1�u�d��Cڐ��X��`�NF�g!�c��W��G��1�o����Z��53<z`���.��w� s׃��ߖ+�vh��3yt�b}�9
�6�s3K
�W�  �0�هF@���>�X֥Qh�ʰv�BQ�R
ʮ�<�4;�ڊ2�8y� �g���6M(��]�|'�U@�ș�B
�8du!�&'�NOB��ț��3�K��fW��
                           \Rheg�=��F�R;�u�F�s9���&����,��|r��o�E۲�T��V$&�����uf\������v��Z~&�Au��{��ى"m�ʨ���U����2�8�#0F@'������
                                                                                                                                           l���R�XL��~A��̱���p��9��8�iH��nC�i4��^t;����۪���d�V�����7��=S&��2�u�#v~�L`�k���v�0
                            �[���"<���~�z��j,���X=�zmKD/|���(�p��M���⥁}_�!��GџC��2|�G��<ফe��nb"x ?�e�s��;���r;ﲃ�]�9"l��;�}�w�ٮjwR[�C����#O�
                                                                                                                                                      �������#a����s�km���$a�����\)�$�o��Ә�K��FR�*�ý�l�Z
            �
             &�`_�D�WӠ�>`T��0��| c��⿎K%��n:���~(�����.{��}< /~�^!A��$\���c�<�Á
"��k�_��t����t�n�5�^t�وF��l<V�����oo?
                                        `O���3p��ĝ�S�X�G�x��Ź+�
khan@khan-P55A-UD3P:~$

However, the same command works just fine in another computer. is there anything i need to be setting in shell in order to get this in correct format ?

i m using ubuntu 14.04 64bits.(Linux khan-P55A-UD3P 3.13.0-40-generic #69-Ubuntu SMP Thu Nov 13 17:53:56 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux)

any ideas? a screenshot of the command can be seen here as well: https://i.stack.imgur.com/MlV2Q.png

Marcin asked you to save the output to a file; Does "cat output.xml |gunzip" produce the correct output? Try "hexdump -C output.xml" and post the first line or two. Try "curl -v url; echo $?" and post the headers and return value too. (My guess would be that either the website or a caching proxy is sending it compressed but not sending the correct headers indicating that for some reason or curl is not decompressing it despite getting the headers for some reason (Not compiled with libz support?)) — Aleksi Torhamo, Nov 29 '14 at 10:52

mh-samiei · Answer 1 · 2020-01-04T06:09:21.083

curl will automatically decompress the response if you set the --compressed flag:

curl --compressed "http://example.com"

--compressed (HTTP) Request a compressed response using one of the algorithms libcurl supports, and save the uncompressed document. If this option is used and the server sends an unsupported encoding, curl will report an error.

refrence of answer

if you need use this option only for gzip content encoding, use this command

curl -sH 'Accept-encoding: gzip' http://example.com/ | gunzip -

score 3 · Answer 2 · answered Nov 29 '14 at 08:50

3

I think is connected with default encoding of your terminal (which default is UTF-8). You can try to pass the stream to the file, for instance:

curl "http://weather.yahooapis.com/forecastrss?w=1225955&u=c" > response

I had the same problem with rest webservice, when I was passing bytes (Pdf content format inside Data Handler). Without passing stream I was receiving data encoded in UTF-8 in terminal and also when I was using soapUi.

answered Nov 29 '14 at 08:50

Marcin Erbel

1,597
6
32
51

unfortunately it returns the same result ! khan@khan-P55A-UD3P:~$ curl "http://weather.yahooapis.com/forecastrss?w=1225955&u=c" > output.xml khan@khan-P55A-UD3P:~$ cat output.xml �Vێ�6}b¾��Z�,��m /�-�-��THj�}��C�b�23g.g�}+ x�%�$�|\&*2[�?�o�3�2��BI�${��G/^Dژ��i�-i�g6�zIrk��x�ugޞ�Jy�*�4-�࠘q5��v;ozJgt��!��t��tS)�q>��I� – Shrouk Khan Nov 29 '14 at 08:56
What if you open this file using for instance VIM? – Marcin Erbel Nov 29 '14 at 12:06

score 1 · Answer 3 · edited May 23 '17 at 12:02

1

Try setting the charset of the terminal to utf-8. A google got me this: https://unix.stackexchange.com/questions/28136/set-gnome-terminals-default-charset-to-utf8

Before you set the encoding check to make sure that indeed is the issue by determining the current charset as in:

How to get terminal's Character Encoding

edited May 23 '17 at 12:02

Community

1
1

answered Nov 29 '14 at 09:26

Khanna111

3,627
1
23
25

score 1 · Answer 4 · answered May 04 '20 at 07:34

1

There is one more solution to add Accept-Charset header:

 curl -vvv -H "Accept-Charset: utf-8" http://example.com/ > tmpFile

answered May 04 '20 at 07:34

Roman Kazanovskyi

3,370
1
21
22

score -1 · Answer 5 · edited Nov 05 '22 at 11:46

-1

Sometimes is as easy as removing Accept-encoding: gzip from your request.

edited Nov 05 '22 at 11:46

Tyler2P

2,324
26
22
31

answered Nov 01 '22 at 21:42

user1630663

1
1

But there is no such thing in the OP's request. And that "Accept-Encoding" field has already been covered in previous answers. Be sure when answering old question, with already upvoted answers, to bring new information. – chrslg Nov 05 '22 at 10:25

curl command output has wrong encoding

5 Answers5