0

I'm trying to send a dict with labels to the check_mk web-api - however the web-api uses python2 - and the devs have insisted on validating that the labels are Unicode.

My example dict in p3:

"": [
    {
        "condition": {
            "service_description": [
                {
                    "$regex": "Callcentre Queue: OTU"
                }
            ]
        },
        "value": {
            u"max_check_attempts": u"3"
        },
        "options": {
            "description": "Label - max_check_attempts\nService - Callcentre Queue: OTU"
        }
    }]

but when sent, I get the label max_check_attempts must be unicode type error.

Traceback (most recent call last):
  File "./cmk_import.py", line 415, in <module>
    process_service_rulelist()
  File "./cmk_import.py", line 309, in process_service_rulelist
    api().set_ruleset('service_label_rules', total_lbl_ruleset)
  File "/usr/local/lib/python3.6/dist-packages/check_mk_web_api/__init__.py", line 724, in set_ruleset
    return self.make_request('set_ruleset', data=data, query_params={'request_format': 'python'})
  File "/usr/local/lib/python3.6/dist-packages/check_mk_web_api/__init__.py", line 164, in make_request
    raise CheckMkWebApiException(result)
check_mk_web_api.exception.CheckMkWebApiException: Check_MK exception: ERROR: The label ID 'max_check_attempts' is of type <type 'str'>, but should be unicode. Affected Rule {'value': {'max_check_attempts': '3'}, 'condition': {'service_description': [{'$regex': 'Callcentre Queue: OTU'}]}, 'options': {'description': 'Label - max_check_attempts\nService - Callcentre Queue: OTU'}}

Error is from checkMK itself issued by the check_mk web-api. As the listening agent is Python2 - it's trying to validate type() as unicode

I know that P3 has incorporated unicode into str - but wondering if there is a way to override and keep the dict unicode chars without rewriting the affected parts in python2.

Web request is handled by this lib - https://github.com/brennerm/check-mk-web-api

request = urllib.request(self._build_request_path(query_params),WebApi._build_request_data(data, request_format))

The contents of which are: Arg1:

http://localhost/preprod/check_mk/webapi.py?effective_attributes=0&action=get_all_hosts&_username=automation&_secret=foobar

And Snippet:

{%27condition%27: {%27service_description%27: [
{%27%24regex%27: %27VPN: Premier Ping - 1.1.1.1%27}

]}, %27value%27:
{%27check_interval%27: %275%27, %27max_check_attempts%27: %273%27, %27retry_interval%27: %271%27, %27check_period%27: %2724x7%27, %27notification_period%27: %2724x7%27, %27notification_interval%27: %27120%27, %27notifications_enabled%27: %271%27, %27active_checks_enabled%27: %271%27, %27check_freshness%27: %270%27, %27event_handler_enabled%27: %271%27, %27flap_detection_enabled%27: %271%27, %27is_volatile%27: %270%27, %27obsess_over_service%27: %271%27, %27passive_checks_enabled%27: %271%27, %27process_perf_data%27: %271%27, %27retain_nonstatus_information%27: %271%27, %27retain_status_information%27: %271%27}
, %27options%27: {%27description%27: %27Labels for Service - VPN: Premier Ping - 1.1.1.1%27}}
itChi
  • 642
  • 6
  • 19
  • 2
    Python 3 strings are Unicode already. You don't need to do anything special. Even `a` is a Unicode character. The problem is how you make the request to that Web API. Did you use the wrong `content-type` in the request perhaps? – Panagiotis Kanavos Jan 26 '21 at 10:09
  • 1
    `P3 has incorporated unicode into str` no, that's not what happened at all. Python 3 strings **are** Unicode, period. There's nothing to validate or verify. Unicode isn't some kind of escape sequence, it's how **all** characters are converted into bytes. If you have encoding issues with an HTTP request, post your HTTP code – Panagiotis Kanavos Jan 26 '21 at 10:14
  • Apologies I've edited - the 'u' char before "max_check_attempts" & it's value is what is expected by the web-api. I've done the same with curl - web-api accept the python2 formatted 'u' prefix, but not if it's not present. – itChi Jan 26 '21 at 10:15
  • @PanagiotisKanavos - it appears to be related to the specific string I send to the web-api. Do you think it may be the actual web request that needs to render it out of python3 into python2 compatible format? – itChi Jan 26 '21 at 10:18
  • No it's not. Because again, and again, Python 3 strings **are Unicode already**. In Python 3 the `u` prefix has no effect. Even in Python 2, it only affects how the parser/compiler treats the string constant. The resulting in-memory bytes don't contain any prefixes. The problem is the HTTP call - all HTTP calls specify their content type *and encoding* in the `content-type` header. If you have problems, it's because the `Content-type` encoding and the actual body encoding don't match. Post your HTTP code – Panagiotis Kanavos Jan 26 '21 at 10:18
  • **This** page is Unicode (UTF8 specifically), which is why I can type Αυτό Εδώ knowing it will appear without problems. I didn't use any escape sequences, nor did I contact SO to have them change the encoding of the comment. When I click `Add Comment` a POST request is made with `UTF-8` in the `Content-Type`. That's how HTTP works, no matter the language – Panagiotis Kanavos Jan 26 '21 at 10:21
  • Forgive my misunderstanding - but the checkmk config files appear to contain unicode prefix only for certain dict items as follows. Would that break if the `content-type` would blanket set encoding for the whole message?`only_hosts = [ {'condition': {'host_tags': {'criticality': {'$ne': 'offline'}}}, 'value': True, 'options': {'description': u'Do not monitor hosts with the tag "offline"'}}, ] + only_hosts` – itChi Jan 26 '21 at 10:23
  • Post your HTTP code, in the question itself. That `u` prefix is redundant. This has [already been asked before](https://stackoverflow.com/questions/2464959/whats-the-u-prefix-in-a-python-string). In fact Python 3-3.2 removed the prefix altogether, and only added it back in 3.3 to make migration of legacy code easier. It does *nothing* precisely because strings are *already Unicode* – Panagiotis Kanavos Jan 26 '21 at 10:27
  • @PanagiotisKanavos I would appreciate if you could take another look, I've provided the request and the contents, I think you're right in that it's not sending it in the required python2 format; which must contain the unicode indicator in the url request. – itChi Jan 29 '21 at 11:56
  • That's no `unicode indicator`, that's a URL-encoded string. How to handle UTF8 in URLs has **nothing at all** to do with Python. [This has been answered already](https://stackoverflow.com/questions/912811/what-is-the-proper-way-to-url-encode-unicode-characters) – Panagiotis Kanavos Jan 29 '21 at 12:02
  • @PanagiotisKanavos Apologies if I'm using the wrong terminology. So the checkmk API expects the format `{'label': {u'foo':u'bar'}}`. The URL-Encoded string I'm sending is `{%27label%27: {%27foo%27: %27bar%27}` Would encoding it in utf-8 get me: `{%27label%27: {u%27foo%27: u%27bar%27}`? – itChi Jan 29 '21 at 12:31
  • The `u` prefix is purely a Python mechanism to denote the difference between a `str` and a `unicode` in Python 2; in Python 3 that would be the difference between a `bytes` and a `str`, but in Python 3 that entire `u` prefix thing is obsolete, it was just a transition mechanism between Python 2 and 3. If you have *any* string in Python 3, it *is* "Unicode", and you won't ever see the `u` prefix in Python 3 code. This is something other than UTF-8 yet again… – deceze Jan 29 '21 at 12:35
  • Reading between the lines, it sounds like that API expects a **Python 2 dict literal**, in which strings have a `u` prefix…?! That's not JSON and also not anything to do with UTF-8, it merely expects a very language-specific serialisation format produced by mistreating Python 2 dicts, which is now obsolete?! If that is indeed so, you'll have a hard time producing that from Python 3. – deceze Jan 29 '21 at 12:37
  • The other possibility is that there's some internal validation in that library before the request is even sent, and it was coded for Python 2 and it doesn't work right when run on Python 3, since it doesn't understand the difference between `bytes`/`str`/`unicode` between these two language versions correctly. In that case, you'd need to rewrite that library, or run it on Python 2. – deceze Jan 29 '21 at 12:38
  • @deceze this is exactly it, I'm trying to send python3 to the API that runs on python2; and for some reason in a recent change, they started checking type() on the label contents. Which is where I wanted to see if there was even some compatibility layer that I could use to represent the python3 dict as a python2 dict. – itChi Jan 29 '21 at 12:40
  • @deceze the library is written in Python3, there is no validation happening, and the response is purely a check_mk generated web response after the API read the data. – itChi Jan 29 '21 at 12:42
  • So the remote API *expects* to receive standard JSON, and then validates the decoded JSON internally, and finds that something isn't "Unicode" there, and then returns an error to you? Well, if you can only send JSON, then there's no way you can send anything "in Unicode". JSON is JSON, it has strings, and that's that. There's no differentiator to mark up JSON strings as "Unicode". If you put a `u` in front of a JSON string, it ceases to be JSON and should throw validation errors because it's not JSON. – deceze Jan 29 '21 at 12:45
  • @deceze actually - the remote API needs certain formats for certain requests - in this case - it requires a python2 format, probably so that it can do the `type()` validation. – itChi Jan 29 '21 at 13:24
  • 1
    So the API is expecting you to send the string `{u"foo": u"bar"}` to it over HTTP? That means it's probably relying on just doing `str({'foo': 'bar'})` and sending that? And that only works on Python 2, because in Python 3 you won't get the `u` prefixes from that? — That's a terrible API then, expecting a *language specific*, and even more so *specific to an obsolete version of a language-specific*, input format… – deceze Jan 29 '21 at 13:41
  • @deceze Yes exactly. short of `str(dict),re()` I don't think there is a way to achieve this. Sorry for not making my question clearer to start of with - I've raised a question with their support team. They do have a Python3 version of the program in beta which works with the same request, but other previously working parts of the API are now broken... Thanks for all your help. – itChi Jan 29 '21 at 13:56

1 Answers1

0

The API client sends the data in what it terms "python" format which is basically a dict that has had str() called on it. The intention is that the server will call ast.literal_eval() on this dictionary string. The client does the same with the body of the response. converting it back to a Python dict.

Throughout the API there are places where it does support JSON for request/response processing, and this can be set in the query params for some functions. Unfortunately, for the particular request that is failing here (set_ruleset) JSON is not an option.

It's not obvious why the server is failing to process this request. I have tried unquoting the body of the message and running that through literal_eval() in Python 2, and it seems to work.

One thing that I did notice is the embedded new line character in the request dict "Label - max_check_attempts\nService - Callcentre Queue: OTU". It's probably what the server is complaining about when it mentions "label ID 'max_check_attempts'".

Further problem tracing would require looking at the server logs and possibly the code to see what's going on there.

mhawke
  • 84,695
  • 9
  • 117
  • 138
  • It seems to me that the server is possibly `literal_eval`ing it *in Python 2*, and there you obviously get different data types back whether the input includes a `u` or not, and it's then validating *that* eval'd data type. – deceze Jan 29 '21 at 14:02
  • @deceze Maybe, but I tried `literal_eval` in Python 2 on the body of the request that I think it sends, and it passed. Maybe different version of Python 2, or my guess as to what the server does is wrong. THe OP hasn't said, but I suppose that other requests are succeeding, and as I mentioned in my answer some can/do use JSON and others don't. There's still a chance that the embedded new line character might be what the server dislikes, I can imagine it validating against that. – mhawke Jan 29 '21 at 14:15
  • @mhawke thanks - I've removed the embedded newline char, but the request is still failing. `CheckMkWebApiException: Check_MK exception: ERROR: The label ID 'check_period' is of type , but should be unicode. Affected Rule {'value': {'check_period': '24x7', 'notification_interval': '120', 'notification_period': '24x7', 'retry_interval': '1', 'check_interval': '5', 'max_check_attempts': '2'}, 'condition': {'service_description': [{'$regex': 'Router: Uptime'}]}, 'options': {'description': 'Labels for Service - '}}` – itChi Jan 29 '21 at 14:20
  • Interestingly, I did test `api().get_ruleset('service_label_rules')` and sent it back exactly the same, which failed. When printing the request, returned contained the `u`, however after it was processed by python3, the `u` was stripped.check_mk_web_api.exception. – itChi Jan 29 '21 at 14:22
  • How can it contain the `u` prefix if you are running Python 3.6? Did you mean "when printing the response"? – mhawke Jan 29 '21 at 14:28
  • Yes, when printing the response. i.e. `print(api().get_ruleset('service_label_rules'))` – itChi Jan 29 '21 at 14:34
  • It's also possible that the server is assuming that the request body is JSON. The API sets the `Content-Type` header to `application/x-www-form-urlencoded` for all messages. So how is the server supposed to know which format message it received? – mhawke Jan 29 '21 at 14:50
  • As a possible work-around, you could try subclassing the `WebApi` class and overriding the `set_ruleset()` method to remove the hardcoded `query_params={'request_format': 'python'}`. Then it will JSON encode the body. – mhawke Jan 29 '21 at 14:56
  • I've tried sending json format as a curl to the same api method: the reply was that this data format is not allowed. There is another request type where 'value' dict must be passed in as a list of tuples; which is probably where the requirement to have json & python input formats came from in the first place – itChi Jan 29 '21 at 15:45