write log entry to splunk via HTTP in python

Question

We have a python program that needs to send logs to splunk. Our splunk admins have created a service collector HTTP endpoint to publish logs to with the following:

index
token
hostname
URI

We can't find where to input the URI in the splunk python SDK client. For example:

import splunklib.client as client
import splunklib.results as results_util

HOST="splunkcollector.hostname.com"
URI="services/collector/raw"
TOKEN="ABCDEFG-8A55-4ABB-HIJK-1A7E6637LMNO"
PORT=443

# Create a Service instance and log in
service = client.connect(
    host=HOST,
    port=PORT,
    token=TOKEN)

# Retrieve the index for the data
myindex = service.indexes["cloud_custodian"]

# Submit an event over HTTP
myindex.submit("Dummy test python client log")

As you can see I never use the URI variable. The above code results in:

Traceback (most recent call last):
  File "splunk_log.py", line 15, in <module>
    myindex = service.indexes["cloud_custodian"]
  File "/usr/local/lib/python2.7/site-packages/splunklib/client.py", line 1230, in __getitem__
    raise KeyError(key)
KeyError: UrlEncoded('cloud_custodian')

What do you get when you try to list indexes with `service.indexes.keys()` ? — zwer, Dec 08 '17 at 15:11
Ah, gotta love custom types... How about: `print(", ".join([str(x) for x in service.indexes]))` — zwer, Dec 08 '17 at 18:54
`splunklib.binding.HTTPError: HTTP 404 Not Found -- {"text":"The requested URL was not found on this server.","code":404}` — Lightbeard, Dec 08 '17 at 18:59
which makes sense because the client is not pointing to the full URL — Lightbeard, Dec 08 '17 at 19:01
Ok, then try to get the app endpoint (looking at their API docs): `service.get(URI)` and see what happens. — zwer, Dec 08 '17 at 19:17
`s = service.get("services/collector/raw")` results in a stack trace ending in `splunklib.binding.HTTPError: HTTP 405 Method Not Allowed -- {"text":"The requested URL was not found on this server.","code":404}` — Lightbeard, Dec 08 '17 at 20:06

score 5 · Accepted Answer · answered Dec 11 '17 at 16:00

5

Ended up performing a stock POST with requests. I'm not sure if the splunk client is even intended to support the HTTP Event Collector.

import requests

url='https://splunkcollector.hostname.com:443/services/collector/event'
authHeader = {'Authorization': 'Splunk {}'.format('ABCDEFG-8A55-4ABB-HIJK-1A7E6637LMNO')}
jsonDict = {"index":"cloud_custodian", "event": { 'message' : "Dummy test python client log" } }

r = requests.post(url, headers=authHeader, json=jsonDict, verify=False)
print r.text

answered Dec 11 '17 at 16:00

Lightbeard

4,011
10
49
59

1

works for me, would also recommend including `time` in `jsonDict` so your messages are received in order. And you may need, depending on your config, `X-Splunk-Request-Channel` (a random GUID) in your header as well – ObjectNameDisplay May 07 '19 at 02:21
1

I also add 'sourcetype' like so: jsonDict = {"index":"cloud_custodian", "sourcetype": "Thor", "event": {...} } – SoloPilot Oct 29 '19 at 21:06

score 0 · Answer 2 · answered Dec 10 '17 at 17:48

You should look into the HTTP Event Collector in Splunk. It's as simple as enabling it, generating a token, and making the call.

If you wanted to send data to Splunk HEC, it would look like this

<protocol>://<host>:<port>/<endpoint>

https://docs.splunk.com/Documentation/SplunkCloud/6.6.0/Data/UsetheHTTPEventCollector

write log entry to splunk via HTTP in python

2 Answers2