How to get data from SEC Edgar python and a json

Question

on the following page below there is as Data source a json link: https://www.sec.gov/edgar/browse/?CIK=1067983&owner=exclude Data source: CIK0001067983.json -> https://data.sec.gov/submissions/CIK0001067983.json

This is my code (it works fine!):

headers = {
"Host": "www.sec.gov",
"User-Agent": "jo boulement jo@gmx.at",
"Accept-Encoding": "gzip, deflate" 
}

sec_url = "https://data.sec.gov/submissions/CIK0001067983.json"
resp = requests.get(sec_url, headers=headers)
with open("e:\\sec_api_of_1448574_7.html", "w", encoding="utf-8") as my_file:
my_file.write(resp.text)

but as result I get a file looks like this: enter image description here

Error 404: Page Not Found Oops! Page Not Found.

What is here going wrong? The json-link: https://data.sec.gov/submissions/CIK0001067983.json is fine, because download by hand from the page works fine. Hope somebody could give me a hint! Thx!

Why are you manually setting this header `"Host": "www.sec.gov",`? That does not match `data.sec.gov`. — metatoaster, Sep 27 '22 at 05:50

score 3 · Answer 1 · answered Sep 27 '22 at 07:19

thx, for your help ... I have the solution ...

the documentation of the sec.gov says the following:

https://www.sec.gov/os/webmaster-faq#user-agent

but the header "HOST" lead to the "404 page not found" ...

but this header works fine:

headers = {
"User-Agent": "jo boulement jo@gmx.at",
"Accept-Encoding": "gzip, deflate" 
}

crazy! because the documentation says something else :(

score 1 · Answer 2 · answered Sep 27 '22 at 06:10

A web server checks the headers that you send in your request and might decide to return an error page if you don't include certain headers. In this case, it looks like they return an error if you don't include a valid user agent.

This works for me:

import requests

headers = {'user-agent': 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/105.0.0.0 Safari/537.36'}

url = "https://data.sec.gov/submissions/CIK0001067983.json"

payload={}

response = requests.request("GET", url, headers=headers, data=payload)

print(response.text)

yes, thx for your comment, works also for me! – JKR Sep 27 '22 at 07:21 — JKR, Sep 27 '22 at 07:21

score -1 · Answer 3 · answered Sep 27 '22 at 06:05

The HTML that gets returned includes this <script> tag:

<script src="/files/js/js_DkdESgtfPfV7guog-Lhz7nda0K-ISZe0-gHU4CF6Wo0.js"></script>

My guess is that the script referenced by the tag is what causes the JSON data to be returned. A browser will run that script as part of rendering the HTML. The Requests package doesn't do this. It just returns the raw HTML. You might need to use something like Puppeteer or Selenium to get the JSON via that URL.

How to get data from SEC Edgar python and a json

3 Answers3

Linked