0

Trying to loop through some license ids to get data from a website. Example: when I enter id "E09OS0018" in the search box, I get a list of one school/daycare. But when I type the following code in my python script (website link and arguments obtained from developer tools), I get no data in the file. What's wrong with this requests.get() command? If I should use requests.post() instead, what arguments would I use with the requests.post() command (not very familiar with this approach).

flLicenseData = requests.get('https://cares.myflfamilies.com/PublicSearch/SuggestionSearch?text=E09OS0018&filter%5Bfilters%5D%5B0%5D%5Bvalue%5D=e09os0018&filter%5Bfilters%5D%5B0%5D%5Boperator%5D=contains&filter%5Bfilters%5D%5B0%5D%5Bfield%5D=&filter%5Bfilters%5D%5B0%5D%5BignoreCase%5D=true&filter%5Blogic%5D=and')
openFile = open('fldata', 'wb')
for chunk in flLicenseData.iter_content(100000):
    openFile.write(chunk)
Nazim Kerimbekov
  • 4,712
  • 8
  • 34
  • 58
coder101
  • 383
  • 4
  • 21
  • Have you checked whether ``flLicenseData.raise_for_status()`` contains an error? – MisterMiyagi Jun 24 '18 at 09:18
  • What's the output of `f1LicenseData.status_code()`? – Rishabh Agrahari Jun 24 '18 at 12:29
  • No error raised by flLicenseData.raise_for_status(). f1LicenseData.status_code() results in error "TypeError: 'int' object is not callable". Not sure why I get that error because flLicenseData type isn't 'int' (it is class 'requests.models.Response'). openFile.flush() also didn't result in anything meaningful, other that printing the license ID to the file. – coder101 Jun 24 '18 at 19:01
  • OK, figured out. For those looking for a solution, I used the following requests.post command instead: flLicenseData = requests.post('https://cares.myflfamilies.com/PublicSearch/Search?dcfSearchBox=E08BA0001') – coder101 Jun 25 '18 at 15:01

1 Answers1

0

do openFile.flush() before checking the file's content.

Most likely, you are reading the file immediately before the contents are (Actually) written to the file.

There could be a lag between the contents writen to the file handler and contents actually transfered to the physical file, Due to the levels of buffers between the programming language API, OS and the actual physical file.

use openFile.flush() to ensure that the data is written into the file. An excellent explanation of flush can be found here.

Or alternatively close the open file with openFile.close() or use a context manager

with open('fldata', 'wb') as open_file:
   for chunk in flLicenseData.iter_content(100000):
         openFile.write(chunk)
srj
  • 9,591
  • 2
  • 23
  • 27