2

My question is this; Is it possible to request two different URLs at the same time?

What I'm trying to do, is use a Python script to call requests from two different URLs at the same time. Making two PHP scripts run simultaneously (on different servers, running a terminal command). My issue is that I can't do them right after each other, because they each take a specific time to do something, and need to run at the same time, and end at the same time.

Is this possible using urllib2.urlopen? If so, how would I go about doing this?

If not, then what would be a good method to do so?

Currently I have something like: 

import urllib2
...
if cmd.startswith('!command '):
    cmdTime = cmd.replace('!command ','',1)
    urllib2.urlopen('http://example.com/page.php?time='+cmdTime)
    urllib2.urlopen('http://example2.com/page.php?time='+cmdTime)
    print "Finished."

My issue is that they don't run at the same time.

If I did !command 60, then it'll run site.com for 60 seconds, then go to site2.com for 60 seconds and run that one.

Axiom
  • 902
  • 1
  • 10
  • 23
  • 2
    Threading might be useful. http://stackoverflow.com/questions/16181121/python-very-simple-multithreading-parallel-url-fetching-without-queue – Tanveer Alam Jan 16 '15 at 06:10
  • See this post for a similar problem, with solutions http://stackoverflow.com/questions/27021440/python-requests-dont-wait-for-request-to-finish/27022707#27022707 note that "requests" is usually easier to use than "urllib2" but either will work in this case. – Andrew Gorcester Jan 16 '15 at 06:56

2 Answers2

3

I would suggest you to create a function for getting the parsed source, where you should pass a list of url's to be crawled in a list as argument. Later on loop on the list of URL's and use threading. I will post some sample code for you, please modify it accordingly.

import threading
import urllib2


def execute_url_list(urls_list):
    if cmd.startswith('!command '):
    cmdTime = cmd.replace('!command ','',1)
    for url in urls_list:
        urllib2.urlopen(url+cmdTime)

urls_list = ['url1', 'url2']
processes = []
for k in urls_list:
    process = threading.Thread(target=execute_url_list, args=[k])
    process.setDaemon(True)
    process.start()
    processes.append(process)


for process in processes:
    process.join()
Prateek
  • 1,538
  • 13
  • 22
  • Please note there are some [thread safety concerns](http://stackoverflow.com/questions/5825151/are-urllib2-and-httplib-thread-safe) about urllib2, though this example is okay as long as the urls are strings. – KillianDS Jan 16 '15 at 07:50
  • @KillianDS: I do agree with you that threading do have constraints with its usage, but string urls do not produce any problem as such. I have been using it since a long period. Thanks for pointing that out. – Prateek Jan 16 '15 at 07:55
  • Not sure, but I assume that 'processes = []' should be defined outside of the for loop. – Alex Nov 20 '18 at 14:32
  • 1
    @Alex: My bad processes=[] should be defined before the for loop otherwise it will get emptied in every single iteration. Thanks for pointing that out i will update the answer as well. – Prateek May 13 '19 at 03:26
0

I think you can use multi-thread and send one request in one thread. In python, you can inherit threading.Thread class and override run method. You can start two thread and use synchronous in the two thread to make sure the two request is sent almost at the same.

But I think it can't not make sure the php script on the server will be exactly executed at the same time because the network time and system schedule time is not under your control.

Jianguang Shi
  • 46
  • 2