It's important in cases like this to rule out as many potential sources of slowness as you can. When anything in PHP takes a long time, there are any number of things that could cause it, from your server configuration, to the remote system just being slow.
Have you already tried running this query from another server, or from your own local machine? Try to find some common factors, isolate the problem and see if it doesn't help narrow down the problem.
Infusionsoft's performance has been very spotty in my experience. Sometimes it would work perfectly for months on end, but usually not. Whenever we had a problem--some code that suddenly stops working when it had worked for years before or a process that once took less than a second suddenly slowing to a crawl--more often than not we could traced it back to something on their end. And very often it was something that speaks of terrible QA-- Just this week we had an issue where queries on saved searches were returning entirely random information, and if we hadn't validated the entries before doing anything we might have ended up causing some serious issues for our clients. Your issue doesn't sound all that exceptional, sadly. You may have to consider, if the same problem occurs under different test cases that it's something on their end that you can't remedy.
Regarding the API, you are using an older, deprecated version. I'd recommend transitioning from it, since it had all kinds of problems--it would work, mind you, and should still work indefinitely barring a few easily remedied issues regarding HTTP certificates, but its code quality is poor.
Infusionsoft has provided a replacement API which they're just calling infusionsoft-php. Novak Solutions also provides an excellent object-oriented version as well, which they, confusingly enough, call infusionsoft-php-sdk. There's an option to use their XMLHTTP gateway as well,