0

I recently wrote a PHP plugin to interface with my phpBB installation which will take my users' Steam IDs, convert them into the community ids that Steam uses on their website, grab the xml file for that community id, get the value of avatarFull (which contains the link to the full avatar), download it via curl, resize it, and set it as the user's new avatar.

In effect it is syncing my forum's avatars with Steam's avatars (Steam is a gaming community/platform and I run a gaming clan). My issue is that whenever I am reading the value from the xml file it takes around a second for each user as it loads the entire xml file before searching for the variable and this causes the entire script to take a very long time to complete.

Ideally I want to have my script run several times a day to check each avatarFull value from Steam and check to see if it has changed (and download the file if it has), but it currently takes just too long for me to tie up everything to wait on it.

Is there any way to have the server serve up just the xml value that I am looking for without loading the entire thing? Here is how I am calling the value currently:

$xml = @simplexml_load_file("http://steamcommunity.com/profiles/".$steamid."?xml=1");
$avatarlink = $xml->avatarFull;

And here is an example xml file: XML file

Colton McCormack
  • 771
  • 11
  • 23

5 Answers5

2

The file isn't big. Parsing it doesn't take much time. Your second is wasted mostly for network communication.

Since there is no way around this, you must implement a cache. Schedule a script that will run on your server every hour or so, looking for changes. This script will take a lot of time - at least a second for every user; several seconds if the picture has to be downloaded.

When it has the latest picture, it will store it in some predefined location on your server. The scripts that serve your webpage will use this location instead of communicating with Steam. That way they will work instantly, and the pictures will be at most 1 hour out-of-date.

Added: Here's an idea to complement this: Have your visitors perform AJAX requests to Steam and check if the picture has changed via JavaScript. Do this only for pictures that they're actually viewing. If it has, then you can immediately replace the outdated picture in their browser. Also you can notify your server who can then download the updated picture immediately. Perhaps you won't even need to schedule anything yourself.

BenMorel
  • 34,448
  • 50
  • 182
  • 322
Vilx-
  • 104,512
  • 87
  • 279
  • 422
  • I thought about this, but with thousands of pageviews a day and a fairly small chance that the user's avatar has changed there there will probably be much more time spent waiting for everything to run than before. Plus given your solution it would probably be ill-advised to use the new picture on that page as it would take several seconds to check all of them, download the new ones, and then replace the current ones causing the images to refresh much later than is needed. – Colton McCormack Jan 12 '11 at 06:27
1

You have to read the whole stream to get to the data you need, but it doesn't have to be kept in memory.

If I were doing this with Java, I'd use a SAX parser instead of a DOM parser. I could handle the few values I was interested in and not keep a large DOM in memory. See if there's something equivalent for you with PHP.

duffymo
  • 305,152
  • 44
  • 369
  • 561
1

SimpleXml is a DOM parser. It will load and parse the entire document into memory before you can work with it. If you do not want that, use XMLReader which will allow you to process the XML while you are reading it from a stream, e.g. you could exit processing once the avatar was fetched.

But like other people already pointed out elsewhere on this page, with a file as small as shown, this is likely rather a network latency issue than an XML issue.

Also see Best XML Parser for PHP

Community
  • 1
  • 1
Gordon
  • 312,688
  • 75
  • 539
  • 559
0

that file looks small enough. It shouldn't take that long to parse. It probably takes that long because of some sort of network problem and the slowness of parsing.

If the network is your issue then no amount of trickery will help you :(.

If isn't the network then you could try a regex match on the input. That will probably be marginally faster.

Try this expression:

/<avatarFull><![CDATA[(.*?)]]><\/avatarFull>/

and read the link from the first group match.

You could try the SAX way of parsing (http://php.net/manual/en/book.xml.php) but as i said since the file is small i doubt it will really make a difference.

Mihai Toader
  • 12,041
  • 1
  • 29
  • 33
  • I'd probably just use `strpos` which is as fast as it gets. Although either of these solutions will break on even insignificant changes in the XML formatting. – Vilx- Jan 10 '11 at 10:48
  • That should also work. I'm not expecting any interesting speed differences between regexp and strpos in this case anyway :). – Mihai Toader Jan 10 '11 at 10:51
0

You can take advantage of caching the results of simplexml_load_file() somewhere like memcached or filesystem. Here is typical workflow:

  • check if XML file was processed during last N seconds
  • return processing results on success
  • on failure get results from simplexml
  • process them
  • resize images
  • store results in cache
nefo_x
  • 3,050
  • 4
  • 27
  • 40