13

I'm using a JSON file to autopopulate a drop down list. It's by no means massive (3000 lines and growing) but the time taken to refresh the page is becoming very noticeable.

The first time the page is loaded the JSON is read, depending on what option the user has selected dictates which part of the JSON is used to populate the drop down.

It's then loaded on every refresh or menu selection after. Is it possible to somehow cache the values to prevent the need for it to be reloaded time and time again?

Thanks.

EDIT: More Info: It's essentially a unit converter. The JSON holds all the details. When a users selects 'Temp' for example a call is made and the lists are populated. Once a conversion is complete you can spend all day running temp conversions and they'll be fine but everytime a user changes conversion type so now length, the page refreshes and takes a noticeable amount of time.

null
  • 3,469
  • 7
  • 41
  • 90
  • depending on your server side language, have you considered sessions/cookies/database/flat file storage? – andrew Jul 12 '13 at 10:40
  • Hi Andrew, I'm using PHP server side. I did briefly consider a DB but now that the JSON is being utilised I have to stick with this method. – null Jul 12 '13 at 10:52
  • 1
    i meant temp storage - get your json data from whatever source, then temporary store them to sessions for example, to avoid re-loading them – andrew Jul 12 '13 at 11:49
  • 1
    How does a simple unit converter need ~3000+ items of data? Btw, you should strip all whitespaces so the JSON isn't measured in "lines" but just in KB. – Bergi Jul 12 '13 at 13:22
  • @Bergi, I didn't make it clear but there are many many many unit being converted, not just the one :) Thanks for mentioning the whitespace, I can be a bit over the top when it comes to formatting and making sure readability is always good but if it improves the storage then I'll clean it. – null Jul 12 '13 at 13:28
  • 1
    I'm confused, are you trying to cache the JSON, or the HTML of the rendered drop-downs? Based on that fiddle you posted, it looks like you load the JSON server side and use it to render the HTML. My guess is the load time is more likely the HTML traveling the network, or the browser drawing a massive drop-down-list, and not the server side code that reads in the JSON. – Walter Stabosz Jul 12 '13 at 13:45
  • Ideally I think it would the HTML for the drop downs but this comes directly from the JSON. So surely one would be the same as the other in this case? I guess it would either mean caching each drop down list per user option or caching the whole JSON and then accessing the cache to populate the lists instead...I think. – null Jul 12 '13 at 13:49
  • There is going to be a size difference between JSON and rendered HTML. `{"apple":1}` translates to ``. That's 21 extra bytes for the user to download per list item. Still, that's just 2100 bytes for a 1000 item list, so not really a big download time. If you want to render the select lists dynamically on the client side, see my answer here: http://stackoverflow.com/questions/9995017/best-way-to-populate-select-list-with-jquery-json/9995132#9995132 – Walter Stabosz Jul 12 '13 at 13:56
  • Thanks Walter, I had a look. It's a interesting avenue to explore. I wasn't expecting so many options :) – null Jul 12 '13 at 14:19

6 Answers6

15

Unfortunately, I don't know of a standardized global caching mechanism in PHP. This article says that Optimizer Plus, a third party accelerator, is being included in core PHP starting in version 5.5. Not sure what version you are using but you could try that.

On a different note, have you considered file storage as andrew pointed out? I think it combined with $_SESSION could really help you in this case. Let me give you an example that would work with your existing JSON data:

Server Side

Store your JSON data in a .json file on your PHP server:

{
    "data": "some data",
    "data2": "more data",
    "data3": [
        ...
     ],
     etc.
}

Note: Make sure to properly format your JSON data. Remember all strings must be enclosed in double quotes ".

In PHP, use an if statement to decide the appropriate action:

error_reporting(E_ALL);
ini_set("display_errors", "On");
session_start();

if(isset($_SESSION['dataCache'])) {
    echo json_encode($_SESSION['dataCache']);
} else {
    $file = 'data.json';
    if (!is_file($file) || !is_readable($file)) {
        die("File not accessible.");
    }
    $contents = file_get_contents($file);
    $_SESSION['dataCache'] = json_decode($contents, true);
    echo $contents;
}

So lets dig into the above coding a little more. So here's what we are doing in a nutshell:

  1. Turn on error reporting and start session support.
  2. Check to see if we've already read the file for this user.
  3. If so, pull the value from storage and echo it out and exit. If not continue below.
  4. Save off the file name and do a little error checking to ensure PHP can find, open and read the contents of the file.
  5. Read the file contents.
  6. Save the decoded json, which is not an array because of the `true` parameter passed to `json_decode`, into your `$_SESSION` variable.
  7. Echo the contents to the screen.

This will save you the time and hazzle of parsing out JSON data and/or building it manually on the server. It will be cached for the users session so that they can use it through out.

Client Side

I assume you are using ajax to fetch the information? If not correct me, but I was assuming that's where some of your JavaScript comes into play. If so you may consider this:

Store the returned data in sessionStorage on the user's browser when it's returned from the server:

$.ajax({
    ...
    success: function (res) {
        localStorage.setItem("dataCache", JSON.stringify(res));
    },
    ...
});

Or if you use promise objects:

$.ajax({
    ...
}).done(function (res) {
    localStorage.setItem("dataCache", JSON.stringify(res));
});

When you need to read it you can do a simple test:

var data;
// This returns null if the item is not in local storage.
// Since JavaScript is truthy falsy, it will be evaluated as false.

if(localStorage.getItem("dataCache")) {
    data = JSON.parse(localStorage.getItem("dataCache"));
} else {
    // Make ajax call, fetch object and store in localStorage in the success or done callbacks as described above
}

Notes:

localStorage is a new feature in HTML5, so it's not fully supported on all browsers yet. Most of the major ones do however, even as far back as IE8 (I think). However, there is no standardized size limit on how much these browsers are required to hold per site.

It's important to take that into consideration. I can guarantee you probably will not be able to store the entire 30,000 line string in localStorage. However, you could use this as a start. Combined with the server side solution, you should see a performance increase.

Hope this helps.

War10ck
  • 12,387
  • 7
  • 41
  • 54
  • Thats brilliant! Thank! Lots of information and additional details that make it easier to digest. Ajax, for the time being is foreign to me. To access the data I use : http://jsfiddle.net/wKCBD/. If I'm able to combine them I think I'll be one step closer to nirvana. I'm already setting session variables as users have the option of logging in etc, I'm right in thinking I can attach this to that session instead of creating a new one. (sessions are a touch foreign too :)) – null Jul 12 '13 at 13:17
  • @SteveGreen I'm still working on honing my ajax skills as well. It does take some practice, but you can sure see some great results when using it. Much faster than reloading an entire page. Good luck and happy coding buddy! Hope you reach your coding nirvana. :) – War10ck Jul 12 '13 at 13:27
  • I think the OP does neither need PHP nor DOMstorage caching. Just sending the appropriate HTTP cache headers should be enough. – Bergi Jul 12 '13 at 13:34
  • @Bergi That's a good point. :) I didn't try that when I tested the above solution on my local server but that would probably help too and/or be another possible solution altogether. Very good point. – War10ck Jul 12 '13 at 13:38
  • @Bergi, could you elaborate please? – null Jul 12 '13 at 13:43
  • [HTTP Headers: Controlling Cache and History Mechanism](http://stackoverflow.com/q/6491789/1048572) should answer everything, maybe there are even better questions – Bergi Jul 12 '13 at 13:50
  • The OP (and everyone else) should be made aware that storing lots of data in $_SESSION causes *every* request to slow down. Every time someone makes a request, the $_SESSION is repopulated with everything that was serialized on the last request. Storing lots of data on it and then doing lots of requests is a surefire way of DOSing yourself (http://stackoverflow.com/questions/7783563/using-php-session-variable-to-store-large-amonts-of-data) – Stephen Jul 12 '13 at 17:07
  • Localstorage can store 5MB, isn't that more than a 30,000 line string? – Frederik Witte Jul 29 '15 at 08:05
3

I use the browser's cache to ensure that my large chunk of JSON is only downloaded once per session. I program in ASP.NET, but I'm sure PHP has the same mechanisms:

  1. On session start, I generate a random string as session key for my dynamic JavaScripts. This key get stored in the ASP.NET session state under the key JsonSessionID. That way I can refer to it in my page markup.
  2. I have a "generic http handler" (an ashx file) that when called by the browser, returns a .js file containing my JSON.
  3. In my HTML I include the dynamic script: <script type="text/javascript" src="/dynamicJSON.ashx?v=<%= JsonSessionID %>"></script>

The browser will automatically cache any URLs included as scripts. The next time the browser is asked to load a cached script from a URL, it will just load up the file from the local disk. This includes dynamic pages like this.

By adding the ?v= in there, I ensure that the JSON is updated once per session.

Edit

I just realized that your JSON is probably static. If that's the case, you can just put your JSON into a static .js file that you include in your HTML, and the browser will cache it.

// conversionData.js
var conversionData = { "a":1,"b":2,"c":3 };

When you include the conversionData.js, the conversionData variable will be in scope with the rest of your page's JavaScript that dynamically updates the drop-downs.

Edit 2

If you are serving static files, this blog post has a good pattern for cache-busting based on the file's date modified property. i.e. the file is only downloaded when it is changed on the server.

I have yet to find a good method for cache-busting JSON created via database lookup tables, other than per-session. Which isn't ideal because the database could change mid-session.

Community
  • 1
  • 1
Walter Stabosz
  • 7,447
  • 5
  • 43
  • 75
  • Busting the cache only when new units are available (the JSON gets updated) might be even more cache-friendly than per-session-busting. Depends on the update rate of course. Still, +1! – Bergi Jul 12 '13 at 13:36
  • @Bergi: That's true. I added another edit to my answer for how to cache-bust once per static file change. – Walter Stabosz Sep 10 '14 at 13:47
1

Once you've got your JSON data decoded into an object you can just keep the object around, it should persist until a page reload at least.

If you want to persist between reloads you might want to look at HTML5's localStorage etc.

You would need to come up with an age strategy, maybe just dump the current date in there with it as well so you can compare that and expire as needed.

Lloyd
  • 29,197
  • 4
  • 84
  • 98
  • Hi Lloyd, I'm still not sure how this would help. It's essentially a unit converter. The JSON holds all the details. When a users selects 'Temp' for example a call is made and the lists are populated. Once a conversion is complete you can send all day running temp conversions and they'll be fine but everytime a user changes conversion type so now length, the page refreshes and takes a noticeable amount of time. -Added this to post as has more info. – null Jul 12 '13 at 10:56
0

I would suggest storing your json data to a session. On first page load you can write a script to get your json data, then store them into a session. on each page load/refresh afterwards you can check our session to decide what to do - use the session data or fetch again your json data.

This approach suites me for small scale data (for example: an array of products - colors - sizes - prices).

Based on your data you should test you loading times.

andrew
  • 2,058
  • 2
  • 25
  • 33
  • Thanks, I think I'll give it a try. Don't suppose you can tell em how to store it a session? I've googled a bit after the previous suggestion but not found anything useful. This is my current script for obtaining the JSON : http://jsfiddle.net/wKCBD/ – null Jul 12 '13 at 11:59
0

Here is a simple hack:

Create a call to a php file as GET request with parameter "bla-bla.html" or "bla-bla.css"... well you know, it makes browser think it is not a php, but rather "html" or "css". And browser will cache it.

To verify that the trick is working - go to the "network" tab of the browser dev panel and you will see column "type" there along with "transferred" - instead of having php there and actual size, you will find "html" and "(cached)"

This is also good to know when you passing parameters like "blah-blak.html" to the php file and expect it will not be cached. Well, it will be cached.

Tested on FireFox Quantum 57.0.1 (Mac 64bit)

P.S.

Chrome 63 on Mac is capable of recognising real file type in this situation. So it cannot be fooled.

Yevgeniy Afanasyev
  • 37,872
  • 26
  • 173
  • 191
-1

Thinking out of the box here:

but if your list has 3000 lines and growing (as you said) is it possible for you to establish its maximum size ?

let's say the answer is 10,000 (max) items; then do you really need an ajax call ? you could transfer the data straight away with the page

(depending on your architecture of course, you could come out with different solution)

Zo72
  • 14,593
  • 17
  • 71
  • 103
  • Hi, I'm expecting the max size to be around 30,000 or so. – null Jul 12 '13 at 10:53
  • @Steve Green. 1. Would you need to load them all of them anyway (to help auto completion) ? 2 are those small strings (e.g. 20 - 25 chars) ? if so my idea applies... ditch ajax and load them with the page – Zo72 Jul 12 '13 at 11:40