Max Array Length post__not_in Parameter in Wp_Query

Question

I'm using a page on WordPress to make a post request to a local php file to facilitate infinite scrolling. The arguments for the WP_query each time is the following:

$args =  array(
            'post_status' => 'publish',
            'post_type' =>  array( 'post'),
            'post__not_in' => $_POST["LoadedIDs"],
            'posts_per_page' => $numberofResources,
            'orderby'  => 'rand',
            'meta_query' => array(
                array(
                 'key' => 'shares',
                 'value' => 100,
                 'compare' => '>=',
                 'type' => 'numeric',
                ),
            )
    );

LoadedIDs is a list of IDs that have already been fetched, so naturally this gets longer after each request. Eventually I notice that I am getting duplicates back. I assume this is due to a limit on the length of the array for post__not_in. Does anyone know what that is please?

I don't know the answer to your question but I know a way to solve your problem. Don't save all ID's but only the amount of posts you loaded. Then use offset instead of post__not_in — jrswgtr, Nov 10 '19 at 21:03
I have found link for you. you can refer this link also, - https://rudrastyh.com/wordpress/load-more-posts-ajax.html — Jayesh, Nov 11 '19 at 05:10
How are you passing the IDs back to the query? If you are passing them as a string like so: "223, 234, 235, 8738" - Then the query will recognize it as one single value. You should make sure that what you are passing to the query is an actual array and not just a string of values. — DubVader, Nov 25 '19 at 17:54
Thanks but It is passed as any array. It works for the first few times I do it until the array is too long and then starts ignoring them. — Atrag, Nov 26 '19 at 11:05
Try some of the suggestions on this page: https://stackoverflow.com/questions/6135427/increasing-the-maximum-post-size. PHP has built-in settings that limit the size of `$_POST`. — Cave Johnson, Nov 26 '19 at 17:14
Depending on how many posts you have, it might be worth looking into querying all posts in a random order, then use PHP to grab the next X number of not-loaded posts. array_diff() may be your friend. Otherwise, try to avoid in_array() repeatedly in a very large loop (if the array is also large). Storing the loaded post IDs in the array keys and checking isset will be faster than in_array many times. But probably array_diff is your best bet. Strongly recommend you profile how long this takes if you're going to do it this way. — Joel M, Nov 27 '19 at 01:08

Joel M · Answer 1 · 2020-01-09T20:13:53.977

You should profile this to see how long it takes. If you have a very large number of posts (lets say 5-50k), then the query could take some time to execute, but also testing against the posts that are already loaded can be time consuming as well.

Also, I assume that the loaded IDs are being sent from the client side. If this is the case, and there are a very large number of posts, then you could run into issues having to do with getting the data from the client to the server. You should check $_POST to ensure that all of the IDs are present. If you have too many IDs then you might have to resort to something like storing the loaded IDs in session but this can get messy if not done right.

Anyways, this try:

$exclude = is_array( @$_POST['LoadedIDs'] ) ? $_POST['LoadedIDs'] : [];

// use array keys for faster lookup
$_exclude = array_flip( $exclude );

$all_posts = get_posts( [
    'post_status'    => 'publish',
    'post_type'      => [ 'post' ],
    'posts_per_page' => - 1,
    'orderby'        => 'rand',
    'meta_query'     => array(
        array(
            'key'     => 'shares',
            'value'   => 100,
            'compare' => '>=',
            'type'    => 'numeric',
        ),
    )
]);

// get the next X number of posts.
$next_posts = array_slice( array_filter( $all_posts, function ( $p ) use( $_exclude ){
    return ! isset( $_exclude[$p->ID] );
} ), 0, $numberofResources );

It should be noted that this is technically less efficient (than excluding posts in SQL) but I don't think it will make any noticeable difference with hundreds or even a thousand posts in total. — Joel M, Nov 27 '19 at 01:22
If max_input_vars does end up being the issue (which is a definite possibility), then you should json_encode the loaded IDs as a string when passing to the server, then json_decode in php. I've done this work around for other similar issues. It's possible that each element of $_POST['LoadedIDs'] counts as 1 element towards the max_input_vars. — Joel M, Nov 28 '19 at 01:37

Leigh Bicknell · Answer 2 · 2019-11-27T10:27:17.097

I assume this is due to a limit on the length of the array for post__not_in. Does anyone know what that is please?

I don't believe so, a wrote rudimentary test on a site with 1700+ posts+pages to test this case, and it reveals no duplicates ever returned.

Here is my code:

add_action('wp_footer', function() {
    $allPosts = get_posts([
        'post_status'    => 'publish',
        'post_type'      => ['post', 'page'],
        'posts_per_page' => -1,
        'orderby'        => 'rand'
    ]);
    $totalPostCount = count($allPosts);

    $duplicatesFound = false;
    $loadedIds = [];
    while (!$duplicatesFound && count($loadedIds) < $totalPostCount) {
        $args = array(
                'post_status' => 'publish',
                'post_type' =>  array('post', 'page'),
                'post__not_in' => $loadedIds,
                'posts_per_page' => 100,
                'orderby'  => 'rand',
        );

        $posts = get_posts($args);
        $postIds = wp_list_pluck($posts, 'ID');

        $loadedIds = array_merge($loadedIds, $postIds);
        $uniqueIds = array_unique($loadedIds);
        if (count($uniqueIds) < count($loadedIds)) {
            $duplicatesFound = true;
        }
    }


    var_dump($duplicatesFound);
    var_dump($totalPostCount);
    var_dump(count($loadedIds));
    var_dump(count($uniqueIds));
    var_dump($loadedIds);
    var_dump($uniqueIds);
    die();
});

If you have more than 1700 posts, it's possible that it wasn't enough to reproduce the issue in my test, in which case i'd suggest running the above code yourself to be certain.

Assuming you have already verified that your $_POST["LoadedIDs"] contains a valid array of ID's (not a comma separated string). My best guess is that your data is getting lost/cut short in POST, to verify this you could try Joal M's solution. If that doesn't solve your problem this will confirm that your data is being lost in POST.

To fix this you could try increasing some ini settings:

max_post_size = 100M
max_input_vars = 3000
suhosin.post.max_vars = 3000
suhosin.request.max_vars = 3000

max_input_vars is most likely the culprit as this is limited to 1000 by default.

The other alternative is to not pass the ID's back from the client but instead store them on the server perhaps in session as Joal M also suggested, but i'd save this for a final resort.

I briefly checked the code in WP_Query which showed no indication of truncating the list of IDs. You should try storing the WP_Query in a variable (as opposed to using get_posts() and print $query->request to make sure. In the past I've done similar SQL queries with long lists of IN or NOT IN arguments and it was never a problem. Not sure where the issue lies to be honest. — Joel M, Nov 28 '19 at 01:32

Max Array Length post__not_in Parameter in Wp_Query

2 Answers2