Woo-commerce cart validation categories

Question

I tested this code in my Wordpress / Woocommerce website

add_filter( 'woocommerce_add_to_cart_validation', 'add_to_cart_validation_callback', 10, 3 );
function add_to_cart_validation_callback( $passed, $product_id, $quantity) {
// HERE set your alert text message
$message = __( 'ohoh dit gaat niet', 'woocommerce' );

if( ! WC()->cart->is_empty() ) {
    // Get the product category terms for the current product
    $terms_slugs = wp_get_post_terms( $product_id, 'product_cat', array('fields' => 'slugs'));

    // Loop through cart items
    foreach (WC()->cart->get_cart() as $cart_item ){
        if(  has_term( $terms_slugs, 'product_cat', $cart_item['product_id'] )) {
            $passed = false;
            wc_add_notice( $message, 'error' );
            break;
        }
    }
}
return $passed;
}

with this code, you can only buy something from one category. I am looking for something else like this

I have 4 categories - Home - Garden - Outfit - toys

If a user put a product in the cart from toys he can't buy something out another category.

But if the user puts a product in the cart from Garden he can combine this with the categories Home and Outfit.

Recap: the user can combine products out Home, Garden, and Outfit but never with Toys.