Finding models that do not have a HABTM relation or at least one that has a certain attribute

Question

In a Rails 5 application, I have two models associated via a has_and_belongs_to_many association:

class User < ApplicationRecord
    has_and_belongs_to_many :groups
end

class Group < ApplicationRecord
    has_and_belongs_to_many :users
    scope :public_only, -> { where(public: true) }
end

The association table is called users_groups.

I want to find users that do not belong to any groups or belong to at least one group that has the attribute public set to true.

I can write individual scopes for that, based on the answer to this question:

scope :public_only, -> { joins(:groups).where(groups: { public: true }) }
scope :not_in_any_group, -> {
  joins("LEFT JOIN users_groups ON users.id = users_groups.user_id")
  .where("users_groups.user_id IS NULL")
}

But I need to combine these. So I tried an .or:

Users.public_only.or(Users.not_in_any_group)

... which raised an error:

ArgumentError: Relation passed to #or must be structurally compatible. Incompatible values: [:joins, :references]

How do I combine these two apparently incompatible queries? I found this question, but the example there is a little more trivial.