TL;DR It is impossible as of now
Why we need this?
There seems to be Android phones where Chrome has this enabled by default based on the theme mode they choose for the phone (light vs dark).
If you develop templates and have old items from 2017-2018, customers will ask for refunds when they have clients complaining. It is almost impossible to diagnose if you did not know such a feature exist. In many cases the website in un-readable.
It invalidates the dark/light toggle experience on websites (destroys the experience on CSS websites on how to do it). Not being able to detect when this happens is also not helping, we could remove the toggle and serve the dark theme directly or warn the user that something is wrong.
No Solution
There is no way to change it as of now, they even change the background on the images and they do a good job too, .jpeg... . It is not just a simple color swap.
I think they go with "the user is king" approach. If the user wants to enforce it they will side with the user.
It is getting better and better on each update.
It messes up with the color picker in the dev-inspection-tool too...
Even if they were to add a "fix" it would not be available on old browsers. I dont think they even thought of implementing a way to bypass or "white list".
But
There is a conceptual approach here:
https://stackoverflow.com/a/60462984/1427338
I had mixed results with the css. In a simple page it works but in more complex projects there were too many edge cases to handle them all, and no fix for the image(... it replaced the background in image!)