You cannot call functions like setBackground() from within a custom function.
When you run Apps Script code as a custom function it runs with a limited set of "safe" permissions, this is why you are not prompted to authorize the script.
The only way a custom function may interact with the calling sheet is by returning either a single value or a two dimensional array of data, which will be written from the location of the calling cell.
See my response on this similar question: How to change format of cell that runs function - Google Sheets Script Editor
If you must change the background color or perform other kinds of actions, you'll need to trigger the functionality another way, such as from a menu option or sidebar, which will trigger the authorization flow and run with full permissions.