The Challenge
I am developing an Android background service that will communicate with a backend API. Also, the app will be distributed through a mobile device management system and there is no client interaction, such as inputting user info.
You have got yourself a huge challenge here :)
Locking down the API server to genuine and unmodified instances of a mobile app is already a very hard task when user authentication(who) is involved, but when this is missing then things become really difficult, but not impossible.
Before we continue further I would like to first clear a misconception that I usually find among developers of any seniority, that is about the difference between who and what is accessing an API server.
The Difference Between WHO and WHAT is Accessing the API Server
I wrote a series of articles around API and Mobile security, and in the article Why Does Your Mobile App Need An Api Key? you can read in detail the difference between who and what is accessing your API server, but I will extract here the main takes from it:
The what is the thing making the request to the API server. Is it really a genuine instance of your mobile app, or is it a bot, an automated script or an attacker manually poking around your API server with a tool like Postman?
The who is the user of the mobile app that we can authenticate, authorize and identify in several ways, like using OpenID Connect or OAUTH2 flows.
So think about the who as the user your API server will be able to Authenticate and Authorize access to the data, and think about the what as the software making that request in behalf of the user.
In your case you don't have a user, therefore you will need to put your efforts on creating a solution that allows for the API server to have a very high degree of confidence that it can be sure that the request is from what it expects, a genuine and unmodified version of the mobile app where your Android background service is running.
Your Solution to the Challenge
One idea I had was to distribute one-time invitation codes with the app that were preconfigured on the server so each instance of the app may register with the service by generating a username and password, and then it could use OAuth to retrieve access tokens.
As you may be aware by now your solution will only provide the who in the request, not what is doing the request. The what in the request is already referenced by you when you say I do not want to allow any arbitrary instance of the service to have access to the API. This can be mobile apps instrumented at runtime, repackaged mobile apps, cloned mobile apps, bots running automated scritps to make request like the real mobile app or even manually replayed requests by an attacker.
Also, the one-time invitation codes can be easily extracted from the APK binary with reverse engineer techniques, like I show in my article How to Extract an API key from a Mobile App with Static Binary Analysis:
The range of open source tools available for reverse engineering is huge, and we really can't scratch the surface of this topic in this article, but instead we will focus in using the Mobile Security Framework(MobSF) to demonstrate how to reverse engineer the APK of our mobile app. MobSF is a collection of open source tools that present their results in an attractive dashboard, but the same tools used under the hood within MobSF and elsewhere can be used individually to achieve the same results.
During this article we will use the Android Hide Secrets research repository that is a dummy mobile app with API keys hidden using several different techniques.
A Possible Better Solution
How can I securely authorize each instance of the background service so they may access the backend service?
I recommend you to read this answer I gave to the question How to secure an API REST for mobile app?, especially the sections Hardening and Shielding the Mobile App, Securing the API Server and A Possible Better Solution.
The linked answer will allow you to understand the several options you may want to consider, and also highlight the Mobile App Attestation concept as the one that will allow the API server to have a very high degree of confidence to only accept requests from genuine and unmodified instances of the mobile app running the Android background service.
Do You Want To Go The Extra Mile?
In any response to a security question I always like to reference the excellent work from the OWASP foundation.
For APIS
OWASP API Security Top 10
The OWASP API Security Project seeks to provide value to software developers and security assessors by underscoring the potential risks in insecure APIs, and illustrating how these risks may be mitigated. In order to facilitate this goal, the OWASP API Security Project will create and maintain a Top 10 API Security Risks document, as well as a documentation portal for best practices when creating or assessing APIs.
For Mobile Apps
OWASP Mobile Security Project - Top 10 risks
The OWASP Mobile Security Project is a centralized resource intended to give developers and security teams the resources they need to build and maintain secure mobile applications. Through the project, our goal is to classify mobile security risks and provide developmental controls to reduce their impact or likelihood of exploitation.
OWASP - Mobile Security Testing Guide:
The Mobile Security Testing Guide (MSTG) is a comprehensive manual for mobile app security development, testing and reverse engineering.
