Gluon Mobile VideoService doesn't work on iphone in background

Question

I'm using VideoService to playback a local audio file (mp3) in my application, and it works fine while the application is active, both on Android and iPhone. But on iPhone, when the application is in background it doesn't work: nothing happens when service.play() is called. The code is trivial:

            Services.get(VideoService.class).ifPresent(service -> {
                service.getPlaylist().add("1.mp3");
                service.play();
            });

I can see "AVPlayer hidden" and "AVPlayerStatusReadyToPlay" in my IDEA's console.

If the playback is already started and I put my application in background (using iPhone's "home" button or by turning off the screen) - it stops playing and resumes only after I bring the application back to active state manually.

JavaDocs say no specific iOS configuration is required, though I put "audio" in plist's UIBackgroundModes array (doesn't help either).

iPhone 6, iOS 12.1 (16B92)

On Android the same code works just fine both in active and background modes without any problems.

What am I missing?

So far, this will require a change in the jfxmobile plugin, as there are some required changes in the [Launcher class](https://bitbucket.org/javafxports/javafxmobile-plugin/src/811de9a1c490ebc2693cf952d2c27aba2921129b/src/main/resources/ios/sources/BasicLauncher.java#lines-38), as you can see [here](https://stackoverflow.com/a/4771303/3956070). — José Pereda, Jan 15 '19 at 18:00
Ok, I have created a custom Launcher. In order to compile I've got to add two dependencies to compile in my build.gradle: compile 'com.gluonhq:robovm-cocoatouch:2.3.5-ios12' compile 'com.gluonhq:robovm-rt:2.3.5-ios12' But I can't run my application with it: Execution failed for task ':FastRiderFXApp:launchIOSDevice'. > Native Library /Users/aventa/.gradle/caches/modules-2/files-2.1/com.gluonhq/robovm-dist/2.3.5-ios12/e6fbde2bba5ef265f8484d793a2fb620657fbaa3/unpacked/robovm-2.3.5-ios12/bin/libhfscompressor.dylib already loaded in another classloader — oleg97, Jan 17 '19 at 14:27
If you add the launcher to the project's `src/ios/java` folder, it will work fine without adding those dependencies. — José Pereda, Jan 17 '19 at 14:50
Yes, I've put my launcher under `src/ios/java`, added `AVAudioSession.getSharedInstance().setCategory(AVAudioSessionCategory.Playback);` and it runs, but still the audio stops playing as soon as the application goes to background. Could you post your changes in the launcher? — oleg97, Jan 17 '19 at 15:54
I did exactly the same you described in your answer, still no audio in background. It resumes playing right after I bring the application to foreground again. It seems we're still missing something... — oleg97, Jan 17 '19 at 16:32
Make sure you modify the plist file too. I've also added the `setActive(true)` to the launcher, not sure if you had that. With just the changes listed in my answer, it works fine on my iPhone. — José Pereda, Jan 17 '19 at 16:57
I've even copy-pasted your code just in case I've missed something and don't see it - no success. Yes, plist includes the key, double checked. Did grails-clean and deleted cache - nothing. I'll try ask my colleagues to test it on another device, maybe the problem is in my iPhone. — oleg97, Jan 17 '19 at 17:04
It doesn't work on other iPhones either. But - oddly - it works just fine on simulator. — oleg97, Jan 17 '19 at 18:28
Ok, I've found the problem: there was multiple `UIBackgroundModes` keys in my plist, and compiler didn't complain about that, so I've missed it. Only the last one was in effect. I will mark your answer, thank you! Still, there is another problem: this way we can _continue_ playing in background what we've started in foreground. It doesn't work for _start_ playing while in background (e.g. as a reaction to some asynchronous event). How do we achieve that? — oleg97, Jan 17 '19 at 19:31
It looks like you can't start audio from background with AVAudioPlayer, see [this](https://stackoverflow.com/questions/37351815/start-playing-sound-in-background-on-ios). There are other options but not implemented by Video Service. — José Pereda, Jan 17 '19 at 19:59

score 1 · Accepted Answer · answered Jan 17 '19 at 16:22

To be able to play audio in background, and based on this answer, it seems that the current Charm Down Video Service requires some modifications, in order to set the category to AVAudioSessionCategory.Playback.

One possible way to do this, is by modifying applicationDidFinishLaunching, from the iOS Launcher class. The jfxmobile plugin 1.3.16 creates this launcher here.

So we could modify that class, and build a custom version of the jfxmobile plugin, or, as the OP mentioned in the comments, it is possible to create a custom launcher.

The other possible way is adding this to the Charm Down video service directly, but it will require compiling a new version.

Let's try the custom launcher as it won't require building new versions.

Creating a CustomLauncher

Let's copy the default launcher into our project, into the src/ios/java folder, as it requires some specific dependencies for iOS.

Then add the required code to set the Playback option, starting from the main class:

private static final Class<? extends Application> mainClass = your.package.YourGluonApplication.class;

private static final Class<? extends Preloader> preloaderClass = null;

@Override
public boolean didFinishLaunching(UIApplication application, 
    UIApplicationLaunchOptions launchOptions) {

    // Audio settings to play in background mode ---
    try {
        AVAudioSession session = AVAudioSession.getSharedInstance();
        session.setActive(true);
        session.setCategory(AVAudioSessionCategory.Playback);
    } catch (NSErrorException nse) {
        System.out.println("Error AVAudioSession " + nse);
        nse.printStackTrace();
    }
    // --- End Audio settings

    Thread launchThread = new Thread() { ... }
    launchThread.setDaemon(true);
    launchThread.start();

    return true;
}

Using the custom launcher

As commented in the launcher class, the custom launcher can be loaded from the build.gradle file:

jfxmobile {
    downConfig {
        version = '3.8.6'
        plugins 'display', 'lifecycle', 'statusbar', 'storage', 'video'
    }
    ios {
        javafxportsVersion = '8.60.11'
        launcherClassName = 'your.package.CustomLauncher'
        infoPList = file('src/ios/Default-Info.plist')
        ...
    }
}

Allow background audio

The last required step to get audio playing not only when the app is running on the foreground but also when it goes to the background: modify the plist file.

We need to add this key to the Default-info.plist file:

<key>UIBackgroundModes</key>
        <array>
            <string>audio</string>
        </array>

Test

Let's add an mp3 file to src/main/resources/, like 1.mp3, and include this call in our Java code:

Services.get(VideoService.class).ifPresent(service -> {
            service.getPlaylist().add("1.mp3");
            service.play();
        });

Time to deploy to an iOS device:

./gradlew launchIOSDevice

The app should play the audio in both foreground and background mode as expected.

Gluon Mobile VideoService doesn't work on iphone in background

1 Answers1