Pygame Latency Issue

Question

I made a midiPlayer class for an experiment I'm running. It looks like this:

class midiPlayer(object):
    status = NOT_STARTED
    setVolume = 1
    def __init__(self, Sound):
        self.Sound = Sound 
    def play(self):
        freq = 44100    
        bitsize = -16   
        channels = 2    
        buffer = 1024   
        pygame.mixer.init(freq, bitsize, channels, buffer)
        pygame.mixer.music.set_volume(self.setVolume)
        pygame.mixer.music.load(self.Sound)
        pygame.mixer.music.play()
        self.status = STARTED 
    def stop(self):
        pygame.mixer.music.stop()
        self.status = FINISHED
    def setSound(self, music_file):
        self.Sound = music_file
    def busy(self):
        return pygame.mixer.music.get_busy()

On most people's computers, this is running absolutely fine, but on one participant's computer (using Windows 10), there is about 0.6s latency. I don't have the problem on my own machine (MacOS Sierra).

Is there anything he can do to lose the latency? He's tried reducing the buffer size, but this doesn't seem to have any effect.

Here is an example of the class being used in a minimal reproducible example.:

import pygame
from psychopy import core

class midiPlayer(object):
    setVolume = 1
    def __init__(self, Sound):
        self.Sound = Sound 
    def play(self):
        freq = 44100    
        bitsize = -16   
        channels = 2    
        buffer = 1024   
        pygame.mixer.init(freq, bitsize, channels, buffer)
        pygame.mixer.music.set_volume(self.setVolume)
        pygame.mixer.music.load(self.Sound)
        pygame.mixer.music.play()
    def stop(self):
        pygame.mixer.music.stop()
    def setSound(self, music_file):
        self.Sound = music_file
    def busy(self):
        return pygame.mixer.music.get_busy()

# You can download the MIDI file at https://wetransfer.com/downloads/868799aa1aacf7361de9a47d3218d2ee20200818095737/c3ccee ; obviously you'll need to update the file location below to wherever the file is saved on your own computer: 
MIDIFILE = midiPlayer("/Users/sam/Downloads/sounds/A4_version_2_gentlemen_normal.mid")
trialClock = core.Clock()
started_playing = 0
text_printed = 0 
continueRoutine = True

trialClock.reset()
while continueRoutine: 
    t = trialClock.getTime()
    if started_playing == 0:
        MIDIFILE.play()
        started_playing = 1
        
    if t >= 4.2 and text_printed ==0:
        print 'now'
        text_printed = 1
        # this should appear on the 8th note of the tune

    if t >= 6:
        MIDIFILE.stop()
        continueRoutine = False

Are you calling `pygame.mixer.init` anywhere else in your code? if not it doesn't make sense to be reinitialising the `pygame.mixer` every time you play a sound, since the init parameters never change. Similarly, are you ever changing the volume with `pygame.mixer.music.set_volume` elsewhere in your code? It seems to me that these `pygame.mixer` methods are best suited to exist outside your class, in some application init method. — Mark McElroy, Aug 14 '20 at 10:25
Thanks Mark! I don't change the volume at any other point and only ever call `pygame.mixer.init` whenever the midiPlayer class is called. So I guess that means that I should take them out of the class! From what the participant tells me, this saves 0.3s in latency. There's an extra 0.3s of latency to account for still though.. any thoughts on other things I could do to remove this? — Sam Leak, Aug 14 '20 at 10:44
@SamLeak Maybe you're loading the sounds every time instead of loading them all at application startup? It's hard to tell what the problem is without a [mcve]. — Ted Klein Bergman, Aug 14 '20 at 12:01
Thanks @TedKleinBergman I've updated the question to attempt to do that. Does that help? — Sam Leak, Aug 14 '20 at 14:08
@SamLeak No, you're still missing a lot of information. Read how to create a [mcve]. To sum up, you need to provide a full example that we can run so that we can verify your problem, but **not** just a code dump of your entire application. It should be as small of an example as possible. You can do that by stripping out unnecessary stuff (for example, drawing to the window doesn't need to be included as it's probably not the reason for the latency) until the program is small enough, or copy over the problematic code to a new file until it produces the problem. — Ted Klein Bergman, Aug 14 '20 at 14:17
@TedKleinBergman OK, I'll have a go at that when I have a moment. Two problems that come to mind are: 1) I don't have the latency problem on my system, it's only happening on this single participant's system (he's on Windows 10, whereas I'm on MacOS Sierra). I can send him the minimal reproducible example to see if he still has the problem with it, but there's no saying that it would be reproducible on your system (and it definitely wouldn't be on mine). 2) For it to be reproducible then I'd need to upload a few MIDI files to the post - is that possible? Thanks for your help. — Sam Leak, Aug 14 '20 at 14:33
@SamLeak Okay, editing your question and stating that it's an issue on Windows might help people answer your question (am also running MacOS so I won't be able to help). Uploading MIDI files is not ideal, but if you have no other options then that's what you have to do. However, if you don't have access to the machine where the problem is, creating a [mcve] will be very difficult. Maybe it's best to just put emphasis that the problem is with Windows and hope someone have had the same problem — Ted Klein Bergman, Aug 14 '20 at 14:40
@SamLeak By the way, how long is the MIDI tracks on average? Is there are reason you use the music mixer (which streams music) instead of creating [`pygame.mixer.Sound`](https://www.pygame.org/docs/ref/mixer.html#pygame.mixer.Sound) objects? If they're short, it might be better to use the latter. — Ted Klein Bergman, Aug 14 '20 at 14:45
Thanks @TedKleinBergman - I'll follow your advice. There isn't a reason for that me not using pygame.mixer.Sound objects - I'll look into them! The midi files are around 1m30s each and on each round there are up to 24 files that are iterated through. — Sam Leak, Aug 14 '20 at 15:04
@TedKleinBergman I've produced a minimal reproducible version and updated to the original post to include it. 'now' should print on screen on the 8th note of the file (and on my computer it does). On the participant's screen however it prints during the 7th note, as the midi file appears to be playing 0.6s late. — Sam Leak, Aug 18 '20 at 10:04
A weird quirk he's found is that, in the minimal reproducible version, if you set t >= 4.2 to 0 then you would expect 'now' to print 0.6 seconds before the music plays, but apparently it actually prints when the music starts, whether it is set to 0 or to 0.6. Everything from then on is 0.6s out of sync though.. — Sam Leak, Aug 18 '20 at 11:17
@SamLeak There are some things that might be the issue. It could be that you're loading the sound in the `play` method. He might be using a slow hard drive that takes long time to load in the file (although, I think this is very unlikely). Try putting it in the constructor instead. Also, move the mixer initialization `pygame.mixer.init(freq, bitsize, channels, buffer)` so it runs just once at startup. You usually put it right after the `import pygame`. — Ted Klein Bergman, Aug 18 '20 at 11:52
Also, it looks like you're actually using Python 2, is that correct? Python 2 is official no longer supported and should no longer be used. If possible, try upgrade to Python 3. — Ted Klein Bergman, Aug 18 '20 at 11:52
Thanks for that @TedKleinBergman . Moving `pygame.mixer.init` and `pygame.mixer.music.set_volume` out of the class (which @mark-mcelroy has also suggested) has reduced the latency by 0.3s. I tried loading the sound within the class but outside of the play method (I presume that's what you meant?) and sadly that didn't have an effect. Any thoughts on the final 0.3s to lose? You mentioned using `pygame.mixer.Sound` - does this support MIDI files? Sadly I can't update it to Python 3 as the whole experiment is written in Python 2 and it's already underway... Thanks for the help! — Sam Leak, Aug 18 '20 at 15:08
@SamLeak It should support MIDI-files. Load with [`sound = pygame.mixer.Sound(filename)`](https://www.pygame.org/docs/ref/mixer.html#pygame.mixer.Sound) and call with `sound.play()`. That might help. — Ted Klein Bergman, Aug 18 '20 at 15:42
Btw, do you have a window open? If you call `pygame.init()`, then it seems like the mixer will be initialized, and any call to `pygame.mixer.init` won't do anything. Try to call [`pygame.mixer.pre_init`](https://www.pygame.org/docs/ref/mixer.html#pygame.mixer.pre_init) **first** in your program, and then call the `pygame.init`. Then you might be able to reduce the delay by reducing the buffer size (which is quite high in earlier versions of pygame). Which pygame version do you use? — Ted Klein Bergman, Aug 18 '20 at 15:43
@TedKleinBergman Thanks a lot for this. When I try loading `sound = pygame.mixer.Sound(filename)` (using the actual filename obviously) I get a `MemoryError`. I tried calling `pygame.mixer.pre_init(freq, bitsize, channels, buffer)` before calling `pygame.mixer.init()`. My code doesn't include `pygame.init()` anywhere - were you suggesting I use that instead of `pygame.mixer.init()`? We've tried reducing the buffer size, but it doesn't seem to have an effect. One thing that's a little odd is that if `if t >= 4.2` is updated to `if t >= 0` the 'now' prints on screen exactly when the file starts — Sam Leak, Aug 19 '20 at 09:42
(which is odd because the 'now' also prints on screen exactly when the file starts if `if t >= 4.2` is updated to `t >= 0.6`, so this is when the latency seems to begin? The remaining 0.6s increments (1.2, 1.8 etc) are all 0.6s latent. Surely, if consistent, the latency should be such that, if set to `if t >= 0`, 'now' prints on screen ahead of the file playing? — Sam Leak, Aug 19 '20 at 09:44
@SamLeak Is the code in the question the same your testing on? How do you check the latency? — Ted Klein Bergman, Aug 19 '20 at 10:58
@TedKleinBergman yes it's happening with the code in the minimal reproducible example. As it's not happening on my own computer, I'm having the participant check it on his own computer. He's making a musical judgement-each note lasts for 0.6s and the 'now' is appearing one note too early for him (meaning that the Midi is playing one note late). The changes we've discussed have managed to pull it down to 0.3s (again an estimate, so I guess it's now appearing slightly before the target note rather than exactly on the target note). Is there a more accurate way to check to the latency than that? — Sam Leak, Aug 19 '20 at 11:03
@SamLeak I was wondering if the delay might be an measurement error. But now I'm thinking it might be a rendering issue. Maybe the notes are rendered at a delay, but there are no actual delay in the sound? — Ted Klein Bergman, Aug 19 '20 at 11:27

Pygame Latency Issue

0 Answers0