discord.js - bot losing packets in Voice Channel at constant rate

Question

Normally I am able to solve my own issues, but this time I am unable to even find the core reason for the issue and I desperately need help.

When downloading streams and playing them back to voice channels in Discord, I noticed the bot's audio got glitchy/choppy roughly every 20-22 seconds. After days of testing, I figured it was all due to sent packet loss. However, I still am unable find the source of the issue.

What I checked:

• the code (I am 100% sure the issue isn't with my code because it was doing the same exact thing with two different bots operating differently)
• discord.js version (both v12 and v13 yielded same results)
• CPU, disk and RAM usage (nothing out of the ordinary, the server isn't stressed at all)
• opus/ffmpeg-static/fluent-ffmpeg package versions (all at latest versions)
• server speed using both ookla speedtest and speedtest npm package (both showed high speeds, so the issue isn't with connection speed at all)

Additional information:

• node.js: v14.18.0 (the bot works on my PC with node.js of this version, without any issues at all)
• discord.js: v13.2.0
• Ubuntu: 20.04.3 LTS

Discord's bot connection stats after playing over two minutes of music (top lines may have increased numbers because I had to stitch two screenshots together)

The code

Discord.js v13 voice connection code (simplified):

var connectToChannel = async function (channel) {
    const connection = joinVoiceChannel({
        channelId: channel.id,
        guildId: channel.guild.id,
        adapterCreator: channel.guild.voiceAdapterCreator,
    });
    connection.resource = null;
    connection.play = function(resource) {
        connection.on = function(eventType, cb) { player.player.on(eventType, cb); return connection; };
        connection.resource = createAudioResource(resource, {
            inlineVolume: true
        });

        player.player.play(connection.resource);
        connection.subscribe(player.player);
        return connection;
    }
    return connection;
};
// somewhere else...
const connection = connectToChannel(member.voice.channel).then(async connection => {
    connection.play(ytdl(yturl));
}

(I'm using discord-ytdl-core (which doesn't need highwatermark option), but I tried ytdl-core as well. I even tried loading a literal stream from a file and played it back, with exact same results (!!))

Discord.js v12 voice connection code (this time around playing a stream from a file; it doesn't matter where the stream came from, the results stays same.):

var voiceChannel = message.member.voice.channel;
voiceChannel.join().then(connection => {
    console.log("preparing to start");
    const dispatcher = connection.play('./_.ogg', { highWaterMark: 50 });
}).catch(err => console.log(err));

(Adjusting the highWaterMark option, even to 10MB, does not have any effect on the packet loss at all.)

If anyone would have any ideas as to what could be the issue or at least check further to find out the reason, I would be absolutely grateful.