Go from azure text to speach stream to a discord audio stream

Question

I am creating a .net core 3.1 application that uses text to speech from azure on discord. But the problem that I am having is the convertion from the audio stream. I am using NAudio to convert, but it isn't really working. Converting mp3 to discord works perfectly...

I am using Discord.net to do it.

private async Task SpeakAsync(IAudioClient client, string text)
{
    var azureAudio = await new AzureSpeechService().Speak(text);

    using var reader = new WaveFileReader(azureAudio);
    var nAudio = WaveFormatConversionStream.CreatePcmStream(reader);

    Create an empty discord audio stream
    AudioStream audioStream = client.CreatePCMStream(AudioApplication.Voice, 32000);
    await nAudio.CopyToAsync(audioStream, 50);

    await using var discord = client.CreatePCMStream(AudioApplication.Voice);
    try { await stream.CopyToAsync(audioStream); }
    finally { await discord.FlushAsync(); }
}

The azure stream creation code:

public async Task<Stream> Speak(string text)
{
    using var synthesizer = new SpeechSynthesizer(_config, null);
    using var result = await synthesizer.SpeakTextAsync(text);
    switch (result.Reason)
    {
        case ResultReason.SynthesizingAudioCompleted:
            Console.WriteLine($"Speech synthesized to speaker for text [{text}]");
            return new MemoryStream(result.AudioData);
        case ResultReason.Canceled:
        {
            var cancellation = SpeechSynthesisCancellationDetails.FromResult(result);
            Console.WriteLine($"CANCELED: Reason={cancellation.Reason}");

            if (cancellation.Reason == CancellationReason.Error)
            {
                Console.WriteLine($"CANCELED: ErrorCode={cancellation.ErrorCode}");
                Console.WriteLine($"CANCELED: ErrorDetails=[{cancellation.ErrorDetails}]");
                Console.WriteLine("CANCELED: Did you update the subscription info?");
            }

            return null;
        }
        default:
            return null;
    }
}

I am trying to not use the ffmpeg convertor, so I don't need to install extra stuff on the server.