Spoofing the source ip and port of a udp packet among local applications

Question

I have a whitelist that stands between my server and my clients. I want my clients to connect to my server through my whitelist. The point of my whitelist is to ignore any UDP packets that does not have a whitelisted source. If they are in the whitelist, I want to pass the udp packets with their original source ip and source port to my server which is also hosted on the same machine. However when I do, the source ip remains as localhost, I'm able to spoof the port however the ip still remains the same.

My whitelist listens for udp4 messages, creates a new packet from scratch when it receives a message with the source and port of the remote client that just messaged it, and finally attaches the message it just received before passing it onto the server, since the packet is being created from scratch I'd expect the ip and port to be properly set (i.e. spoofed!). However from what I can gather the ip still remains as 127.0.0.1. Below you'll see my dgram listener, packet creation and sending with the raw-socket library.

//Udp proxy
const dgram = require('dgram');
const client = dgram.createSocket('udp4');
const raw = require ("raw-socket");
const socket = raw.createSocket({ protocol: 17,addressFamily:1 });

client.on('message', function(message, remote) {
    const whiteListedClientPort = remote.port;
    const whiteListedClientIP = remote.address;
    console.log(`Received ${message.length} bytes from ${remote.address}:${remote.port}`);
    // Create the UDP packet buffer
    const packet = Buffer.alloc(20 + message.length);
    packet.writeUInt16BE(whiteListedClientPort, 0); // Source port
    packet.writeUInt16BE(targetPort, 2); // Destination port
    packet.writeUInt16BE(8 + message.length, 4); // Length (UDP header + message)
    packet.writeUInt16BE(0x0000, 6); // Checksum (disabled)
    const sourceIpParts = whiteListedClientIP.split('.').map(part => parseInt(part, 10));
    packet.writeUInt8(sourceIpParts[0], 12);
    packet.writeUInt8(sourceIpParts[1], 13);
    packet.writeUInt8(sourceIpParts[2], 14);
    packet.writeUInt8(sourceIpParts[3], 15);
    const destinationIpParts = targethostIp.split('.').map(part => parseInt(part, 10));
    packet.writeUInt8(destinationIpParts[0], 16);
    packet.writeUInt8(destinationIpParts[1], 17);
    packet.writeUInt8(destinationIpParts[2], 18);
    packet.writeUInt8(destinationIpParts[3], 19);
    message.copy(packet, 20); // Message data

    console.log(" source ip is ",whiteListedClientIP," source port is ",whiteListedClientPort)
    socket.send(packet, 0, packet.length, targethostIp, (error) => {
      if (error) {
        console.error(`Error sending IPv4 packet: ${error.message}`);
      } else {
        console.log(`Sent packet to ${targethostIp} with a length of ${packet.length}`);
      }
  });
});
client.on('error', function (err) {
    console.log("something went wrong!")
    this.close();
})
client.bind(3000);

I've also set up a listener to check my work, this basically outputs whatever it gets from my whitelist app.

var dgram = require('dgram');
var s = dgram.createSocket('udp4');
s.on('message', function(msg, rinfo) {
  console.log('I got this message: ' + msg.toString());
  console.log(rinfo)
});
s.bind(3002);
console.log("listening");

Below is one of the sample outputs, in this case I was able to spoof the source port but not the source ip which still remains as localhost.

{ address: '127.0.0.1', family: 'IPv4', port: 3169, size: 14 }

I'm completely stuck on this so I'd appreciate any input!