Failure while unicast data from kernel to user space via netlink

Question

I am new to kernel development and have a problem in uni casting the data from the kernel space to user space over netlink sockets. send_nat() function is to be called from a kernel module to write a user defined struct nat_mntr to netlink socket. But nlmsg_unicast() failed all the time even I tried different configuration in cfg. Kindly help me find out what is wrong in my code.

userdefined.c

    int no_data_request = 1;
    EXPORT_SYMBOL(no_data_request);
    int request_pid = 0; // PID of requesting process
    EXPORT_SYMBOL(request_pid);

void send_nat(struct sock *nl_sk, struct nat_mntr *nat_data, int pid, int group, gfp_t flags, int *sock_closed){

        struct nlmsghdr *nlh;
        struct sk_buff *skb_out;
        int msg_size;
        int res = 200;
        #define MYPROTO 31

        printk(KERN_DEBUG "%s: Entered \n", __FUNCTION__);
        if ( nat_data == NULL ){
                printk(KERN_DEBUG "%s: nat_data is NULL: Leaving \n", __FUNCTION__);
                return ;
        }else {
                printk(KERN_DEBUG "%s: nat_data is filled \n", __FUNCTION__);
        }

        if (nl_sk == NULL) {

                printk(KERN_DEBUG "%s: nl_sk is NULL  \n", __FUNCTION__);
        }else
                printk(KERN_DEBUG "%s: nl_sock is not null \n", __FUNCTION__);

        if (*sock_closed == 1) {

                printk(KERN_DEBUG "%s: sock_closed == 1, creating socket \n", __FUNCTION__);
                struct netlink_kernel_cfg cfg  = {
                        .groups         = 1,
                        .input          = rr,
                };
                nl_sk  = netlink_kernel_create(&init_net, MYPROTO, &cfg);

                if (!nl_sk) {
                        printk(KERN_DEBUG "%s: Error creating socket: sock_closed = %d:  Leaving  \n",  __FUNCTION__ , *sock_closed);
                        return ;
                }
                else {
                        *sock_closed = 0;
                        printk(KERN_DEBUG "%s: Socket created successfully: sock_closed = %d  \n",  __FUNCTION__ , *sock_closed );
                }
        }
        else if (*sock_closed == 0 ){
                printk(KERN_DEBUG "%s:Already created socket.  sock_closed = 0 \n",  __FUNCTION__);
        }
        else {
                printk(KERN_DEBUG "%s: sock_closed status is unknown: sock_closed = %d Leaving  \n",  __FUNCTION__ , *sock_closed);
                return ;
        }

        if( no_data_request){
                printk(KERN_DEBUG "%s: No one has requested data: Leaving  \n",  __FUNCTION__);
                return ;
        }else{
                printk(KERN_DEBUG "%s: Process %d requested the data\n",  __FUNCTION__, request_pid );
        }
        msg_size = sizeof(struct nat_mntr);
        skb_out = nlmsg_new(msg_size, 0);

        if ( !skb_out ) {
                printk(KERN_DEBUG "%s: Failed to skb_out = nlmsg_new(msg_size, 0): Leaving \n",  __FUNCTION__);
                return;
        }

        nlh = nlmsg_put(skb_out, 0, 0, NLMSG_DONE, msg_size, 0); /* NLMSG_DONE */
        NETLINK_CB(skb_out).dst_group = 0; /* not in mcast group */

        if (!nlh) {
                printk(KERN_DEBUG "%s: Failed nlh = nlmsg_put(skb_out, 0, 1,  NLMSG_DONE, msg_size, 0): Leaving  \n",  __FUNCTION__);
                return ;
        }
        else {
                printk(KERN_DEBUG "%s: Successfull nlh = nlmsg_put(skb_out, 0, 0,  NLMSG_DONE, msg_size, 0)  \n",  __FUNCTION__);
        }

        if(memcpy(nlmsg_data(nlh), nat_data , sizeof(nat_data) ) == NULL ) {
                printk(KERN_DEBUG "%s: Failed to memcpy(nlmsg_data(nlh), nat_data , sizeof(struct nat_mntr)) Leaving  \n",  __FUNCTION__);
                return ;
        }
        res =  nlmsg_unicast(nl_sk, skb_out2, request_pid);
        if (res < 0 ){
                printk(KERN_DEBUG "%s: Failed to  nlmsg_unicast(nl_sk, skb_out, request_pid): Leaving \n",  __FUNCTION__);
                return ;
        }
        printk(KERN_DEBUG "%s: Data sent successfully : Leaving \n",  __FUNCTION__);
}


// Callback of kernel socket. 
void rr(struct sk_buff *skb){
        printk(KERN_DEBUG "%s: Entered \n", __FUNCTION__);
        struct nlmsghdr *nlh;
        nlh = (struct nlmsghdr *)skb->data;
        printk(KERN_DEBUG "Request received \n");
        request_pid = nlh->nlmsg_pid; /* pid of sending process */
        no_data_request = 0; // Someone is out there
        printk(KERN_DEBUG "%s: Leaving:\n", __FUNCTION__);
}

kernel_module.c

#define NAT_GROUP 21
struct sock *nl_sk_ud = NULL;
EXPORT_SYMBOL(nl_sk_ud);
int sock_closed = 1;
EXPORT_SYMBOL(sock_closed);
struct nat_mntr *data = NULL;
EXPORT_SYMBOL(data);

any_kernel_function(){

....

data = get_info(skb, 0, l3proto, l4proto, &target, mtype); // Returns pointer to struct nat_mntr
send_nat(nl_sk_ud, data,  0, NAT_GROUP, 0, &sock_closed);

....

}

I think I have the answer. However, it can be wrong because I don't know what your userspace client does. Could you please post the userspace code? — Yd Ahhrk, Dec 04 '15 at 18:23
User space application just receive the **struct nat_mntr** from socket and display its contents in user space. I followed [How to use netlink socket to communicate with a kernel module? question.] (http://stackoverflow.com/questions/3299386/how-to-use-netlink-socket-to-communicate-with-a-kernel-module?rq=1) — Mazhar, Dec 05 '15 at 08:01

score 1 · Accepted Answer · answered Dec 09 '15 at 00:14

If your kernel module writes an answer, then the userspace request will receive two responses: an ACK (crafted automatically by the kernel) and the actual response.

I think people don't notice this because normally the kernel module answers quickly - before the ACK. So the userspace client receives the answer and ignores whatever comes next (which includes the ACK) until the next request.

In your code, the kernel module doesn't answer immediately. It waits until data is available and it fetches it later. This probably happens:

Userspace client sends a request.
Kernel module stores the pid.
Linux answers ACK to the client.
Client receives the answer, incorrectly parses it as a bogus nat_mntr instead of an ACK, and closes the socket.
Kernel module sends the answer once it has the data. nlmsg_unicast() returns error code -111 because the client is no longer listening.

One way to solve this is by having the client expect two packets - and ignore the first one, the ACK.

BTW: That's not the only problem with your code.

When you do this

nl_sk = netlink_kernel_create(&init_net, MYPROTO, &cfg);

you're assigning the socket to a local variable. If you call the function again, nl_sk will be uninitialized even if sock_closed is 0.

You're never deallocating the socket. Because you can't open two sockets in kernelspace on the same protocol, subsequent socket creations will irremediably fail until you reboot. (this will affect you if you need to recompile, for example.)
The code is racy. At the very least, no_data_request and request_pid should probably be atomic integers.
Don't do this:

nlh = (struct nlmsghdr *)skb->data;

This is better:

nlh = nlmsg_hdr(skb);

And this is even better (because it does some validations and Netlink paperwork for you):

netlink_rcv_skb(skb, &rr2);

Thanks a lot @Yd for ans. Let me check modifications you suggested. BTW These variables are not local, as my actual code look like this. I thought netlink kernel socket is independent of netlink client socket i.e. it doesn't matter if someone is listening or not, we can write to kernel side socket. please correct me if I am wrong. How to make client receive more than one replies i.e. it should receive till we don't stop it? — Mazhar, Dec 09 '15 at 08:04
@Mazhar From my experience, a Netlink socket really does expect somebody to be listening - even when it's multicasting a message, which is really weird. You can always ignore the result code though, if that makes sense. — Yd Ahhrk, Dec 09 '15 at 20:51
As for your second question: libnl-3's documentation uses a `while (1)` to keep the client listening forever: infradead.org/~tgr/libnl/doc/core.html#_multicast_example. I don't think that's the right thing to do, though. What I do is use the `NLMSG_DONE` flag - the client listens to messages until the flag arrives. Once that happens, it closes the socket. (libnl-3 doesn't do this for you, though; you have to do it manually.) — Yd Ahhrk, Dec 09 '15 at 21:01

Failure while unicast data from kernel to user space via netlink

1 Answers1

Linked