Without knowing the error returned, it's hard to say... My first though is your permissions on your file descriptor. I've seen similar issues before. First, you can get some more information about the failure if you take a look at the return of the ioctl:
#include <errno.h>
int main(int argc, char* argv[])
{
long ret;
int fd = fopen("/dev/aes", "r+");
ret = ioctl(fd, 0, 1);
if (ret < 0)
printf("ioctl failed. Return code: %d, meaning: %s\n", ret, strerror(errno));
fclose(fd);
}
Check the return values and this should help give you something to search on. Why check? See the bottom of the post...
Next in order to check if it is permissions issue, run the command:
ls -l /dev/aes
if you see something like:
crw------- 1 root root 10, 57 Aug 21 10:24 /dev/aes
Then just issue a:
sudo chmod 777 /dev/aes
And give it a try again. I bet it will work for you. (Note I ran that with root permissions since root is the owner of my version of your mod)
If the permissions are already OK, then I have a few more suggestions:
1) To me, the use of fopen/fclose is strange. You really only need to do:
int fd = open("/dev/aes");
close(fd);
My system doesn't even let your code compile as is.
2) Your IOCTL parameter list is old, I don't know what kernel version your compiling on, but recent kernels use this format:
long aes_ioctl(struct file *file, unsigned int ioctl_num, unsigned long ioctl_param){
Note the removal of the inode and the change of the return type. When I ran your code on my system, I made these changes.
Best of luck!
Note: Why should we check the return when we're "not getting into the ioctl"? Let me give you an example:
//Kernel Code:
//assume include files, other fops, exit, miscdev struct, etc. are present
long hello_ioctl(struct file *file, unsigned long ioctl_num, unsigned long ioctl_param) {
long ret = 0;
printk("in ioctl");
return ret;
}
static const struct file_operations hello_fops = {
owner: THIS_MODULE,
read: hello_read,
unlocked_ioctl: hello_ioctl,
};
static int __init hello_init(void) {
int ret;
printk("hello!\n");
ret = misc_register(&hello_dev); //assume it worked...
return ret;
}
User space code:
//assume includes
void main() {
int fd;
long ret;
fd = open("/dev/hello");
if(fd) {
c = ioctl(fd, 0, 1);
if (c < 0)
printf("error: %d, errno: %d, meaning: %s\n", c, errno, strerror(errno));
close(fd);
}
return;
}
So what's the output? Lets assume bad file permissions on /dev/hello (meaning our user space program can't access /dev/hello).
The dmesg | tail shows:
[ 2388.051660] Hello!
So it looks like we didn't "get in to" the ioctl. What's the output from the program?
error: -1, errno: 9, meaning: Bad file descriptor
Lots of useful output. Clearly the ioctl call did something, just not what we wanted. Now changing the permissions and re-running we can see the new dmesg:
[ 2388.051660] Hello!
[ 2625.025339] in ioctl
