5

I have a socket server using IO::Socket::Async and Redis::Async for message publishing. Whenever there is a message received by the server, the script would translate the message and generate acknowledge message to be sent back to the sender so that the sender would send subsequent messages. Since translating the message is quite expensive, the script would run that portion under a 'start' method. However, I noticed that the Moar process eating my RAM as the script is running. Any thought where should I look to solve this issue? Thanks!

https://pastebin.com/ySsQsMFH

use v6;
use Data::Dump;
use experimental :pack;
use JSON::Tiny;
use Redis::Async;

constant $SOCKET_PORT = 7000;
constant $SOCKET_ADDR = '0.0.0.0';
constant $REDIS_PORT = 6379;
constant $REDIS_ADDR = '127.0.0.1';
constant $REDIS_AUTH = 'xxxxxxxx';


constant $IDLING_PERIOD_MIN = 180 - 2; # 3 minutes - 2 secs
constant $CACHE_EXPIRE_IN = 86400; # 24h hours

# create socket
my $socket = IO::Socket::Async.listen($SOCKET_ADDR, $SOCKET_PORT);

# connnect to Redis ...
my $redis;
try {
    my $error-code = "110";
    $redis = Redis::Async.new("$SOCKET_ADDR:$SOCKET_PORT");
    $redis.auth($REDIS_AUTH);

    CATCH {
        default {
            say "Error $error-code ", .^name, ': Failed to initiate connection to Redis';
            exit;
        }
    }
}

# react whenever there is connection
react {
    whenever $socket -> $conn {

        # do something when the connection wants to talk
        whenever $conn.Supply(:bin) {
            # only process if data length is either 108 or 116
            if $_.decode('utf-8').chars == 108 or $_.decode('utf-8').chars == 116 {
                say "Received --> "~$_.decode('utf-8');
                my $ack = generateAck($_.decode('utf-8'));  # generate ack based on received data
                if $ack {
                    $conn.print: $ack;
                }else{
                    say "No ack. Received data maybe corrupted. Closing connection";
                    $conn.close;
                }

            }
        }
    }
    CATCH {
        default {
            say .^name, ': ', .Str;
            say "handled in $?LINE";
        }
    }
}

### other subroutines down here ###
Zarul Zakuan
  • 510
  • 3
  • 13
  • Can we see the code in the generateAck() function? I'm wondering if there's something being created in a thread that's not being cleaned up. – Scimon Proctor Dec 04 '18 at 09:15
  • What are you doing with the `$redis` object you create at the top of the script? And is the memory loss happening when the script is idle (nothing connecting to it) or only when it's getting connections? – Scimon Proctor Dec 04 '18 at 09:18
  • I use redis to store and publish the translated message. It's happening when it's getting connections – Zarul Zakuan Dec 04 '18 at 09:21
  • generateAck is quite big. this will take a while. hold on. anyway, thank you so much for helping me out – Zarul Zakuan Dec 04 '18 at 09:21
  • Also... I can see it closing the connection if the received data isn't 108 or 116 chars long but not anywhere else. I've not played about with IO::Socket::Async enough to know but that looks like it's a potential memory leak there? If it's holding connections forever. I'll need to check the docs. – Scimon Proctor Dec 04 '18 at 09:26
  • https://pastebin.com/qd0jLj21 the whole code. – Zarul Zakuan Dec 04 '18 at 09:27
  • I don't want to close the connection everytime the server sends out ack message, as the messages are coming continuously whenever the source receives the ack message. – Zarul Zakuan Dec 04 '18 at 09:28
  • so I think this line : `await Promise.anyof($translated, Promise.in(1));` doesn't do what you think. The `$translated` promise will still be running in a thread. – Scimon Proctor Dec 04 '18 at 09:41
  • why is that? I thought after 1 second if the thread still running, it should be closed down? https://docs.perl6.org/type/Promise#method_anyof – Zarul Zakuan Dec 04 '18 at 09:59
  • Try the following : `my $a = start { sleep 2;say "a"; };await Promise.anyof( $a, Promise.in(1) );say "Done";sleep 2` Note that the await anyof returns after the first Promise completes but the "a" is still printed. You need to break the `$translated` promise to stop it running. – Scimon Proctor Dec 04 '18 at 10:20
  • I don't think that's it. Even if I let that block of code to run synchronized, it still eating my ram – Zarul Zakuan Dec 04 '18 at 13:36
  • Is this the same machine you've mentioned elsewhere, that is, 512 MBytes memory? – jjmerelo Dec 04 '18 at 16:47
  • yes @jjmerelo the same one – Zarul Zakuan Dec 04 '18 at 19:28
  • 1
    i commented out all the Redis operations and memory usage has been stabilized. So I guess it's the Redis::Async module.. ugh.. – Zarul Zakuan Dec 05 '18 at 02:30
  • 3
    Please check if rakudo 2018.11 still has that problem. I fixed something just before the release that prevented memory growth of up to about 2 gigabytes when the program wasn't doing anything if a ThreadPoolScheduler has been created at some point. Also, please try putting `start { loop { Array.new(1, 2, 3) } }` somewhere, if that prevents memory bloat, then it's most probably the thing i'm talking about – timotimo Dec 05 '18 at 02:51
  • I have 512MB of RAM :-) I'm 90% confident Redis::Async is the culprit here. I have moved all the redis calls under one sub and make sure it creates one object for each connection made. But still, although the ballooning rate is slower, it's still eating my ram.. gosh.. – Zarul Zakuan Dec 05 '18 at 08:58
  • @ZarulZakuan Have you tried what timotimo suggested? Have you used `--profile` (i.e. `perl6 --profile ...`)? – raiph Dec 05 '18 at 15:04
  • @raiph I only have half gig ram so i will never reach 2gb to test that.. you mean --profile=heap? yes I did that against my code, but I don't know how to analyze the dump – Zarul Zakuan Dec 05 '18 at 23:29
  • @ZarulZakuan I meant tried the latest Rakudo 2018.11 as timotimo says in his comment. timotimo will likely help you with analyzing the dump if you do as he suggests in his comment. – raiph Dec 06 '18 at 04:45
  • ok I will try but not at the moment. still experimenting with other codes and compiling a new rakudo will take some time. or is there a way that I can upgrade it without having to reinstall? – Zarul Zakuan Dec 06 '18 at 04:51
  • @timotimo unfortunately the installation failed. my cpu spiked up to 100% during nqp compilation and it just died with "No registered operation handler for 'writeuint'" – Zarul Zakuan Dec 06 '18 at 06:13
  • i think i had enough. I'm going to use inline perl5 and use the Redis module instead. – Zarul Zakuan Dec 06 '18 at 06:14
  • 1
    @ZarulZakuan the error message you got is most likely a version disparity between rakudo and nqp, so maybe some other build step failed, or the `--prefix` didn't match, or something different. Anyway, for now I wish you the best of luck with `Inline::Perl5`, which is quite an amazing module. – timotimo Dec 06 '18 at 10:00
  • 1
    @timotimo thank you so much for your help. I really appreciate that. For now I think its much safer to use p5 module until p6 has reputable redis module. – Zarul Zakuan Dec 06 '18 at 10:04

1 Answers1

1

The issue was using the Async::Redis. Jonathon Stowe had fixed the Redis module so I'm using Redis module with no issue.

Zarul Zakuan
  • 510
  • 3
  • 13