0

Trying to solve the problem of reading phone numbers from a file (line by line) and hash them concurrently (md5) in go with a maximum of 100 concurrent process maximum, creating a fixed number of goroutines. tried using go pipeline and bounded parallelism but did not work. any guidance, please?

this is what I tried, https://play.golang.org/p/vp7s512l8w4

Jau L
  • 904
  • 2
  • 8
  • 20
  • Show what you tried. – Charlie Tumahai May 14 '18 at 02:34
  • this is what I tried, not sure if that how it should be done, https://play.golang.org/p/vp7s512l8w4 – Jau L May 14 '18 at 02:44
  • See https://stackoverflow.com/questions/18405023/how-would-you-define-a-pool-of-goroutines-to-be-executed-at-once-in-golang – Charlie Tumahai May 14 '18 at 02:57
  • thanks, the only part I'm confused about is when reading the content from a file since I have to read line by line. how would you apply it to the example above, tried many times did not work. – Jau L May 14 '18 at 03:00
  • https://play.golang.org/p/ZRYAr_OZK6d maybe this is what you want – Ksir May 14 '18 at 03:15
  • Thanks, this is exactly what I'm looking for. however, it keeps waiting after its done and throws an error "fatal error: all goroutines are asleep - deadlock!" this is what I tried https://play.golang.org/p/08Y3Ar0B4Tg – Jau L May 14 '18 at 03:24
  • oh maybe this is why :-) https://play.golang.org/p/vhIXiwDi9BC – Jau L May 14 '18 at 03:29

1 Answers1

2
package main

import (
    "bufio"
    "fmt"
    "os"
    "sync"
)

func hashPhoneNumers(phoneNO string, ch chan struct{}, group *sync.WaitGroup) {
    do_hash(phoneNO)
    <-ch
    group.Done()
}

func do_hash(s string) {
    fmt.Println("do hash: ", s)
}

func main() {
    c := make(chan struct{}, 100)
    file, err := os.Open("file.txt")
    if err != nil {
        fmt.Println(err)
        os.Exit(-1)
    }
    defer file.Close()
    line := bufio.NewScanner(file)

    wg := sync.WaitGroup{}
    for line.Scan() {
        // with a maximum of 100 concurrent process maximum
        c <- struct{}{}
        wg.Add(1)
        go hashPhoneNumers(line.Text(), c, &wg)
    }

    // wait all goroutines done
    wg.Wait()
    close(c)
}

this works fine

Ksir
  • 396
  • 2
  • 5
  • Thanks a lot. question, if do_hash is a heavy operation, and there are 100 workers, is it going to wait until one at least is released? – Jau L May 14 '18 at 03:41
  • wait until all the 100 worker goroutines finish theirs work and quit – Ksir May 14 '18 at 03:46
  • nice, so what exactly the purpose of c <- struct{}{} in this context? – Jau L May 15 '18 at 19:52
  • like a counter, c <- struct{}{} push an item into the chan buffer, and <- ch consume an item, This mechanism is used to restrict the maximum number of goroutines – Ksir May 16 '18 at 08:01