I wrote a program to compute the similarity scores between a query and a target document. The sketch is like below:
type Dictionary struct {
Documents map[int][]string
Queries map[int][]string
}
type Similarity struct {
QID int
DocID int
Sim float64
}
func (dict * Dictionary) CalScore(qID, docID int) float64 {
query := dict.Queries[qID]
document := dict.Documents[docID]
score := calculation(query, document) // some counting and calculation
// for example: count how many words in query are also in document and so on,
// like tf-idf things.
return score
}
// Calculate the similarity scores for each group.
func SimWorker(index int, dict *Dictionary, simsList *[][]Similarity, wg *sync.WaitGroup) {
defer wg.Done()
for i, sim := range (*simsList)[index] {
// Retrieving words from Dictionary and compute, pretty time consuming.
(*simsList)[index][i].Sim = dict.CalScore(dict.Queries[sim.QID], dict.Documents[sim.DocID])
}
}
func main() {
dict := Dictionary{
// All data filled in.
}
simsList := [][]Similarity{
// Slice of groups of structs containing
// pairs of query id and doc id.
// All sims scores are 0.0 initially.
}
var wg sync.WaitGroup
for i := range simsList {
wg.Add(1)
go SimWorker(i, &dict, &simsList, &wg)
}
wg.Wait() // wait until all goroutines finish
// Next procedures to the simsList
}
Basically I have a slice of groups of query-doc id pairs, query ids within each group are the same while doc ids are all different. The procedure is pretty straightforward, I just get the strings from the dictionary, then compute the score applying some algorithms.
Firstly I did all these in sequence (not using goroutines) and it took several minutes to compute the scores for each group and several hours in total. Then I expected some speed improvements by introducing goroutines like above. I created a goroutine for each group since they access different parts in the Dictionary and [][]Similarity. But it turned out that the speed didn't improve and somewhat decreased a little bit (the number of goroutines I'm using is around 10). Why did this happen and how do I improve the program to actually speed up the computation?
