I have the following code, which I was working on Linux, which worked perfectly.
It would read a DB, load it in memory with SQLite3 and then ask all the cores to select and see if they find such information on the DB(read-only).
If they did or didn't, then the next value would be selected through multiprocessing.lock so that each core/thread would have a unique value(make sure that no two threads ask the same question).
In total, the python script would use up around 3-4GB of RAM and all threads (16) on Linux(Ubuntu).
On windows, the same code loads up 16 different DBs (taking 50+GB of ram), and it doesn't respect the multiprocessing.lock value.
Below is the code, which I have simplified but tested and still works the same way.
Do you know how I can achieve the same result in windows?
Code and outputs below:
import multiprocessing.shared_memory
import sqlite3
import time
threadLock = multiprocessing.Lock()
firstValueer = multiprocessing.shared_memory.ShareableList(['5'])
DATABASE = r'DB.sql'
t0 = time.time()
db_path = r'DB.db'
source = sqlite3.connect(db_path)
connection= sqlite3.connect("file::memory:?cache=shared", uri=True,check_same_thread=False)
source.backup(connection)
connected = connection.cursor()
t01 = time.time()
total = t01 - t0
print("loading DB:", total)
def select(varA,varB,varC,varD):
connected.execute("SELECT 1 FROM tableA where var in (?,?,?,?)" ,(varA,varB,varC,varD))
data = connected.fetchone()
if data is None:
value = 0
else:
value = 1
return value
def getNextValue():
global firstValueer
with threadLock:
tmpint = int(firstValueer[0])
tmpint += 1
firstValueer[0] = str(tmpint)
return int(firstValueer[0])
def main(firstValue):
while firstValue < 100:
print (firstValue)
firstValue = getNextValue()
t1 = time.time()
total = t1 - t0
print("with CPU:", total)
if __name__ == '__main__':
for cpu in range(multiprocessing.cpu_count()):
multiprocessing.Process(target = main, args = (getNextValue(), )).start()
Windows output(PyCharm)
loading DB: 0.001062154769897461
loading DB: 0.0016663074493408203
6
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
with CPU: 0.0048923492431640625
loading DB: 0.0010585784912109375
7
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
loading DB:71
0.0005996227264404297
72
73
74
75
76
77
78
79
9
80
81
682
83
7
84
85
8
86
987
88
10
89
11
90
12
91
92
13
93
94
14
95
15
16
96
17
97
18
98
19
99
20
with CPU: 0.00489377975463867221
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
with CPU: 0.004536867141723633
loading DB: 0.0010526180267333984
8
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
with CPU: 0.004870176315307617
loading DB: 0.0011360645294189453
11
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
with CPU: 0.0038933753967285156
loading DB: 0.0016775131225585938
10
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
with CPU: 0.004441499710083008
loading DB: 0.0
13
loading DB: 0.00109124183654785166
7
8
9
10
12
11
6
7
812
13
9
14
10
15
11
16
12
17
13
14
18
15
1619
20
21
17
22
1823
19
2420
2125
22
26
23
27
28
29
30
31
32
33
2434
35
25
36
26
37
27
38
28
39
29
40
30
41
31
42
32
43
33
44
34
45
35
46
47
4836
49
37
50
38
51
39
52
40
53
41
54
42
55
43
5644
5745
58
46
59
47
60
48
61
49
62
50
63
51
64
65
52
66
5367
54
68
55
69
70
56
71
57
72
58
73
59
74
60
75
61
76
62
77
63
78
64
79
65
80
66
81
67
82
68
83
69
84
70
85
71
86
72
87
73
88
74
89
75
90
76
91
77
92
78
93
79
94
80
9581
82
83
96
84
97
85
98
86
87
99
88
with CPU:89
0.00487518310546875
90
91
92
93
94
95
96
97
98
99
with CPU: 0.00487518310546875
loading DB: 0.0010840892791748047
15
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
with CPU: 0.0032472610473632812
loading DB: 0.0
14
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
with CPU: 0.0027418136596679688
loading DB: 0.0011019706726074219
17
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
with CPU: 0.003863811492919922
loading DB: 0.0010428428649902344
16
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
with CPU: 0.0026226043701171875
loading DB: 0.0
19
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
loading DB: 0.001070261001586914
85
86
87
88
89
90
91
18
92
93
94
6
95
796
97
8
98
999
with CPU: 10
0.002686738967895508
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
with CPU: 0.003235340118408203
loading DB: 0.0010516643524169922
20
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
with CPU: 0.0020956993103027344
loading DB: 0.0010623931884765625
21
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
with CPU: 0.0027017593383789062
Process finished with exit code 0
Linux output:
loading DB: 0.0004832744598388672
6
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
7
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
with CPU: 0.0084991455078125
with CPU: 0.011128902435302734
with CPU: 0.012439250946044922
82
with CPU: 0.012716293334960938
8
with CPU: 0.013614892959594727
Process finished with exit code 0
