i use txcouchbase to request data from couchbase

Question

i use couchbase txcouchbase to operator;and i get a question;my code is below:

1 import multiprocessing
2 from threading import Thread
3 from couchbase import Couchbase
4 from twisted.internet import reactor
5 import txcouchbase.connection as TxConnection
6 from txcouchbase.connection import Connection as TxCouchbase
7 import couchbase
8 db = None
9 class A:
10     def __init__(self):
11         pass
12 
13     def on_get(self,ret):
14         pass
15 
16 def worker(key):
17     '''
18     l = []
19     for i in range(1000000):
20         l.append(i)
21     '''
22     db = TxCouchbase(bucket = 'zonedb', 
23             host = '192.168.1.30', 
24             timeout = 100, 
25             lockmode = couchbase.connection.LOCKMODE_WAIT)
26     a = A()
27     while 1:
28         for k in key:
29             db.get(k, quiet = True).addCallback(a.on_get)
30 
31 if __name__ == "__main__":
32     TxConnection.experimental.enable()
33     keys = ['kuwo.cn', 'apple.com', \
34             'jslgroup.com', 'zjoldns.com.cn', \
35             'tmall.com', 'kaitao.cn', \
36             'alibaba.com', 'telemetryverification.net', \
37             'qq.com', 'baidu.com', 'pyrc.net', 'xh568.com']
38     p = Thread(target = worker, args = (keys,))
39     p.start()
40     reactor.run() 

and the web console show me that couchbase have three thousand opr/second;

and when i take out the notes

17     '''
18     l = []
19     for i in range(1000000):
20         l.append(i)
21     '''

the code is:

1 import multiprocessing
2 from threading import Thread
3 from couchbase import Couchbase
4 from twisted.internet import reactor
5 import txcouchbase.connection as TxConnection
6 from txcouchbase.connection import Connection as TxCouchbase
7 import couchbase
8 db = None
9 class A:
10     def __init__(self):
11         pass
12 
13     def on_get(self,ret):
14         pass
15 
16 def worker(key):
17 
18     l = []
19     for i in range(1000000):
20         l.append(i)
21
22     db = TxCouchbase(bucket = 'zonedb', 
23             host = '192.168.1.30', 
24             timeout = 100, 
25             lockmode = couchbase.connection.LOCKMODE_WAIT)
26     a = A()
27     while 1:
28         for k in key:
29             db.get(k, quiet = True).addCallback(a.on_get)
30 
31 if __name__ == "__main__":
32     TxConnection.experimental.enable()
33     keys = ['kuwo.cn', 'apple.com', \
34             'jslgroup.com', 'zjoldns.com.cn', \
35             'tmall.com', 'kaitao.cn', \
36             'alibaba.com', 'telemetryverification.net', \
37             'qq.com', 'baidu.com', 'pyrc.net', 'xh568.com']
38     p = Thread(target = worker, args = (keys,))
39     p.start()
40     reactor.run() 

so the code have a big size list variable;and couchbase web console me couchbase has zero opr/second; and why is it ;how can i to do; i thank it twisted's wrong but i don't know how to resolve it

Answer 1

There are several issues:

• .addCallback() suggests that you are calling a twisted api from a different thread. In general, twisted api is not thread-safe. You could use reactor.callFromThread to run non-thread-safe code in the event loop (main thread) from another thread
• if it is a blocking non-asynchronous code (e.g., an infinite loop) then you could use reactor.callInThread to run a function in a different thread
• if you want to call twisted api in a loop; you could restructure the code to use twisted.internet.task.LoopingCall (code example) or twisted.internet.task.Cooperator.coiterate (here are code examples)