TL.DR;
It is a complex request. I don't think can be achieved with a trick. Or you own the process in which is running blender (meaning that you import it's api), or you attach to the process (using gdb, but I don't know if then you can use the IDE you wanted) or you use an IDE with pydevd included. And even so, I don't know how much you can achieve.
Synchronize two python process is not trivial. The answer shows a little of that.
PyDev.Debugger
You want to find a way to synchronize two python objects that lives in different python instances. I think that the only real solution to your problem is to setup a pydevd server and connect to it. It is simpler if you use one of the supported IDE, like PyDEV or PyCharm since they have everything in place to do so:
The work carried out by pydev is not trivial, the repository is quite a project. It is your best bet, but I cannot guarantee you it will work.
Process Communication
The usual communication solution will not work because they serialize and deserialize (pickle and unpickle) data behind the scenes. Let's have an example, implementing a server in the blender process that receives arbitrary code as a string and execute it, sending back the last result of the code. The result will be received by the client as a Python object, thus you can use your IDE interface to inspect it, or you can even run some code on it. There are limitations:
- not everything can be received by the client (e.g. class definition must exist in the client)
- only pickable objects can transit on the connection
- the object in the client and in the server are different: modifications made on the client will not be applied on the server without additional (quite complex) logic
This is the server that should run on your Blender instance
from multiprocessing.connection import Listener
from threading import Thread
import pdb
import traceback
import ast
import copy
# This is your configuration, chose a strong password and only open
# on localhost, because you are opening an arbitrary code execution
# server. It is not much but at least we cover something.
port = 6000
address = ('127.0.0.1', port)
authkey = b'blender'
# If you want to run it from another machine, you must set the address
# to '0.0.0.0' (on Linux, on Windows is not accepted and you have to
# specify the interface that will accept the connection)
# Credits: https://stackoverflow.com/a/52361938/2319299
# Awesome piece of code, it is a carbon copy from there
def convertExpr2Expression(Expr):
r"""
Convert a "subexpression" of a piece of code in an actual
Expression in order to be handled by eval without a syntax error
:param Expr: input expression
:return: an ast.Expression object correctly initialized
"""
Expr.lineno = 0
Expr.col_offset = 0
result = ast.Expression(Expr.value, lineno=0, col_offset=0)
return result
def exec_with_return(code):
r"""
We need an evaluation with return value. The only two function
that are available are `eval` and `exec`, where the first evaluates
an expression, returning the result and the latter evaluates arbitrary code
but does not return.
Those two functions intercept the commands coming from the client and checks
if the last line is an expression. All the code is executed with an `exec`,
if the last one is an expression (e.g. "a = 10"), then it will return the
result of the expression, if it is not an expression (e.g. "import os")
then it will only `exec` it.
It is bindend with the global context, thus it saves the variables there.
:param code: string of code
:return: object if the last line is an expression, None otherwise
"""
code_ast = ast.parse(code)
init_ast = copy.deepcopy(code_ast)
init_ast.body = code_ast.body[:-1]
last_ast = copy.deepcopy(code_ast)
last_ast.body = code_ast.body[-1:]
exec(compile(init_ast, "<ast>", "exec"), globals())
if type(last_ast.body[0]) == ast.Expr:
return eval(compile(convertExpr2Expression(last_ast.body[0]), "<ast>", "eval"), globals())
else:
exec(compile(last_ast, "<ast>", "exec"), globals())
# End of carbon copy code
class ArbitraryExecutionServer(Thread):
r"""
We create a server execute arbitrary piece of code (the most dangerous
approach ever, but needed in this case) and it is capable of sending
python object. There is an important thing to keep in mind. It cannot send
**not pickable** objects, that probably **include blender objects**!
This is a dirty server to be used as an example, the only way to close
it is by sending the "quit" string on the connection. You can envision
your stopping approach as you wish
It is a Thread object, remeber to initialize it and then call the
start method on it.
:param address: the tuple with address interface and port
:param authkey: the connection "password"
"""
QUIT = "quit" ## This is the string that closes the server
def __init__(self, address, authkey):
self.address = address
self.authkey = authkey
super().__init__()
def run(self):
last_input = ""
with Listener(self.address, authkey=self.authkey) as server:
with server.accept() as connection:
while last_input != self.__class__.QUIT:
try:
last_input = connection.recv()
if last_input != self.__class__.QUIT:
result = exec_with_return(last_input) # Evaluating remote input
connection.send(result)
except:
# In case of an error we return a formatted string of the exception
# as a little plus to understand what's happening
connection.send(traceback.format_exc())
if __name__ == "__main__":
server = ArbitraryExecutionServer(address, authkey)
server.start() # You have to start the server thread
pdb.set_trace() # I'm using a set_trace to get a repl in the server.
# You can start to interact with the server via the client
server.join() # Remember to join the thread at the end, by sending quit
While this is the client in your VSCode
import time
from multiprocessing.connection import Client
# This is your configuration, should be coherent with
# the one on the server to allow the connection
port = 6000
address = ('127.0.0.1', port)
authkey = b'blender'
class ArbitraryExecutionClient:
QUIT = "quit"
def __init__(self, address, authkey):
self.address = address
self.authkey = authkey
self.connection = Client(address, authkey=authkey)
def close(self):
self.connection.send(self.__class__.QUIT)
time.sleep(0.5) # Gives some time before cutting connection
self.connection.close()
def send(self, code):
r"""
Run an arbitrary piece of code on the server. If the
last line is an expression a python object will be returned.
Otherwise nothing is returned
"""
code = str(code)
self.connection.send(code)
result = self.connection.recv()
return result
def repl(self):
r"""
Run code in a repl loop fashion until user enter "quit". Closing
the repl will not close the connection. It must be manually
closed.
"""
last_input = ""
last_result = None
while last_input != self.__class__.QUIT:
last_input = input("REMOTE >>> ")
if last_input != self.__class__.QUIT:
last_result = self.send(last_input)
print(last_result)
return last_result
if __name__ == "__main__":
client = ArbitraryExecutionClient(address, authkey)
import pdb; pdb.set_trace()
client.close()
At the bottom of the script there is also how to launch them while having pdb as "repl". With this configuration you can run arbitrary code from the client on the server (and in fact it is an extremely dangerous scenario, but for your very specific situation is valid, or better "the main requirement").
Let's dive into the limitation I anticipated.
You can define a class Foo on the server:
[client] >>> client = ArbitraryExecutionClient(address, authkey)
[client] >>> client.send("class Foo: pass")
[server] >>> Foo
[server] <class '__main__.Foo'>
and you can define an object named "foo" onthe server, but you will immediately receive an error because the class Foo does not exist in the local instance (this time using the repl):
[client] >>> client.repl()
[client] REMOTE >>> foo = Foo()
[client] None
[client] REMOTE >>> foo
[client] *** AttributeError: Can't get attribute 'Foo' on <module '__main__' from 'client.py'>
this error appears because there is no declaration of the Foo class in the local instance, thus there is no way to correctly unpickle the received object (this problem will appear with all the Blender objects. Take notice, if the object is in some way importable, it may still work, we will see later on this situation).
The only way to not receive the error is to previously declare the class also on the client, but they will not be the same object, as you can see by looking at their ids:
[client] >>> class Foo: pass
[client] >>> client.send("foo")
[client] <__main__.Foo object at 0x0000021E2F2F3488>
[server] >>> foo
[server] <__main__.Foo object at 0x00000203AE425308>
Their id are different because they live in a different memory space: they are completely different instances, and you have to manually synchronize every operation on them!
If the class definition is in some way importable and the object are pickable, you can avoid to multiply the class definition, as far as I can see it will be automatically imported:
[client] >>> client.repl()
[client] REMOTE >>> import numpy as np
[client] None
[client] REMOTE >>> ary = np.array([1, 2, 3])
[client] None
[client] REMOTE >>> ary
[client] [1 2 3]
[client] REMOTE >>> quit
[client] array([1, 2, 3])
[client] >>> ary = client.send("ary")
[client] >>> ary
[client] array([1, 2, 3])
[client] >>> type(ary)
[client] <class 'numpy.ndarray'>
We never imported on the client numpy but we have correctly received the object. But what happen if we modify the local instance to the remote instance?
[client] >>> ary[0] = 10
[client] >>> ary
[client] array([10, 2, 3])
[client] >>> client.send("ary")
[client] array([1, 2, 3])
[server] >>> ary
[server] array([1, 2, 3])
we have no synchronization of the modifications inside the object.
What happens if an object is not pickable? We can test with the server variable, an object that is a Thread and contains a connection, which are both non pickable (meaning that you cannot give them an invertible representation as a list of bytes):
[server] >>> import pickle
[server] >>> pickle.dumps(server)
[server] *** TypeError: can't pickle _thread.lock objects
and also we can see the error on the client, trying to receive it:
[client] >>> client.send("server")
[client] ... traceback for "TypeError: can't pickle _thread.lock objects" exception ...
I don't think there is a "simple" solution for this problem, but I think there is some library (like pydevd) that implements a full protocol for overcoming this problem.
I hope now my comments are more clear.
