-1

I'm trying to convert a pure Python module into Cython compatible one using EasyCython module.
The issue however is, upon trying to execute the EasyCython, it fails with the error:

G:\Proc\python\FV>easycython F_V.pyx
Compiling F_V.pyx because it changed.
[1/1] Cythonizing F_V.pyx
C:\Users\Rika\Anaconda3\Lib\site-packages\Cython\Compiler\Main.py:369: FutureWarning: Cython directive 'language_level' not set, using 2 for now (Py2). This will change in a later release! File: G:\Proc\python\FV\F_V.pyx
  tree = Parsing.p_module(s, pxd, full_module_name)

Error compiling Cython file:
------------------------------------------------------------
...
    # or the network would mistake some one else for someother people! (for example someone has a profile image
    # while the other doesnt, when confronted with the profile image, the network most likely find more features
    # from the person that has already a profile image of him in the fbank (unless the change is noticeable
    # this really can be a major issue)))
    # @benchmark
    cpdef _identify_id(self, input_img_embedding, list embedding_list, bint short_circut=True, bint accumulate_score=False):
         ^
------------------------------------------------------------

F_V.pyx:929:10: cdef statement not allowed here
Traceback (most recent call last):
  File "C:\Users\Rika\Anaconda3\Lib\runpy.py", line 193, in _run_module_as_main
    "__main__", mod_spec)
  File "C:\Users\Rika\Anaconda3\Lib\runpy.py", line 85, in _run_code
    exec(code, run_globals)
  File "C:\Users\Rika\Anaconda3\Scripts\easycython.exe\__main__.py", line 7, in <module>
  File "C:\Users\Rika\Anaconda3\Lib\site-packages\begin\main.py", line 54, in start
    collector=self._collector)
  File "C:\Users\Rika\Anaconda3\Lib\site-packages\begin\cmdline.py", line 253, in apply_options
    return_value = call_function(func, signature(ext), opts)
  File "C:\Users\Rika\Anaconda3\Lib\site-packages\begin\cmdline.py", line 236, in call_function
    return func(*pargs, **kwargs)
  File "C:\Users\Rika\Anaconda3\Lib\site-packages\easycython\easycython.py", line 77, in main
    ext_modules = cythonize(ext_modules),
  File "C:\Users\Rika\Anaconda3\Lib\site-packages\Cython\Build\Dependencies.py", line 1102, in cythonize
    cythonize_one(*args)
  File "C:\Users\Rika\Anaconda3\Lib\site-packages\Cython\Build\Dependencies.py", line 1225, in cythonize_one
    raise CompileError(None, pyx_file)
Cython.Compiler.Errors.CompileError: F_V.pyx

Here are the two methods that are being annotated with cython types.

#@benchmark
cpdef _identify_id(self, input_img_embedding, list embedding_list, bint short_circut=True, bint accumulate_score=False):
    # These are the underlying types for the arguments and local variables used here
    # input_img_embedding_type: <class 'torch.Tensor'> shape:torch.Size([1, 512])
    # feature1 type: <class 'torch.Tensor'> shape: torch.Size([1, 512])
    # x1 type: <class 'torch.Tensor'> shape: torch.Size([1, 512])
    # cosine type: <class 'numpy.float32'> shape: ()
    # np.clip(cosine) type: <class 'numpy.float64'> shape: ()
    #
    cdef float min_theta = 1000.0
    cdef float total_theta = 0.0
    cdef char* id_name = 'None' #None
    for (name, feature1) in embedding_list:
        id_name = name
        x1 = feature1 / np.linalg.norm(feature1)
        cosine = np.dot(input_img_embedding.squeeze(0), x1.squeeze(0))
        cdef float cosine = np.clip(cosine, -1.0, 1.0)
        cdef float theta = math.acos(cosine)
        cdef float theta = theta * 180 / math.pi

        if short_circut:
            if theta < self._threshold:
                return id_name, theta

        if theta < min_theta:
            min_theta = theta

        total_theta += theta

    # the elses from now on are for debugging purposes
    if not short_circut and not accumulate_score:
        if min_theta < self._threshold:
            return id_name, min_theta
        else:
            return 'unknown', min_theta

    if accumulate_score:
        final_score = total_theta/len(embedding_list)
        if final_score < self._threshold:
            return id_name, final_score
        else:
            return 'unknown', final_score

    return 'unknown', theta # min_theta

#@benchmark
cpdef _check_in_fbank(self, img):
    """Checks whether a given image is represented in the face bank.

    Arguments:
        img {torch.tensor} -- input image to be verified

    Returns:
        tuple(name, theta)
    """
        # These are the underlying python types
        # img type: <class 'torch.Tensor'> shape: torch.Size([3, 112, 112])
        # feature0 type: <class 'torch.Tensor'> shape: torch.Size([1, 512])
        # x0 type: <class 'torch.Tensor'> shape: torch.Size([1, 512])

    with Benchmark_Block("model frwd took: "):
         feature0 = self.model(img.unsqueeze(0)).cpu().detach()
         x0 = feature0 / np.linalg.norm(feature0)

    cdef list lst = []
    for img_list in self._fbank_embeddings:
        cdef tuple f = self._identify_id(x0, img_list, short_circut=self.short_circut, accumulate_score=self.accumulate_score)
        lst.append(f)

    cdef tuple min_val  = min(lst, key=lambda t: t[1])
    print(f'lst of returned results : {lst}. The minimum is: {min_val} in {len(self._fbank_embeddings)} enteries')
    return min_val

What am I missing here?

Hossein
  • 24,202
  • 35
  • 119
  • 224
  • The decorator (subject of the duplicate I've marked) is one issue - it isn't allowed on a `cdef` function. The the `int*` is another issue - it isn't allowed on a `def` function (because it has no equivalent Python type). `cpdef` has all the restrictions of both `cdef` and `def` functions (see https://stackoverflow.com/questions/28362009/definition-of-def-cdef-and-cpdef-in-cython/41976772#41976772 for a description of that) – DavidW Jun 01 '20 at 07:40
  • With that said - the code you show doesn't match the error you show so who knows. If this+the duplicate doesn't help you solve it then I'm happy to reopen it though – DavidW Jun 01 '20 at 07:42
  • Thanks, but if I remove all the int* and also the decorator I still get the same error. I updated the error log, I forgot to update it previosuly. – Hossein Jun 01 '20 at 07:48
  • 1
    I'm afraid I don't know at the moment but I've removed the duplicate flag – DavidW Jun 01 '20 at 08:03
  • Thanks I apprecaite it. meanwhile I add some more information concerning the types of arguments and other local variables, so it could show what I'm doing wrong. – Hossein Jun 01 '20 at 08:15
  • Is this in a class of some kind? (the `self` argument would kind of suggest it is). The error message definitely implies the full context of where it's defined is important – DavidW Jun 01 '20 at 08:18
  • Yes, its a class method. I have a module, which basically is a Class, and the two methods that need to be optimized are these two. so I tried annotating them using cython and this is where I'm stuck now! Should I be cdefing the class as well? – Hossein Jun 01 '20 at 08:20
  • Having a real [mcve] would be of great help here. – ead Jun 01 '20 at 08:22
  • Thanks everyone, thanks to the @DavidW good point, I found the issue, since I was using a class, I had to use cdef with it as well. basically I am writting an extension type, so it had to use cdef . – Hossein Jun 01 '20 at 08:32

1 Answers1

1

The issue happens to be that When using a python class, one should also use cdef as well. This kind of class is called an Extension Type.
From Cython documentation:

... a Cython extension type definition looks a lot like a Python class definition. Within it, you use the def statement to define methods that can be called from Python code. You can even define many of the special methods such as init() as you would in Python.

Note:

After doing so, in order to be able to use my class in Python, I had to inherit from it and instead use the inherited class. That is I had to do : 

cdef class MyClass():
    def __init__(self, args)
        ... 

    cpdef func1(self,...):
        ...

    cpdef func2(self, ....):
        ...
    ...

class MyClass2(MyClass):
    def __init__(self, args):
       super().__init__(args)
       pass

And in you client code : 

# use MyClass2 from now on
obj = MyClass2(args)
...
Hossein
  • 24,202
  • 35
  • 119
  • 224
  • 1
    The one thing I'd add is that you probably don't really need to be using `cpdef` functions. Cython still optimizes `def` functions with suitably typed variables. All `cpdef` does is make it slightly quicker to call from C. Since these look like large functions I doubt that the calling overhead is significant. Anyway, glad you got it sorted. – DavidW Jun 01 '20 at 08:44
  • @DavidW, Thanls alot, really appreciate your kind help. – Hossein Jun 01 '20 at 09:09