0

LLAMACPP Pycharm I am trying to run LLAMA2 Quantised models on my MAC referring to the link above.

When I run the below code on Jupyter notebook, it works fine and gives expected output. However, it gives a sigerror while running on Pycharm. Could someone assist with the same?

from langchain.llms import LlamaCpp

import os
print(os.path.isfile("../../AL/llama-2-7b-chat.ggmlv3.q4_0.bin"))

n_gpu_layers = 2  # Metal set to 1 is enough.
n_batch = 512  # Should be between 1 and n_ctx, consider the amount of RAM of your Apple Silicon Chip.

llm = LlamaCpp(
    model_path="../../AL/llama-2-7b-chat.ggmlv3.q4_0.bin",
    n_gpu_layers=n_gpu_layers,
    n_batch=n_batch,
    f16_kv=True,  # MUST set to True, otherwise you will run into problem after a couple of calls
    verbose=True,
)

prompt = """
Question: What's capital of Australia?
Answer: 
"""
llm(prompt)

Output when running directly through terminal

llama_print_timings:        load time =   248.89 ms
llama_print_timings:      sample time =    91.66 ms /   133 runs   (    0.69 ms per token,  1451.05 tokens per second)
llama_print_timings: prompt eval time =   248.86 ms /    17 tokens (   14.64 ms per token,    68.31 tokens per second)
llama_print_timings:        eval time =  3752.79 ms /   132 runs   (   28.43 ms per token,    35.17 tokens per second)
llama_print_timings:       total time =  4250.88 ms
Exception ignored in: <function Llama.__del__ at 0x16d52cc10>
Traceback (most recent call last):
  File "/Users/<User Name>/anaconda3/lib/python3.10/site-packages/llama_cpp/llama.py", line 1558, in __del__
TypeError: 'NoneType' object is not callable

Line 1558 in Llama.py

    def __del__(self):
    if hasattr(self, "model") and self.model is not None:
        llama_cpp.llama_free_model(self.model)########Line 1558
        self.model = None
    if hasattr(self, "ctx") and self.ctx is not None:
        llama_cpp.llama_free(self.ctx)
        self.ctx = None

Output on Pycharm enter image description here

Output on Jupiter Notebook enter image description here

Jason
  • 676
  • 1
  • 12
  • 34
  • You can reduce the batch size and check, seems to be memory allocation issue. – ZKS Aug 22 '23 at 08:03
  • @ZKS - Why is it giving a memory allocation issue through pycharm and not on Jupypter notebook? – Jason Aug 22 '23 at 08:16
  • The Java Virtual Machine (JVM) running PyCharm allocates some predefined amount of memory. The default value depends on the platform. If you are experiencing slowdowns, you may want to increase the memory heap. https://www.jetbrains.com/help/pycharm/increasing-memory-heap.html – ZKS Aug 22 '23 at 08:20
  • @ZKS - I changed the maximum heap size from 2048MiB to 16384MiB which is the max on my machine but still giving the same error. Updated in description -output when I run directly using terminal – Jason Aug 22 '23 at 14:10

0 Answers0