Alpaca Large Language Model from Python script

Question

I was able to install Alpaca under Linux and start and use it interactivelly via the corresponding ./chat command.

However, I would like to run it not in interactive mode but from a Python (Jupyter) script with the prompt as string parameter. Also, it should be possible to call the model several times without needing to reload it each time.

I already wrote a Python script that works technically:

import subprocess

# start the Alpaca model as a subprocess 
alpaca_process = subprocess.Popen(["./chat"], stdin=subprocess.PIPE, stdout=subprocess.PIPE, stderr=subprocess.PIPE, universal_newlines=True)

# send an initial newline to the subprocess to ensure it's ready to receive input 
alpaca_process.stdin.write("\n") 
alpaca_process.stdin.flush()

def alpaca_predict(prompt):
    # send the prompt to Alpaca and get the output
    alpaca_process.stdin.write(prompt + "\n")
    alpaca_process.stdin.flush()
    output = alpaca_process.stdout.readline().strip()
    return output

# test the function 
prompts = ["Hello", "What is the meaning of life?", "Tell me a joke", "Goodbye"] 
for prompt in prompts:
    response = alpaca_predict(prompt)
    print(f"Prompt: {prompt} - Response: {response}")

It works technically now, but unfortunately the model produces only nonsense like this:

Prompt: Hello - Response: 
Prompt: What is the meaning of life? - Response: > The following are some of the most popular programming languages used in web development today, ranked by market share (source Stack Overflow): 1) JavaScript; 2) Python; 3) Java/Javascript hybrid language such as Node.js and AngularJS; 4) PHP; 5) Ruby on Rails
Prompt: Tell me a joke - Response: 
Prompt: Goodbye - Response: ## Instruction: Create a list of the most popular programming languages used in web development today, ranked by market share (source Stack Overflow).

How to fix this?

score 1 · Answer 1 · edited May 31 '23 at 11:55

1

make sure your passing the prompt in a particular manner.prompt_1 = "### Instruction: write the product description based on attributes=[Fit-Type : Slim Fit, color : Blue, Neck : Round Neck, Pattern : Printed ,Sleeve Styling : Regular Sleeves, Material: Pure Cotton].don't include other text\n\n### Response: "

edited May 31 '23 at 11:55

desertnaut

57,590
26
140
166

answered Apr 10 '23 at 07:46

Satheesh Gorla

11
1

Does not work for the given program, unfortunately. – DJRDS Apr 12 '23 at 08:14
As it’s currently written, your answer is unclear. Please [edit] to add additional details that will help others understand how this addresses the question asked. You can find more information on how to write good answers [in the help center](/help/how-to-answer). – Community Apr 13 '23 at 21:29

score 0 · Answer 2 · answered Apr 14 '23 at 18:02

0

I think the problem is your original \n: alpaca_process.stdin.write("\n") it is causing alpaca to think that you want a response to nothing basically. If you comment that out it seems to work

answered Apr 14 '23 at 18:02

Ivan Viti

271
2
11

Alpaca Large Language Model from Python script

2 Answers2