1

I want to do two things:

  1. Use OpenAI API in streaming mode (Using Langchain Agent).
  2. Print the answer in streaming mode using Gradio

I know how to generate streaming messages using Gradio, and I know how to activate streaming mode for OpenAI; However, I don't know how to adapt the code to accomplish both things at the same time. Here is my base code that needs modification; I don't know how to play with generators and callbacks.

from langchain.agents import load_tools
from langchain.agents import initialize_agent
from langchain.agents import AgentType
from langchain.chat_models import ChatOpenAI
import gradio as gr
import time
from langchain.callbacks.base import BaseCallbackHandler
from dotenv import load_dotenv


class MyCallbackHandler(BaseCallbackHandler):
    def on_llm_new_token(self, token, **kwargs) -> None:
        print(token)
        
def myfun(input):
    llm = ChatOpenAI(streaming=True, callbacks=[MyCallbackHandler()])
    tools = load_tools(["wikipedia", "llm-math"], llm=llm)
    agent = initialize_agent(
        tools, llm, agent=AgentType.CONVERSATIONAL_REACT_DESCRIPTION, verbose=False
    )
    return agent.run(input)


with gr.Blocks() as demo:
    chatbot = gr.Chatbot()
    msg = gr.Textbox()
    clear = gr.Button("Clear")

    def user(user_message, history):
        return "", history + [[user_message, None]]

    def bot(history):
        bot_message = myfun(input=history[-1][0])
        history[-1][1] = ""
        for character in bot_message:
            history[-1][1] += str(character)
            time.sleep(0.03)
            yield history

    msg.submit(user, [msg, chatbot], [msg, chatbot], queue=False).then(
        bot, chatbot, chatbot
    )
    clear.click(lambda: None, None, chatbot, queue=False)
    
demo.queue()
demo.launch()

I want to add memory to the agent, and each Gradio user has a session with memory. How can I do that with the States in Gradio?

Eric Bellet
  • 1,732
  • 5
  • 22
  • 40

0 Answers0