I want to do two things:
- Use OpenAI API in streaming mode (Using Langchain Agent).
- Print the answer in streaming mode using Gradio
I know how to generate streaming messages using Gradio, and I know how to activate streaming mode for OpenAI; However, I don't know how to adapt the code to accomplish both things at the same time. Here is my base code that needs modification; I don't know how to play with generators and callbacks.
from langchain.agents import load_tools
from langchain.agents import initialize_agent
from langchain.agents import AgentType
from langchain.chat_models import ChatOpenAI
import gradio as gr
import time
from langchain.callbacks.base import BaseCallbackHandler
from dotenv import load_dotenv
class MyCallbackHandler(BaseCallbackHandler):
def on_llm_new_token(self, token, **kwargs) -> None:
print(token)
def myfun(input):
llm = ChatOpenAI(streaming=True, callbacks=[MyCallbackHandler()])
tools = load_tools(["wikipedia", "llm-math"], llm=llm)
agent = initialize_agent(
tools, llm, agent=AgentType.CONVERSATIONAL_REACT_DESCRIPTION, verbose=False
)
return agent.run(input)
with gr.Blocks() as demo:
chatbot = gr.Chatbot()
msg = gr.Textbox()
clear = gr.Button("Clear")
def user(user_message, history):
return "", history + [[user_message, None]]
def bot(history):
bot_message = myfun(input=history[-1][0])
history[-1][1] = ""
for character in bot_message:
history[-1][1] += str(character)
time.sleep(0.03)
yield history
msg.submit(user, [msg, chatbot], [msg, chatbot], queue=False).then(
bot, chatbot, chatbot
)
clear.click(lambda: None, None, chatbot, queue=False)
demo.queue()
demo.launch()
I want to add memory to the agent, and each Gradio user has a session with memory. How can I do that with the States in Gradio?