I am writing a Chrome extension where I need to stream responses from OpenAI endpoints in the content script. The content script runs when the context menu is clicked. I am using Craco to build the react file as the content script of the extension. I can't call the API from the background script since I am using eventsource-parser
library. That's the main reason I am using Craco so that I can import that lib inside content script.
Here is the code to extract text from the stream:
const getStream = async (data) => {
const nextTokens = await getNextTokens(data);
const reader = nextTokens.getReader();
const decoder = new TextDecoder();
let done = false;
let text = '';
while (!done) {
const { value, done: doneReading } = await reader.read();
done = doneReading;
const chunkValue = decoder.decode(value);
console.log(chunkValue)
text += chunkValue;
}
}
Here is the code for getNextTokens
:
const getNextTokens = async (prompt) => {
const url = "https://api.openai.com/v1/chat/completions";
// Get config from storage
const {
apiKey,
model,
temperature,
maxTokens,
topP,
frequencyPenalty,
presencePenalty,
} = await getConfig();
// Create request body
//const data = { model: model, messages: [{role: "user", content: prompt }], stream: true }
const encoder = new TextEncoder()
const decoder = new TextDecoder()
let counter = 0
// Create headers
const requestHeaders = {
'Content-Type': 'application/json',
Authorization: "Bearer " + apiKey,
}
const res = await fetch(url, {
headers: requestHeaders,
method: 'POST',
body: JSON.stringify({
model: model,
messages: [{role: "user", content: prompt }],
max_tokens: 1000,
temperature: 1,
stream: true,
}),
})
const stream = new ReadableStream({
async start(controller) {
// callback
function onParse(event) {
if (event.type === 'event') {
const data = event.data
// https://beta.openai.com/docs/api-reference/completions/create#completions/create-stream
if (data === '[DONE]') {
console.log('DONE')
controller.close()
return
}
try {
const json = JSON.parse(data)
const text = json.choices[0].delta?.content || ''
if (counter < 2 && (text.match(/\n/) || []).length) {
// this is a prefix character (i.e., "\n\n"), do nothing
return
}
const queue = encoder.encode(text)
controller.enqueue(queue)
counter++
} catch (e) {
// maybe parse error
controller.error(e)
}
}
}
// stream response (SSE) from OpenAI may be fragmented into multiple chunks
// this ensures we properly read chunks and invoke an event for each SSE event stream
const parser = createParser(onParse)
for await (const chunk of res.body) {
parser.feed(decoder.decode(chunk))
}
},
})
return stream
};
This is always returning the Object is not async iterable
error. Did anyone face this issue before? I'd appreciate some help. Thanks.