21

We have a use case for ChatGPT in summarizing long pieces of text (speech-to-text conversations which can be over an hour).

However we find that the 4k token limit tends to lead to a truncation of the input text to say half or so due to the token limit.

Processing in parts does not seem to retain history of previous parts.

What options do we have for submitting a longer request which is over 4k tokens?

Brendan Hill
  • 3,406
  • 4
  • 32
  • 61
  • 1
    You could tell it that you are going to send multiple pieces of text and it does not need to respond until you explicitly ask for a response. Then send your text in chunks. – Barish Namazov Mar 06 '23 at 06:31
  • This person is asking about the ChatGPT API, not the public ChatGPT web interface. The web version does some sort of smart summarization/compression of any previous conversation once it gets too long, as it must still have the same limits under the hood. For the API you're on your own. Each request to the API is totally separate and retains no "memory" of the previous API calls. I too am wondering what the best solution here is. – jameslol Mar 08 '23 at 04:38
  • OP perhaps you could attempt to implement whatever it is that the web ChatGPT is doing - split all your text into chunks of about 3000 words, get a summary of each in separate API calls, and then send all the summaries in another API call, to get a "summary of summaries" – jameslol Mar 08 '23 at 04:43
  • @jameslol It is not always possible to correctly break the text into pieces – Victor Kushnerov Mar 13 '23 at 07:54
  • @BarishNamazov Could you provide an example of a prompt on how to send a long text to ChatGPT? – Victor Kushnerov Mar 13 '23 at 07:58

4 Answers4

7

The closest answer to your question would be in the form of Embeddings.

You can find an overview of what they are here.

I recommend you review this code from the OpenAI Cookbook Github page that used a Web Crawl Q&A example to explain embeddings.

I used the code from Step 5 onwards and altered the location of the text to poin it to my file containing the long piece of text.

From:

# Open the file and read the text
with open("text/" + domain + "/" + file, "r", encoding="UTF-8") as f:
    text = f.read()

to:

# Open the file and read the text
with open("/my_location/long_text_file.txt", "r", encoding="UTF-8") as f:
    text = f.read()

And modified the questions at Step 13 to what I needed to know about the text.

Aklelka
  • 221
  • 2
  • 4
4

Another option is the ChatGPT retrieval plugin. This allows for creation of a vector database of your document's text which can be then processed by the LLM. See https://github.com/openai/chatgpt-retrieval-plugin

RonanOD
  • 876
  • 1
  • 9
  • 19
0

You can use GPT-4 for long contexts

As stated in OpenAI:

GPT-4 is capable of handling over 25,000 words of text, allowing for use cases like long form content creation, extended conversations, and document search and analysis.

Masoud Gheisari
  • 949
  • 1
  • 9
  • 21
0

One approach to handle long text is to divide it into smaller fragments, retrieve the appropriate pieces according to your task, and then send them through an API call.

Here's a project that is capable of processing PDFs, txt and doc files, as well as web pages. It allows you to converse with the document. In your case, you could ask a general question like "what is the document about" to receive a summary, and then inquire for more specific details.

Chiyu Song
  • 1