2

I'm working on a chatbot powered by OpenAI. I'm using the new gpt-3.5-turbo model with ChatCompletion requests. I've already come up with a way for the bot to remember conversations for each individual user using a HashMap and it all works. The bot is able to remember all the previous responses and generate a new response using the history as context. However, the more responses in the query, the more tokens that it'll use, increasing costs. I want make it where if there is a certain amount of inactivity (like 2 minutes of not talking to the bot), it'll clear the history and start a fresh new conversation to save on tokens. What would be the best way to accomplish this?

Here is my code:

private static OpenAiService service;
    
private static Map<String, List<ChatMessage>> conversations = new HashMap<String, List<ChatMessage>>();
            
public static void initializeOpenAi() {
    service = new OpenAiService(Settings.OPENAI_ACCESS_KEY);
}

public static String generateChatResponse(User user, String prompt) {
    List<ChatMessage> conversation;
        
    if (conversations.containsKey(user.getId())) {
        conversation = conversations.get(user.getId());
    } else {
        conversation = new ArrayList<ChatMessage>();
    }
        
    if (conversation.size() < 1) {
        ChatMessage system = new ChatMessage();
        system.setRole("system");
        system.setContent("Omnis is a chatbot with sarcasm");
        conversation.add(system);
    }
        
    ChatMessage userPrompt = new ChatMessage();
    userPrompt.setRole("user");
    userPrompt.setContent(prompt);
    conversation.add(userPrompt);
        
    ChatCompletionRequest chatRequest = ChatCompletionRequest.builder()
            .model("gpt-3.5-turbo")
            .maxTokens(1000)
            .temperature(0.9)
            .topP(0.3)
            .frequencyPenalty(0.9)
            .presencePenalty(0.0)
            .messages(conversation)
            .build();
        
    ChatCompletionResult chatResult = null;
        
    try {
        chatResult = service.createChatCompletion(chatRequest);
    } catch (Exception e) {
        System.out.println("An OpenAI request failed!");
            
        if (e.getMessage().contains("timeout")) {
            return "Your request timed out. Please try again after a breif wait for try a different request.";
        }
            
        if (e.getMessage().contains("Rate limit")) {
            return "Rate limit for chat requests has been reached!";
        }
            
        return "Something went wrong with your request. Cause of error is " + e.getMessage();
    }
        
    if (chatResult != null) {
        System.out.println("Created chat completion request that used " + chatResult.getUsage().getTotalTokens() + " tokens with " + chatResult.getModel());
    }
        
    String response = chatResult.getChoices().get(0).getMessage().getContent();
        
    ChatMessage assistantPrompt = new ChatMessage();
    assistantPrompt.setRole("assistant");
    assistantPrompt.setContent(response);
    conversation.add(assistantPrompt);
        
    conversations.put(user.getId(), conversation);
        
    return response;
}

The "generateChatResonse" method gets called each time a message is sent by a user. It get's the user's ID and pulls the conversation from the HashMap if it exists, if not it creates a new conversation.

I don't know what to try.

I don't want to clear the entire HashMap, just the entry associated with the inactive user.

Dylan Wedman
  • 61
  • 6
  • 2
    On a related note: [*Java time-based map/cache with expiring keys*](https://stackoverflow.com/q/3802370/642706) – Basil Bourque Mar 03 '23 at 19:17
  • Do you want to clear the entire map, or clear the “conversation” list? Edit your Question to clarify. – Basil Bourque Mar 03 '23 at 19:23
  • I've updated the title. I don't want to clear the entire map, just the entry containing the list of chat messages. – Dylan Wedman Mar 03 '23 at 19:31
  • I looked at the link you sent and found Guava. It has a HashMap like object named CacheBuilder that has time-based expiration of entries measured since last access or write. I think this might be an option. – Dylan Wedman Mar 03 '23 at 19:45
  • If the Guava `CacheBuilder` solves your issue, know that on Stack Overflow you are welcome and encouraged to post and accept an Answer to your own Question. – Basil Bourque Mar 03 '23 at 20:18
  • I changed your title and body to use the more general `Map` rather than `HashMap`. Unless your code requires the specific features of a concrete class such as `HashMap`, use the more general superclass or interface. – Basil Bourque Mar 03 '23 at 20:20
  • This is a solved problem: You *actually* want a cache with expiry. There are libraries for this. – Bohemian Mar 03 '23 at 20:54
  • You must use a local datetime object because of midnight ! Use date-time variable to update as the very last line of activity input code of an activity action. Use first lines of activity input code to obtain that last updated stored date-time object (one common reference) and use java.time.temporal Period to add 2 minutes then use compare() method by casting the LocalDateTime to java.time.temporal. TemporalAccessor and use the static ChronoLocalDateTime? .from( ) method to convert that time and compare the later one as a chronodate – Samuel Marchant Mar 04 '23 at 06:21
  • Plenty of java.time info and IANA time zone java update info. https://drive.google.com/file/d/1gjHmdC-BW0Q2vXiQYmp1rzPU497sybNy/view?usp=drivesdk – Samuel Marchant Mar 04 '23 at 06:26

2 Answers2

1

I've added Guava to my project and replaced the Map with a LoadingCache object. It takes the same type of parameters and allows me to easily set the time before expiration.

private static LoadingCache<String, List<ChatMessage>> conversations;

public static void initializeOpenAi() {
    service = new OpenAiService(Settings.OPENAI_ACCESS_KEY);
        
    conversations = CacheBuilder.newBuilder()
            .maximumSize(10000)
            .expireAfterWrite(2, TimeUnit.MINUTES)
            .build(
                new CacheLoader<String, List<ChatMessage>>() {
                    @Override
                    public List<ChatMessage> load(String key) throws Exception {
                        return conversations.get(key);
                    }
                });
}

public static String generateChatResponse(User user, String prompt) {
    List<ChatMessage> conversation = conversations.getIfPresent(user.getId());
        
    if (conversation == null) {
        conversation = new ArrayList<ChatMessage>();
            
        ChatMessage system = new ChatMessage();
        system.setRole("system");
        system.setContent("Omnis is a chatbot with sarcasm");
            
        conversation.add(system);
    }

    // Remaining code is unchanged
}
Dylan Wedman
  • 61
  • 6
  • I can't accept my own answer until tomorrow. But I can confirm that this works the way I need it to. – Dylan Wedman Mar 04 '23 at 09:32
  • I already answered my own question before the site was like, "Nah, there's already answers for this question in another post", even though my answer provides the implementation rather than just an explanation. I can't accept the answer yet because I submitted it at like 4 AM. Gotta love it here. – Dylan Wedman Mar 04 '23 at 18:38
-1

How about a thread ? Check if there are changes , and if not for an amount of time to clear it ?

Ftoy
  • 26
  • 6
  • Can you explain in more detail, I don't know the full extent of Threads. I mainly use them for running tasks that otherwise would cause the execution to hang. – Dylan Wedman Mar 03 '23 at 18:53
  • You could use an SortedMap to track userId->timeLastActive. You could then use a thread to iterate over that data and purge user ids from conversations. –  RichardFeynman Mar 03 '23 at 19:10