Sliding Window Chat Memory

Sliding Window Chat Memory

As a chat conversation grows, LLM context windows can fill up rapidly. A common strategy is to keep a sliding window of recent messages. However, you must never prune the system prompt, which contains core behavioral instructions and is always at the index 0 position.

In this challenge, we represent a token count simply as the character length of the message's content string (len(message["content"])).

Tasks

Implement the function manage_chat_memory(messages, max_tokens) to:

1. Return the list of messages pruned from the left (removing older messages first) such that the sum of the lengths of the remaining messages is $\le max_tokens$.
2. Keep all messages with "role": "system" (the system prompts) at the beginning of the returned list, regardless of pruning.
3. If the total length of the system messages alone exceeds max_tokens, return only the system messages.

Example

messages = [
    {"role": "system", "content": "SYSTEM_INSTRUCTIONS"}, # len = 19
    {"role": "user", "content": "Hello"},                 # len = 5
    {"role": "assistant", "content": "Hi there!"},        # len = 9
    {"role": "user", "content": "How are you?"}           # len = 12
]
max_tokens = 36
# System prompt (19) + U1 (5) + A1 (9) + U2 (12) = 45 tokens.
# Prunes 'Hello' (U1) to bring total to 19 + 9 + 12 = 40 (still > 36).
# Prunes 'Hi there!' (A1) to bring total to 19 + 12 = 31 (<= 36).
# Returns: [{"role": "system", "content": "SYSTEM_INSTRUCTIONS"}, {"role": "user", "content": "How are you?"}]