I run a collection of small local LLMs on my home network.
It is a Raspberry Pi 5 with the Hailo AI Hat 2+. Using this mainly as a self teaching device, I quickly ran into a hard limitation. The pre-compiled HEF models that operate on my setup have a VERY hard token limit of 2048 tokens for each response, and in for contextual memory in general, which makes extremely long conversations something I am not able to actually fiddle with until I had an idea.
Since that 2048 token data travels to and from the OpenWeb UI chat interface where I sit on one side over to hailo-ollama, I had an out of the box idea inspired by Nintendo's old practices that they did on their old cartridge games to 'get more space'.
Bank switching -
Yes, this is a wild idea.
I conceptualized, directed, supervised, and dog-fooded a software bridge that sits between hailo-ollama and OpenWeb UI as an almost proxy. This gives the user the ability to provided a lot more input into the LLM in dynamically created 'banks' that exist on the Raspberry Pi's RAM and as needed the information that is stored in the banks is dynamically swapped in and out of the chat messages, transparent to myself. It uses the x-openwebui-chat-id for per-chat RAM state and it adds /save and /forget commands.
I dog-fooded this myself, and use it extensively to continue to test and further refine this, unfortunately, I am a network engineer, and not a coder, so I had to use vibe-coding for this, so I would REALLY like some human eyes on this thing.
With a default behavior of unless specifically told by the end user with a /save, the chat banks are automatically discarded after 5 minutes or when the Raspberry Pi 5 is restarted. Otherwise, the banks NEVER touch the microSD card and exist only in RAM, and that information is moved between the Raspberry Pi's RAM and the AI Hat's RAM. /forget tosses the banks away.
I would REALY love some human eyes on this thing, and some honest feedback.
I have put this thing up on Github, so please go through and scrutinize this.
https://github.com/varchild/hailo-bank-switching-bridge
Thank you!!