Hello, today I’m releasing an important patch to the 1.4.0-BETA.

The most important part is the “low memory mode”. This is an option used on the server to reduce the RAM used by the CPU. This is an experimental feature that’s why I’m only releasing it in the BETA branch for now.

It divides by ~2 the RAM used for the AIs. Moreover, it’s not even slowing down the generation! See the result of my experiment at the end of this article.

How to update to the 1.4.0-BETA.1?

(The game should tell you by itself to run the installer)

• Download as usual
• Erase the files with the new ones
• Run the installer
• Update the environment

Limitations

• It doesn’t change anything when using the GPU.
• It doesn’t work with the KoboldAI’s models (yet)

Changelog

• Added the lorebook German localization
• Fixed the import of lorebooks not working.
• Implemented the low memory mode (You can disable it in the advanced options in the AI options)

Result of my tests

I ran multiples generation with gpt2-large. With my hardware running the AI I got the following results:

• Average generation time with the low_memory_mode ON: 6.01 seconds
• Average generation time with the low_memory_mode OFF: 6.62 seconds

As you can see the generation time on my device is almost the same. The good new is we don’t have any lose of performances.

Moreover, I checked the RAM used for a few models:

• gpt2-xl -> 6.5GB instead of 12.8GB
• gpt2-large -> 3.4GB instead of 5GB
• gpt2-medium -> 1.7GB instead of 3.3GB
• gpt2 -> 700MB instead of 1.2GB

While playing the server might use a few more hundred MB