NEC claims the NEC cotomi Pro and NEC cotomi Light deliver the same high performance as global LLMs, but more than ten times the speed.

NEC explains that to improve the performance of an LLM, a model needs to be made larger, but slows down operating speeds.

In NEC’s case, it has improved speed and performance by developing a new training method and architecture.

NEC claims NEC cotomi Pro’s response time is approximately 87% faster than GPT-4 using an infrastructure of two graphics processing units (GPU).

It also claims NEC cotomi Light works like GPT 3.5-Turbo, but can process requests at high speed with an infrastructure of about 1 to 2 GPU.

Specifically, in an in-house document retrieval system using a technique called RAG, the system achieved a correct response rate higher than GPT-3.5 without fine-tuning and a response rate higher than GPT-4 after fine-tuning, with a response time that is approximately 93% faster.

This first appeared in the subscription newsletter CommsWire on 26 April 2024.