Title: Liberating AI from Token-Based Models: Embracing Large Concept Models that Savor Sentences and Love Ideas

Title: Liberating AI from Token-Based Models: Embracing Large Concept Models that Savor Sentences and Love Ideas

In this article, we delve into an intriguing innovation in the realm of AI and large language models (LLMs). Instead of nerding out on individual words, as is the norm with most LLMs, let's explore the notion of focusing on whole sentences as our primary unit of measure. This groundbreaking shift would drastically alter how we approach generative AI and LLMs, transitioning from old-school large language models to cutting-edge large concept models (LCMs).

The AI community is grappling with the question of whether we're treading on repetitive ground with our current architecture successes. While some cling to the mantra, "If it ain't broke, don't fix it," others are convinced that we're hitting the top of what contemporary LLMs can achieve, with not much gas left in the tank before reaching a brick wall.

Rather than sticking with the conventional tokenization approach, where you provide words to AI, and it returns words—granted, via tokenization—why not consider an alternative? What if rid ourselves of the word-by-word focus and view a sentence as a whole, treating it as a representative of concepts?

This concept-based approach focuses on identifying and dealing with underlying concepts instead of words and sentences per se. To achieve this, enter a sentence, the concept encoder identifies the implied concepts, feeds those into an LCM for processing, and produces a set of resulting concepts. These concept-heavy internal outputs then get converted back into sentences for user presentation.

Fascinatingly, this new approach takes advantage of a multi-dimensional structures called "concepts-only spaces," which relates concepts in numeric formats to other concepts in numeric formats. This amazing feature enables AI to switch easily to your preferred language, without the need for tweaking its programming, as it's working with concepts rather than words.

Curious about the workings of LCMs? Check out this groundbreaking research paper – "Large Concept Models: Language Modeling in a Sentence Representation Space." The research team lays out the salient features and advantages of this captivating concept-based approach.

So ya, it's a different kind of trip from the familiar word-at-a-time focus. But why not venture outside the box, be open-minded, and embrace new ideas and novel AI approaches? As legendary scientist Albert Einstein once sagely observed, "Creativity is intelligence having fun." Let your imagination run wild—just remember to keep a modicum of practicality in tow. And as the honorable Mr. Einstein also advised, "Creativity is contagious. Pass it on."

1. This shift towards focusing on whole sentences as the primary unit of measure in AI and large language models (LLMs) might lead to the development of more advanced generative AI and LLMs, transitioning from traditional large language models to novel large concept models (LCMs).
2. The innovation in AI and LLMs, where concepts are identified and processed through a transformer encoder decoder architecture, could potentially bypass the limitations of contemporary LLMs, providing a new pathway for the AI community.
3. Anthropic, Claude from Google's Gemini, Meta's Llama, Microsoft's Copilot, and OpenAI's ChatGPT (including GPT4-o, o1, o3, and pro plus) are examples of large language models (LLMs) that implement different approaches to language understanding and generation, all striving to offer improved performance and novelty.
4. The concept-based approach to LLMs, allowing developers to easily switch between languages without changing programming, is made possible by exploiting the structure of "concepts-only spaces" in AI, a new frontier in the field of artificial intelligence and generative AI.

Read also: