The aim of this thesis is to explore the execution of attention mechanisms for Artificial Intelligence directly within a cutting-edge Non-Volatile Memory (NVM) technology.

Attention mechanisms represent a breakthrough in Artificial Intelligence (AI) algorithms and represent the performance booster behind “Transformers” neural networks.
Initially designed for natural language processing, such as ChatGPT, these mechanisms are widely employed today in embedded application domains such as: predicting demand in an energy/heat network, predictive maintenance, and monitoring of transport infrastructures or industrial sites.
Despite their widespread use, attention-based workloads demand extensive data access and computing power, resulting in high power consumption, which may be impractical to target embedded hardware systems.

The non-volatile memristor technology offers a promising solution by enabling analog computing functions with minimal power consumption while serving as non-volatile storage for AI model parameters. Massive linear algebra algorithms can be executed faster, at an ultra-low energy cost, when compared with their fully-digital implementation.
However, the technology comes with limitations, e.g., variability, the number of bits to encode model parameters (i.e. quantization), the maximum size of vectors processed in parallel, etc.

This thesis focuses on overcoming these challenges in the context of embedded time-series analysis and prediction.
The key task is exploring the mapping of attention-based mechanisms to a spin-based memristor technology developed by the SPINTEC Laboratory.
This involves quantizing and partitioning AI models to align with the hardware architecture without compromising the performance of the prediction, and exploring the implementation of particular AI blocks into the memristor analog fabric.

This thesis is part of a collaboration between CEA List, Laboratoire d’Intelligence Intégrée Multi-Capteur, the Grenoble Institute of Engineering and Management and the SPINTEC Laboratory.
Joining this research presents a unique opportunity to work within an interdisciplinary and dynamic team at the forefront of the AI ecosystem in France, with strong connections to influential industrial players in the field.