In a global context of task automation, artificial neural networks are currently used in many domains requiring the processing of data from sensors: vision, sound, vibration.
Depending on different constraints, the information processing can be done on the Cloud (SIRI, AWS, TPU) or in an embedded way (NVidia's Jetson platform, Movidius, CEA-LIST's PNeuro/DNeuro). In this second case, many hardware constraints must be taken into account when dimensioning the algorithm. In order to improve the porting on hardware platforms, LIST has developed innovative state-of-the-art methods allowing to aggressively quantize the parameters of a neural network as well as to modify the coding of the activations to reduce the number of calculations to be performed.
The energy efficiency of neuromorphic architectures with equivalent technology is constrained by the classic paradigm of flexibility vs. efficiency. In other words, the more different tasks (and networks) an architecture is capable of performing, the less energy-efficient it becomes. While this relationship cannot be circumvented for a wide variety of algorithms, neural networks are parametric functions, learned for one and therefore potentially adaptable to other tasks by partial modification of the topology and/or parameters.
One technique, CLIP, seems to provide an answer, with a strong capacity for adaptation to a variety of tasks and the possibility of using multimodality. In its original form, CLIP is presented as a method for matching text and images to create a classification task.
The aim of this thesis is to study the hardware implementation of CLIP by proposing a dedicated architecture. The thesis is organized into 3 main phases, beginning with a study of CLIP's mechanisms, the operations to be performed and the consequences for embedding networks. Secondly, hardware optimizations applicable to CLIP, such as quantization (or others) and an estimation of flexibility vs. applicative generality. Finally, an architectural and implementation proposal to measure energy efficiency.