tinyML Talks on January 16, 2023 “Lightweight Neural Network Architectures” by Andrii Polukhin from Data Science UA

We held our next tinyML Talks webcast. Andrii Polukhin from Data Science UA presented Lightweight Neural Network Architectures on January 16, 2023.

Have you ever wondered what makes neural networks, like MobileNetV3, FBNet, and BlazeFace, so special? These networks may be found in commonplace items like our phone or TV.
Their primary challenge—and area of interest—lies in the efficient creation of a neural network for low-power devices.

This lecture will cover the following topics:

  • the rationale for the design of layers like Fire module and Squeeze-and-Excitation;
  • the best methods for determining the number of model parameters, width, and depth of architecture layers, such as EfficientNet or Model Rubik’s Cube algorithms;
  • the SOTA solutions that will enable you to accelerate and optimize certain layers of your neural network.

Those who want to understand how to create a lightweight, effective neural network will find this session to be interesting.

The main areas of Andrii’s interests are the research of Deep Learning architectures, methods of training deep networks, intuition and mathematical theorems behind it, interaction of AI with the surrounding world.

Watch on YouTube:
Andrii Polukhin

Download presentation slides:
Andrii Polukhin

Feel free to ask your questions on this thread and keep the conversation going!

There are a few publications as a survey list for efficient and lightweight architectures for edge-devices:

Tay, Yi, Mostafa Dehghani, Dara Bahri, and Donald Metzler. “Efficient Transformers: A Survey.” arXiv, March 14, 2022.

Zhao, Tianming, Yucheng Xie, Yan Wang, Jerry Cheng, Xiaonan Guo, Bin Hu, and Yingying Chen. “A Survey of Deep Learning on Mobile Devices: Applications, Optimizations, Challenges, and Research Opportunities.” Proceedings of the IEEE 110, no. 3 (March 2022): 334–54.

Treviso, Marcos, Tianchu Ji, Ji-Ung Lee, Betty van Aken, Qingqing Cao, Manuel R. Ciosici, Michael Hassid, et al. “Efficient Methods for Natural Language Processing: A Survey.” arXiv, August 31, 2022.