We held our ninth tinyML Talks webcast with two presentations: Unmesh Kurup from LG Electronics America Research Lab has presented A weight-averaging approach to speeding up model training on resource-constrained devices and Brandon Rumberg from Aspinity has presented Analog ML Is Relevant—Because Most Sensor Content Isn’t on June 23, 2020 at 8:00 AM and 8:30 AM Pacific Time.
Unmesh Kurup (left) and Brandon Rumberg (right)
Training machine learning models on edge devices has definite advantages for security, privacy and latency. However, techniques such as Deep Neural Networks (DNNs) are unsuitable given the resource constraints on such devices. Optimizing DNNs is especially challenging due to the nonconvex nature of their loss function. While gradient-based methods that use back-propagation have been crucial to neural network adoption, optimal convergence of the loss function is still time-consuming, volatile, and needs many finely tuned hyperparameters. One key hyperparameter is the learning rate. A high learning rate can produce fast results faster but at the increased risk of the model never converging. In this talk, I explain one of the advances from our lab that show that by manipulating the model weights directly using their distributions over batch-wise updates, we can achieve significant intermediate improvements in training convergence, and add more robustness to the optimization process with negligible cost of additional training time. More importantly, this approach allows deep neural networks to be trained at higher than usual learning rates resulting in fewer epochs which reduces resource use and allows for lower total training time.
Unmesh Kurup is Senior Manager, AI in the Advanced AI group at LG Electronics. His current research focus is in the training and deployment of machine learning models for edge devices. Prior to joining LG, he was a Senior Data Scientist at Robert Bosch Research Technology Center where he developed and lead multiple data science projects in the areas of Energy, Healthcare, and ADAS. He has been working at the intersection of AI and ML for over 20 years.
Power has always been a challenge for battery-operated always-on devices, and with additional privacy and communication requirements pushing more data processing to the device, power is an even more critical constraint. Aspinity will discuss how analog ML cultivates an always-on system architecture that mimics the brain’s ability to use a small amount of energy up front to determine which data are important before committing higher power resources to further analysis. This approach allows designers to partition more efficient always-on systems that determine which sensor data are important while the data are still analog and subsequently eliminate the digitization and higher-power analysis of irrelevant data that will simply be thrown away.
Over the last decade, Brandon Rumberg has focused on the full stack of low-power sensing technologies, spanning integrated circuit design, embedded systems and signal processing, and software development kit creation and system integration. These combined skills provided the foundation for his new architectural approach to solving the power, size and cost issues with always-on higher-bandwidth signal-processing devices. Brandon holds multiple patents, has developed and taught three engineering courses, and has authored 20 publications—one of which earned him a Best Paper Award at the International Symposium on Quality Electronic Design, 2015. He received Ph.D., M.S., and B.S. degrees in Electrical/Computer Engineering from West Virginia University.
Feel free to ask your questions on this thread and keep the conversation going!