We held our next tinyML Talks webcast. Blair Newman and Danil Zherebtsov from Neuton.AI and Tamas Daranyi from Silicon Labs presented Buttonless Remote Control - Reproduce on your device on October 10, 2023.
IMPORTANT: Please register here
Edge Devices or Always-on devices often possess limited functionality, memory, and battery life that do not meet business requirements. This forces developers to find a balance between data analysis, functionality, complexity, energy capacity, and consumption. At Neuton.AI, we have developed an innovative approach to help people create compact neural networks that can recognize complex activities with minimal memory and energy consumption. In this tutorial, you will learn how to enhance the ability of users to control consumer electronics devices with just hand gestures and how to create a similar TinyML solution of only 4Kb in total footprint by yourself.
With a universal gesture-based remote control, you can easily access and control any Bluetooth-enabled media system or presentation slides without physical contact. Neuton’s Gesture Recognition Model leveraging Silicon Labs xG24 Dev Kit for EFR32MG24 Wireless SoC can recognize eight different types of gestures with almost 99% accuracy, including swipe right, swipe left, double tap, double knock, clockwise rotation, counterclockwise rotation, idle, and an unknown class. Moreover, the Inference time for this model is less than 2.3 ms, and it has a memory footprint of only 4.2 KB in Flash and 1.4 KB RAM consumption.
This webinar will include:
- Deep dive into the solution creation process
- Different TinyML use cases’ overview
With strong tech expertise and 20+ years of leadership experience, Blair Newman provides unprecedented insights into the future of AI’s development and use. As Neuton’s CTO, Blair is engaged in overseeing our business solutions as well as ensuring high-quality services are delivered to our clients. Prior to Neuton, Blair held various leadership roles at T-Systems North America, a Division of Deutsche Telekom, being responsible for providing strategic direction and leadership in the areas of Dynamic Services (Cloud Computing), SAP Hosting, Application Operations, Managed Hosting and Infrastructure Services.
Full-stack machine learning engineer with over 8 years of experience in the field. Before joining the Neuton team in 2018, he executed various end-to-end complex machine learning projects in multiple domains: telecom, networking, retail, manufacturing, marketing, fraud detection, oil and gas, engineering, and NLP. As a head of Machine Learning & Analytics at Neuton, Danil is working on the development of an automated TinyML platform facilitating sensor and audio data processing. Danil is an active contributor to the open-source community, with over 300,000 active users of tools featured in his repository. As an inspired writer, Danil publishes articles popularizing data science and introducing new methodologies for solving engineering tasks.
Tamas Daranyi, Product Manager, responsible for driving AI/ML initiatives at Silicon Labs. He holds an MSc, Electrical Engineering. He has working experience in various IoT focused projects in the past 10+ years ranging from the IoT end-node up to cloud service product development. Prior to Silicon Labs he had engineering and product management roles at GE, LogMeIn (Xively IoT) and Google Cloud IoT, CloudAI.
=========================
Watch on YouTube:
Blair Newman, Danil Zherebtsov and Tamas Daranyi
Download presentation slides:
Blair Newman, Danil Zherebtsov and Tamas Daranyi
Feel free to ask your questions on this thread and keep the conversation going!
=========================
In a system, where is tinyML running at the edge or at the base controller or both?
It can run in either of locations you’ve mentioned. Neuton models are designed to fit directly on the edge.
Does tinyML have OpenCV features or we want to have them co-exist on the MCU for vision based controllers/apps?
Currently Neuton.ai is not working with image or video data.
I am assuming new gestures can be programmed in b y end users? Can you speak to this please - thank you
New or custom gestures require collecting the sensor data for these specific gestures and labeling these samples accordingly. Then raw data needs to be cleaned, processed and fead to the Feature Extraction pipeline with a predefined window. After the features are extracted - you can consider the training data to be ready for machine learning. Model training comes next and as a final step this model has to be embedded into the MCU for testing/inference. Neuton.ai executes the full cycle of these activities.
Is this model as accurate for both right-handed and left-handed gestures?
The model in the repository is meant to be used with your right hand. Left/right hand inference is certainly possible, you only need to collect a sufficient amount of data representing left/right hand examples of each gesture.
Kb is kelvin bit - let’s assume they mean kB for kilo byte.
Kilo Bytes - that’s right.
How much power does the solution consume during a gesture activity?
It is different for every specific device. Subjectively, given that it consumes less than 1% of CPU resource, you can consider that your overall MCU utilization with whatever business logic you have programmed, will include <1% overhead for such model operation.
Very cool demos.
Thanks for the kind words.
Can you explain the axis of the Data Cleaning graph?
X is the index of the signal, Y is the value of the signal.
'@Danil - what about people gesturing in a slow manner? They might take longer to finish the gesture. Can the system auto-adjust to understand the user behavior?
A production grade model should have multiple various examples of different people executing activities. Of course the model can take that into account. It all comes down to precise training data collection.
Where is the training done. On Xg24… , cloud or somewhere else
Training is done in the Google Cloud.
Can these models be run on raspberry pi pico?
Sure, these models can run on any MCU which can run C code.
What kind of ML/AI algorithms are used? just Neural Networks or also other ML solutions?
The underlying algorithm is a proprietary Neural Network architecture developed in the Neuton labs. It is not open-source. Our framework was designed specifically for application on the EDGE; with this in mind we have developed an algorithm that produces the tiniest models possible out of the box without a requirement of any post-training compression techniques.
Practically how long can the MCU work on a single battery? Has this been tested and checked?
This is subject to many different factors: the type of the MCU, the amount of operations it is programmed to execute, the utilization of this MCU in operation, the amount of power, the clock speed, sleep time, etc. There is no rule of thumb to say how long will a particular MCU work on battery. This can only be tested after all the mentioned conditions are known.
You mentioned the model is proprietary. Does it mean it’s provided as a blackbox? Is there any technical info provided, compatibility with popular ML frameworks?
When working on custom use cases with our clients our final model that goes into the device is provided as a source code.
Great demo, impressive software.
Thank you!
Can the gesture model be made PCB position independent, or is the orientation always critical?
There are techniques of data collection and feature engineering to compensate for changing sensor positions. This is not a problem.
Is there a rule behind selecting different hand gestures as different classes?
A rule of thumb: this gesture should be distinctive and different from your random movements. Another story if you want to classify say hand moving up - then the training samples should include this generic activity.
With a closed model how are development activities such as debug, incremental improvements accomplished?
Our customer engagement model accounts for all the engineering work including debugging and improvements to the point when it is acceptable for production grade deployment. When our engineers develop the model which satisfies the defined requirements - it does not need further improvements.