Home > Tracks > DSP & Machine Learning

Get Started with TinyML

Jan Jongboom - Watch Now

TinyML is opening up incredible new applications for sensors on embedded devices, from predictive maintenance to health applications using vibration, audio, biosignals and much more! 99% of sensor data is discarded today due to power, cost or bandwidth constraints. 

This webinar introduces why ML is useful to unleash meaningful information from that data, how this works in practice from signal processing to neural networks, and walks the audience through hands-on examples of gesture and audio recognition using Edge Impulse.

What you will learn:

  • What is TinyML and why does it matter for real-time sensors on the edge
  • Understanding of the applications and types of sensors that benefit from ML
  • What kinds of problems ML can solve and the role of signal processing
  • Hands-on demonstration of the entire process: sensor data capture, feature extraction, model training, testing and deployment to any device
M↓ MARKDOWN HELP
italicssurround text with
*asterisks*
boldsurround text with
**two asterisks**
hyperlink
[hyperlink](https://example.com)
or just a bare URL
code
surround text with
`backticks`
strikethroughsurround text with
~~two tilde characters~~
quote
prefix with
>

Doini
Score: 0 | 3 months ago | no reply

Thank you very much for the nice presentation!

KVas
Score: 0 | 4 months ago | no reply

Thanks for the great session and the great platform... definitely triggers my interest more.. Will definitely try this... I have been working in embedded (automotive) for long.. so any tips from you as to how to delve more into ML + embedded world?

msaur
Score: 1 | 4 months ago | 1 reply

Thanks for that great presentation.
A question which comes to my mind is the obsolescence of trained model on embedded devices.
Many of the the embedded devices out other will Not have reprogramming ability, living there on their own, sometimes for several years (product lifetime). And I am talking about the kind which are treated like off the shelf devices (say electronic relays).
While this is exactly why you said that they need to be intelligent - but the longer their lifetime, the more the new use cases/system configurations in which they would be fit in future. From business pov, how can we manage the problem of training model getting obsolete?

janjongboomSpeaker
Score: 1 | 4 months ago | no reply

Very interesting question. For devices that'll never be updated in the field you'll have a similar approach as you do today with rigid testing before deploying, you'll just have more finegrained thresholds and conclusions which your ML model can determine. For devices that have some binary delivery mechanism there are many more possibilities. One interesting thing about TensorFlow Lite is that it can decouple the runtime from the model definition, so you can update the neural network without recompilation, making it easy to iterate on new models. But regardless, this is embedded, so we should always be careful about updates.

erikm
Score: 0 | 5 months ago | no reply

Great talk! I'll have to try this!

pran0506
Score: 0 | 5 months ago | 1 reply

Hi Jan,
Amazing presentation! I got motivated to learn & work in this field.
I'm just a fresher in the industry, recently graduated (Class of 2020) . Is there any suggestions/guidance you can provide to learn ML implementation on Microcontrollers ?

Thank you so much for the motivation!
Regards

janjongboomSpeaker
Score: 1 | 5 months ago | 1 reply

In order: 1) walk through the tutorials on vibration and audio analysis (https://docs.edgeimpulse.com/docs/continuous-motion-recognition), 2) For a deeper understanding, read the TinyML Book (https://tinymlbook.com) - written by Dan Situnayake and Pete Warden, 3) If this still interests you I'd consider diving deeper into ML itself. E.g. with https://www.coursera.org/learn/machine-learning (free course by Andrew Ng).

pran0506
Score: 0 | 5 months ago | no reply

Thank You so much sir!! Will follow these! :)

Miro
Score: 1 | 5 months ago | 1 reply

Excellent and highly motivating presentation. I've learned a lot. Thank you.

janjongboomSpeaker
Score: 0 | 5 months ago | no reply

Thanks Miro!

DrewFustini
Score: 1 | 5 months ago | 2 replies

What is CMSIS?

Can this work with Cortex M0?

What is minimum RAM for CMSIS-NN?

EEngstrom77
Score: 1 | 5 months ago | no reply

Christopher Seidl explains CMSIS in his "How to get started with Arm Cortex-M55 software development" session this AM.
https://www.embeddedonlineconference.com/session/How_to_get_started_with_Arm_Cortex-M55_software_development

janjongboomSpeaker
Score: 0 | 5 months ago | no reply

Hi Drew, CMSIS is a vendor-independent hardware abstraction layer for microcontrollers developed by Arm. CMSIS-NN is the library for doing things that many neural networks depend on, like 8-bit matrix multiplication, efficiently in hardware. This will work on any Cortex-M MCU, but naturally processing speed depends on the underlying MCU. It's probably possible to do vibration analysis (like the gesture demo) on an M0 - that demo takes ~4K of RAM at the moment (but I don't have any numbers for inference speed on smaller MCUs), but larger models like audio / still image processing will require a Cortex-M4 and up.

MarkusADH
Score: 0 | 5 months ago | 1 reply

Awesome!
Do you have tools/Interfaces to easily incorporate CI/CD to this?
In your session you used about 13 minutes of training for a quite good result. What is the low end normal amount of data to get a good accuracy?

Thanks!

janjongboomSpeaker
Score: 0 | 5 months ago | no reply

Hi Markus, you can use the Edge Impulse API (https://docs.edgeimpulse.com/reference-link/edge-impulse-api) to automatically retrain your project, and then run all samples in the test set against the new model. This is what some of our users use in their CI pipeline. New data comes in, automatically retrains, and if performance drops they'll get an alert.

Training data depends on your usecase. E.g. keyword spotting requires a lot more data, but simple audio models (scene recognition) can be done in about 20 minutes (e.g. https://www.edgeimpulse.com/blog/audio-based-shower-timer-with-a-phone-machine-learning-and-webassembly/). More is always better though! For one of our customers we have thousands of hours worth of data.

Khalil.Rashid
Score: -1 | 5 months ago | 1 reply

What are cost model for Impulse Studio?

janjongboomSpeaker
Score: 0 | 5 months ago | no reply

Free for individual developers, SaaS model for enterprises. The latter gives you access to the dataset builder tools, to team collaboration, and allowing you to store datasets on your own servers. All code that is generated will always be royalty-free, no BOM stacking regardless of whether you're using the free or enterprise plan, and no per-device fee (enterprise plan is billed per project/month).

Justin
Score: 3 | 5 months ago | 1 reply

Nice talk Jan! Was that your gran at the end? XD
I agree the ML hype is real, as the market has seen an explosion of applications to a variety of sectors both commercial and industrial.
By the way, do you have any interesting industry success stories, any stellar feedback from companies who got so much more from their $40?
Also, about the EI (Edge Impulse) dev GUI, I haven't tried it yet but is it possible to export the entire learning algo. via Python? It sometimes needs a little tweeking depending on the nature of the time-series data, adjustments on parameters in PCA and LDA (something of the sort..) Would be awesome to get access to the raw code (we could just run the modified one through our local Python IDE)

Speaking of contributing to the pool of knowledge in ML, does EI need to adhere to any existing standards? Given there are so many ML implementation out there, I'm not familiar if there are any..
Also really interested to know why 'Edge Impulse' (there must be an interesting background behind coining the company name hehe)

I'm really glad EI removed all the Math involved in ML training to make things more comprehensive (and not just for data scientists). Thanks so much for your wonderful talk!

janjongboomSpeaker
Score: 1 | 5 months ago | no reply

Hi Justin, thanks a lot! No, not my real grandma ;-)

By the way, do you have any interesting industry success stories, any stellar feedback from companies who got so much more from their $40?

The larger players are doing very similar things as we are applying here, mostly around voice (Amazon, Google, Apple) and they see lots of value. But the movement outside of these players has just started up (TensorFlow Lite for Microcontrollers is barely a year old, and CMSIS-NN has been usable for maybe a month or 3) so I think we're still a bit off of large scale success for ML there. We are doing some very promising deployments with customers, but before they have reached scale we'll be some time out.

Also, about the EI (Edge Impulse) dev GUI, I haven't tried it yet but is it possible to export the entire learning algo. via Python? It sometimes needs a little tweeking depending on the nature of the time-series data, adjustments on parameters in PCA and LDA (something of the sort..) Would be awesome to get access to the raw code (we could just run the modified one through our local Python IDE)

Yeah, absolutely. All our signal processing code is here (https://github.com/edgeimpulse/processing-blocks) in Python, and here in C++ (https://github.com/edgeimpulse/inferencing-sdk-cpp). Neural network code is in Keras (just click on the little dots, and select Edit as iPython notebook to get something to run locally). You can always get data from intermediate steps (e.g. after DSP but before Neural Network) from the Dashboard page in .npy files.

Speaking of contributing to the pool of knowledge in ML, does EI need to adhere to any existing standards? Given there are so many ML implementation out there, I'm not familiar if there are any..

Yeah, on the ML side we try to use as many standard things as possible. We use Keras & TensorFlow for training, TensorFlow Lite for inferencing (we're a contributor, and our lead TinyML engineer used to be in the TF team at Google), CMSIS-NN and CMSIS-DSP for hardware optimization, etc. Trying not to reinvent the wheel here, we're a small team :-)

I'm really glad EI removed all the Math involved in ML training to make things more comprehensive (and not just for data scientists). Thanks so much for your wonderful talk!

Awesome, thanks!

Yunus
Score: 0 | 5 months ago | 1 reply

Hi. Thanks for great Session.
Have you ever worked with very low sample amount of sensor(accelerometer) data? E.g. sensor is sampling at 100 Hz and you are detecting quick movements with ~20 samples (200 ms). For example that movement might be part of your up-down movement (or just up) done within 200ms.
Can ML be useful in these very low sample amount cases? Or is it even wise to work with that kind of data?

janjongboomSpeaker
Score: 0 | 5 months ago | no reply

Hi Yunus, difficult question. Some customers are doing things with much higher frequency data (e.g. vibration analysis at 1000Hz) and then smaller samples are enough to accurately look at discrepancies (if you have something that rotates at 400Hz you can get an idea that way). I'd suggest to grab some data together and try it out (it's very little data after all!).

alexanderentinger
Score: 0 | 5 months ago | 1 reply

Well thank you for the talk! ;) Great introduction in what you are doing. Loved the Arduino Nano 33 BLE there (Full Disclosure: I work for Arduino ;) ).

janjongboomSpeaker
Score: 0 | 5 months ago | no reply

Thanks! Yeah, it's a supernifty little board, and we're going to be doing a lot more with it in the near future!

Norwood
Score: 0 | 5 months ago | 1 reply

Thanks and great session! What training tool/utility do you hook into, if not something custom? Also, what model did you use for the AXL demo?

janjongboomSpeaker
Score: 0 | 5 months ago | no reply

For the feature extraction we use FFT + Power Spectral Density based on Welch's method. See the code here: https://github.com/edgeimpulse/processing-blocks, then paired with a small fully connected neural network (2 hidden layers of 20 and 10 neurons) trained with TensorFlow.

janjongboomSpeaker
Score: 3 | 5 months ago | 1 reply

Hello world! Let me know if you have any questions, I'll be here from 7AM-9AM EDT / 1PM-3PM CET! Slides are available from the left!

AlexMayer
Score: 0 | 5 months ago | no reply

Thank you for this very interesting talk on TinyML and Edge Impulse! Now if you'll excuse me, I'll take a dive into that Webinar on TinyML in your blog :-)

PatrickMcDaid
Score: 0 | 5 months ago | 1 reply

Congratulations Jan. Great lecture, great demo. Great start to ESC Online.

janjongboomSpeaker
Score: 0 | 5 months ago | no reply

Thanks!

OUR SPONSORS