Home > Tracks > DSP & Machine Learning

The Past, Present, and Future of Embedded Machine Learning

Pete Warden - Watch Now - Currently watching: 0

Pete Warden, from Google's TensorFlow Lite Micro project, will be talking about how machine learning on embedding devices began, and where it's heading. ML has been deployed to microcontrollers and DSPs for many years, but until recently it has been a niche solution for very particular problems. As deep learning has revolutionized the analysis of messy sensor data from cameras, microphones, and accelerometers it has begun to spread across many more applications. He will discuss how voice interfaces are leading the charge for ML on low-power, cheap devices, and what other uses are coming. He'll also look into the future of embedded machine learning to try to predict how hardware, software and applications will be evolving over the next few years.

M↓ MARKDOWN HELP
italicssurround text with
*asterisks*
boldsurround text with
**two asterisks**
hyperlink
[hyperlink](https://example.com)
or just a bare URL
code
surround text with
`backticks`
strikethroughsurround text with
~~two tilde characters~~
quote
prefix with
>

EEngstrom77
Score: 1 | 3 days ago | 1 reply

Pete we are glad to have you part of the Embedded community! Thank you for the gift of TinyML. I at times experienced some irrational guilt over sensor data that either was not collected or collected and having no idea what to do with it. ML still looks like a black box to me but excited to see how we can use it with our embedded sensor to revision the world around us.

petewardenSpeaker
Score: 0 | 3 days ago | no reply yet

Thanks for your kind words! Embedded ML can be a lot of fun, so I look forward to seeing what projects you create with it.

Pi
Score: 1 | 4 days ago | 1 reply

You mention how math-intensive these algorithms are, and I'm reminded of this article from DSP Related about how DSPs are perfectly suited for ML but chip manufacturers failed to capture that market.
What, if any, DSPs have optimized libraries for TinyML (or similar platforms)? Do you think they have an advantage over ARM and other general purpose chipsets, or is it application specific?

janjongboom
Score: 1 | 3 days ago | no reply yet

I'm not Pete, but I saw that this one went answered. We're starting to see them in the market now, e.g. Eta Compute ECM3532 has a separate DSP where they run their neural networks, which should be available publicly this year.

alexanderentinger
Score: 2 | 5 days ago | 1 reply

Arduino listening in and appreciating the friendly mention ;)

petewardenSpeaker
Score: 0 | 5 days ago | no reply yet

Happy to help spread the word, it's been great collaborating!

DaveN
Score: 2 | 5 days ago | 1 reply

Please recommend a good platform for experimentation (micro eval board with audio and enough processing/ram/etc)?

petewardenSpeaker
Score: 0 | 5 days ago | no reply yet

I recommend picking up an Arduino Nano BLE 33 Sense, since it has a microphone and IMU, is easy to program, and only costs $30. I'm using that for workshops myself, since it's easy to show people how to get running.

DaveN
Score: 3 | 5 days ago | 1 reply

Is training on embedded hopeless?

petewardenSpeaker
Score: 0 | 5 days ago | no reply yet

Not at all! That is a whole different talk that I need to do, but the summary is that there are all sorts of possible ways of training. The simplest is to just "personalize" by tuning the results of the last layer to match, for example, a particular user's voice on a wakeword model. Transfer learning can also be very powerful. Full backprop training isn't usually that useful on embedded models though, because you generally need a lot of labeled data, which you often don't have.

deltronics
Score: 1 | 5 days ago | 2 replies

Please do you know of any benchmarking applications and reference results for Tensorflow Lite for Microcontrollers i.e similar to benchmark_model from TFlite? Also, are there any Tensorflow Lite for Microcontroller models in place similar ( probably customized or cutdown) to existing standard models such as mobilenet, alexnet etc.

lorenzorizzello
Score: 0 | 5 days ago | 1 reply

hi, hope you don't mind if I join the conversation. I'll let Pete reply about benchmarking, but concerning standard models it depends on model size and type of operations (here the list of operations currently supported by TF lite micro). I have successfully used a pre-trained mobilenetv1, if you want, you can check my talk where I talk a bit about it.

deltronics
Score: 1 | 4 days ago | no reply yet

Thanks , I will check your talk.

petewardenSpeaker
Score: 0 | 5 days ago | 1 reply

We have started on benchmarking, you can see the initial two models we're focused on at https://github.com/tensorflow/tensorflow/tree/master/tensorflow/lite/micro/benchmarks
One of them is a MobileNet v1 0.25 model for person detection, and we also have that in the examples folder.
We are working with the MLPerf group to expand and formalize the benchmarks, and we'd welcome help if you're interested.

deltronics
Score: 0 | 4 days ago | no reply yet

I had a quick look at this in the past but thought they were customized speech models for TLFM. I will have another look. Thank you.

PatrickMcDaid
Score: 4 | 5 days ago | 2 replies

Think your webcam may have frozen. didn't get to see that cat Dude. ;-)

petewardenSpeaker
Score: 0 | 5 days ago | no reply yet

I realized that, I think his cuteness overloaded the camera! :)
Here's an old video that's all about him though: https://petewarden.com/2014/04/08/how-to-add-a-brain-to-your-smart-phone/

ThomDenholm
Score: 0 | 5 days ago | no reply yet
This post has been deleted by the author
nicastellanos
Score: 1 | 5 days ago | 1 reply

Hi Peter, thanks for the session! Whats your opinion on the future of embedded applications in an Reinforcement Learning space. Do you see a possibility of using these technologies for edge RL agent systems and if so what advancements do you think will need to be made to make these systems possible. Thanks!

petewardenSpeaker
Score: 1 | 5 days ago | no reply yet

To be honest, I've not used RL myself, mostly because it seems so hard to get working well! I've seen good results in papers, but colleagues who've dabbled in it have found it very "touchy" and sensitive to initial conditions. The underlying math seems possible to support on embedded devices, but I've not focused on that area because getting the models working well for practical problems seems so tough. I'd love to hear about others' experiences in this area though.

JeanGoulet
Score: 3 | 5 days ago | 1 reply

Interesting point of view on how the Internet and Wifi in IoT, is a burden for the battery. I wonder to what extent 5G will change that (make it worse? better?). And yes indeed how comical yet sad that so many simple devices fail to operate (motion sensing lights that are worthless even though people are present; no reaction from so many defective motion-sensing faucets and soap dispensers and hand dryers). Are they mostly because of dead batteries? Or mostly terrible products? Hopefully, ML products of high quality, high intelligence, high longevity, and low cost, will one day change little things, and big things, for the better.

petewardenSpeaker
Score: 0 | 5 days ago | no reply yet

That's a great perspective, I am definitely interested in improving the everyday experience of users, and I do think ML can be more reliable than existing methods for things like faucets and lights. It will be up to us to make sure we get the engineering right of course, since ML's no magic bullet, but at least it gives us new options.

OUR SPONSORS