Debugging Embedded Devices at Scale: Effective Techniques for Diagnosis and Resolution

In this presentation, I’ll walk through effective approaches for detecting, diagnosing, debugging, and resolving issues in embedded firmware and devices deployed at a large scale, such as in populations of hundreds of thousands or millions. While much has been written on monitoring smaller fleets, typical strategies like onsite debugging, debugger-based diagnosis, and manual log analysis can fail when dealing with massive populations of devices.

The presentation focuses on low-level debugging techniques like fault handling and exception parsing on the device, as well as a novel approach to capturing core dumps on Cortex-M MCUs. The presentation provides guidance on collecting core dumps and diagnosing crashes and faults without a debugger. I’ll also explore the changes in developer behavior that can be implemented when core dump functionality is integrated into firmware, and dumps are received centrally. 

This presentation includes content that is often overlooked by online resources, which assume that firmware works flawlessly and bugs are not introduced by developers. We know this is not the case and have developed these strategies to offer real-world solutions for new and experienced firmware engineers who are eager to tackle the challenges of debugging at scale.

Go to Session

Live Q&A - Debugging Embedded Devices at Scale: Effective Techniques for Diagnosis and Resolution

Live Q&A with Tyler Hoffman for the theatre talk titled Debugging Embedded Devices at Scale: Effective Techniques for Diagnosis and Resolution

Go to Session

Memfault Demo

The video showcases Memfault, a powerful device reliability platform that empowers hardware companies and firmware engineers to monitor their devices' behavior in the field, from just a few devices to millions. The platform features an intuitive crash analysis tool that mimics the experience of a debugger connected over JTAG to a device. With Memfault, developers can efficiently diagnose issues such as crashes, asserts, performance problems, and connectivity issues.

Go to Session

Essential Device and Firmware Metrics

As embedded engineers, we love data. Our desk is littered with tools that help capture tons of data, such as oscilloscopes, logic analyzers, debuggers, tracers, and power meters. However, once a device (or thousands) leave our desk and are shipped to customers, all of these tools are paperweights. It's now up to the devices to report issues back to the developers.

This is where metrics come in. Throughout my career, metrics have been the most powerful and simplest way to monitor thousands to millions of devices. This talk covers what metrics are, how to capture them, and explores the seemingly infinite metrics you can capture and creative uses to help you solve real-world, elusive device issues, such as power consumption, performance, and battery life issues.

Go to Session

Live Q&A - Essential Device and Firmware Metrics

Live Q&A with Tyler Hoffman for the talk titled Essential Device and Firmware Metrics

Go to Session

The Best Defense is Offensive Programming

Let's face it. The firmware we write has bugs, and we need to defend against and accept failures when our system experiences bugs. This is when we typically employ defensive programming practices, but if not used correctly, they may cause more problems than they solve.

There is an alternative and complimentary set of programming practices, often called "offensive programming", which takes defensive programming and flips it on its head. Instead of defending against errors in firmware, this technique will surface them immediately using liberal asserting and proper fault handling.

This talk will cover offensive programming techniques, prerequisites to implementing the suggestions in your firmware, and how you can use them to quickly and easily track down and root cause 1 in 1,000 hour bugs and keep your sanity at the same time.

Go to Session

Live Q&A - The Best Defense is Offensive Programming

Live Q&A with Tyler Hoffman for the talk titled The Best Defense is Offensive Programming

Go to Session