Home > Meetings >

Live Q&A - Mars Perseverance Software

Steve Scandore - Watch Now - Duration: 58:22

Live Q&A with Steve Scandore for the keynote session titled Mars Perseverance Software
italicssurround text with
boldsurround text with
**two asterisks**
or just a bare URL
surround text with
strikethroughsurround text with
~~two tilde characters~~
prefix with

Score: 2 | 5 months ago | 1 reply

Fabulous presentation and Q&A Steve. You mentioned that wrt to coding you do not allow any developers to use semaphores. Just curious why that is the case. I have a hunch :-)

Steve SSpeaker
Score: 0 | 5 months ago | 1 reply

That's correct, the general use of semaphores (task locks in general) in the software is not allowed. This avoids some classic misuse and unexpected task dependencies (e.g.: inversions, deadlocks) in an architecture where we want tasks to be as independently operating and deterministic as possible. Using semaphores also complicates runtime analysis and testing in a system with processing deadlines. In short, we remove or cautiously use conventions which may question the operation of the code. In many cases we have easily redesigned code to avoid the casual use of a semaphore. Having said all that, we do have cases where waivers to this rule are granted. For example, IPC waits on message are implemented using a semaphore. By not allowing them, and then using waivers in the few places where they are really required helps ensure their overall safe use and operation of the system.
As I mentioned in the talk, we have learned a lot over the years. Here's a link to a related dependency problem from our past: https://www.youtube.com/watch?v=C2xKhxROmhA

Score: 0 | 5 months ago | no reply

Thanks Steve. I used to work on the avionics software for the F-16 at General Dynamics in the late 80's. Back then we had no RTOS, just a homegrown "cyclic executive". We weren't even allowed to pass parameters to a function because it took too long to push and pop data from the stack so everything was in global data. There was even a complete software team just to manage the global data. I am really glad to hear that VxWork is now used. I use to work with them at various contractors when I work for Rational Software. The Ada days. :-)

12:40:54	 From  Dave Nadler : Could you tell us a bit about how you tested the software?
12:41:10	 From  Keith J : Thanks for that Steve.  That was awesome to see how advanced things have become from a compute standpoint... I'm old enough to remember Apollo - although as a young kid.
12:41:18	 From  Tom.Davies : Awesome presentation
12:41:28	 From  Matjaž Finc : There is no room for error on such missions. How do you cope with the stress of "what if my code goes wrong" while developing and also during the mission? Which mission stage makes you the most nervous?
12:42:27	 From  Jeremy Schreiber : Awesome talk!  How large is the development team?  What type of development process (agile, waterfall, etc) do you follow to pull off a project of this size and complexity?
12:42:30	 From  Raul Pando : How much do you rely on Over The Air (Space) updates :)?
12:42:31	 From  Alex Burka : Can you comment any more on what went wrong with the first Ingenuity flights and what was the fix that "works 85% of the time"?
12:42:43	 From  Radu Pralea : C only? C++? All 100 tasks handled by a single core running at 200  MHz (<1% of computing power of a Raspberry Pi)?
12:42:47	 From  Matjaž Finc : Which QP kernel did you use? QKX?
12:42:48	 From  David : Was all imaging and other high data components passed over RS422 or 1553 or were there additional highers speed buses?
12:43:18	 From  Radu Pralea : *10%
12:43:18	 From  afwaanquadri : What framework did you use for state machines ?
12:44:13	 From  Jonnyvb : Following on from the stress of "what if my code goes wrong" question from Matjaz - what sort of processes do you go through when something does go wrong to stop that kind of issues happening again and to learn the lessons from it?
12:44:18	 From  David Potter : Can you talk about your top down architecture and associated documentation process?
12:44:47	 From  Dave Comer : my apologies in advance for the naïve question. I worked on the Galileo mission back in the 1980's How, or did, that mission help the current efforts on Mars?
12:46:12	 From  Radu Pralea : Do you use TDD? :)
12:46:36	 From  ken H : can you tell us how many person-hours went into software development? What percent was test/validation?
12:48:23	 From  Miro Samek : Very interesting that you mention the following practices used by NASA: event-driven architecture, threads structured as event-loops, blocking in one place only, NO blocking during message processing. These best practices are collectively known as the "Active Object" design pattern. Do you use this name ("Active Object") to quickly reference to your architecture?
12:48:23	 From  Davy Baker : If you could start over,  what would you do differently ?
12:49:35	 From  Alex : What was the biggest enabler (e.g. test bed/automated builds) for the firmware development?
12:49:51	 From  David Kanceruk : I imagine you use a build server. How long does it take to compile the code?
12:50:48	 From  afwaanquadri : Did you have any User-Interface to test specific modules of the software?
12:51:35	 From  Meenal Burrows : How big is the flight software team?
12:53:55	 From  Simon Voigt Nesbo : Was there anything that didn't work? That we wouldn't know just watching the news
12:54:06	 From  Gopinath : What caught my eye is how low the frequencies are in the system - processor running at 135 kHz, buses at 8 Hz, 64 Hz, etc. Is there a reason for this? EMI?
12:55:53	 From  Miro Samek : For anyone interested in the NASA software architecture used originally on the Pathfinder, which apparently is still very much influencing the current missions, there is a paper: "Managing Concurrency in Complex Embedded Systems" by Dr. David Cummings (you can google for it).
12:56:33	 From  Simon Voigt Nesbo : The slides said 132 MHz for the CPU, not 135 kHz
12:59:30	 From  Tim Michals : Are the checklists and design methodology open source or available?
13:00:24	 From  David Potter : What code analysis tools? 
13:01:30	 From  Gopinath : Correct, my bad. But even 132 MHz is low.
13:01:52	 From  Leopy : Is everything human-coded, or some functions on the boad computer are dealt with ML/AI?
13:05:00	 From  jvillasante : Fantastic! Are you hiring? :)
13:06:15	 From  David Potter : Are your software design rules available to the public?
13:09:56	 From  Dave Comer : Is there a SysML talk or material that the public can access?
13:09:56	 From  Michael Kirkhart : https://yurichev.com/mirrors/C/JPL_Coding_Standard_C.pdf
13:15:34	 From  Tom.Davies : What tools do you use to autogenerate the code?
13:16:10	 From  Radu Pralea : How do you deal with real-time stuff in the sw simulation environment (I guess the models of the "peripherals" could be slower than the actual hardware) so how do you test the actual software (which I assume would depend on real timings on the real system), in a fully simulated environment, Do you have some timing abstraction layer taking care of this?
13:16:25	 From  Tom.Davies : Now that Perseverance is on the surface, how long will you remain on the project before moving onto the next project?
13:17:26	 From  Kurtovic, Tarik (1.59) : On-target unit testing is mandatory in some industries. Do you (need to) do on-target unit testing?
13:18:10	 From  Dave Nadler : What caused the resets during transit to Mars?
13:19:21	 From  Dave Comer : Alpha particle....This was a key concern in testing SRAMs, EPROMs, FLASH, EEPEOM....
13:23:54	 From  Jay : Can you talk about telemtry and logging? What do you log, how large memory footprint, how do you encode the logs? How do you ensure that you can use these to diagnose unexpected events?
13:30:24	 From  Andrei : Might be a silly question (and I may have missed it).. Are the coms being encrypted with publicly available algorithms?
13:34:30	 From  Dave Nadler : I need a desktop pyro simulation...
13:34:39	 From  Meenal Burrows : :-)
13:35:09	 From  afwaanquadri : This was great! Thanks for your presentation!
13:35:10	 From  Keith J : Thank you very much Steve!  Fascinating stuff. Very much appreciated you taking the time.
13:35:37	 From  Dave Nadler : Thanks Steve - Awesome work and awesome presentation!
13:36:43	 From  Simon Voigt Nesbo : Yeah thanks for the great presentation. And thanks for answering my question :)
13:36:54	 From  Stefan Petersen : Thanks Steve! Great hearing about your work and set ups, great presentation and QnA.
13:37:07	 From  Meenal Burrows : Brilliant keynote and awesome Q&A. Thanks Steve for your time with us all.
13:37:52	 From  Rob Meades : That was excellent, many, many, thanks.
13:38:03	 From  Jay : Great presentation and discussion!
13:38:04	 From  Gopinath : Excellent presentation, Steve. Thank you very much.
13:38:06	 From  mdohring : Wonderful presentation!  Thank you very much!
13:38:08	 From  Yuriy Kozhynov : Thanks a lot!!!
13:38:10	 From  Juan : thank you!
13:38:11	 From  Erwin : Awesome Job you do and a great talk! Fantastic to get this insights on how you work!
13:38:11	 From  Dan Rittersdorf : Great talk and QA, Steve!   Thank you so much.
13:38:17	 From  Tom.Davies : Thank you Steve
13:38:23	 From  David Pastl : Great presentation, thank you Steve!
13:38:25	 From  Jose E. : Thanks for the presentation!
13:38:26	 From  Doug Peters : Great talk!!
13:38:29	 From  Eric : Thank you for a great presentation!
13:38:44	 From  James G : Awesome! Extremely interesting and enlightening, Steve. Thank you!
13:38:45	 From  Andrey Shevelov : Great presentation! Thanks a lot!
13:38:46	 From  PeteMehn : Well done.  Thanks for sharing!
13:38:54	 From  Michael Kafarowski : Thank you!
13:39:00	 From  Sam : thank you.
13:39:06	 From  Leandro Pérez : Thanks
13:39:27	 From  Christopher Long : Thank you