Using events for error handling?

[zimri-leisher]

Hello,
I'm looking to write an error handling system and wondering if using the F' event system would work. Are F' events only designed for spacecraft-to-ground communication, or would it be reasonable to listen to F' events inside of F' components and handle errors that way? For example, if we used events, if a sensor fails, I might emit a FATAL event which would notify another component to go into safe mode.
This is more of a general principles question. I'm sure, implementation-wise, that this is technically possible, but maybe it's not a good idea for a critical system (error handling) to depend on the events system working if events are only designed to be transient notifications.
Let me know your thoughts,
Thanks,
Zimri

[timcanham]

@zimri-leisher You are correct, events are meant for transmission to the ground. You could tap into the flow onboard, but then you would have to decode them with separate code to discern which event it was. Also, if the event has throttling, you could miss them altogether.

What you are referring to is fault protection, or how a system responds to issues in the system. Each mission typically has a unique fault response design, so projects here at JPL have spun their own ports and components to handle the requirements for that system.

[zimri-leisher]

Ok, understood. Good to know that we aren't missing an obvious, premade solution. Thanks for the response!

[ljreder]

@timcanham I read this and think it might be a good idea in general if the framework had a generic Event port. I am thinking something like what is done in Quantum framework for state-machines where the port contains a generic signal id and then either an extended object or generic serializable to handle a general payload of data. Perhaps talk to Garth Watney. This would not only be useful for multiple state machines but also for multiple control algorithms implemented across components. Perhaps some of interest to @LeStarch. The whole topic is somewhat reministic of the old CORBA Event Service of some 20 something years ago sending events between distributed objects.

[SterlingPeet]

@timcanham , I think @ljreder is on to something here. I have seen this confusion between the term Event, as used with respect to event driven programming and especially GUIs, and how F Prime (or perhaps JPL??) define software events to be the definition 1 from Mariam Webster. This is a common education stumbling block for developers getting started in the framework.

The namespace collision for this term has had me idly wondering what an F Prime Event actually is, what should it be called if it was not an event? I think other software disciplines would oft describe this as a logging entry, (or confusingly as a log event) which is probably closer to what is intended here. The builtin LogEvent makes it harder, because that is the log line, a text string describing the situation from within the software. In an enterprise logging middleware, the F Prime Event would just be one of several plausible compression transfer options, and log string would go in one side and come out the other. The Event in F Prime seems to be a compressed log entry, designed for transfer to the ground, and not an event signal in the the Event Programming sense.

Unfortunately, we can't just call it a Signal, because that collides with a different core implementation. Maybe it is the same concept, but we definitely wouldn't want to just use that name without supplementary explanation.

Any legitimate solution I can see is clearly going to involve some strong documentation and [re]education about internal F Prime terms, seeing as how F Prime Events as the are currently implemented are rather fundamental parts of the framework.

[ljreder]

@SterlingPeet I have been away from the main F′ development for awhile and have not had a chance to go through current documentation so perhaps someone is using the term "F Prime Event" where they should actually be always saying "F Prime Log Event". F Prime does have a so called Log Event Reporting system or infrastructure or better yet type of telemetry packet. The idea is to have a set of Log Event Messages that are associated with ID's. The message strings look like C printf strings. The "F Prime Log Event" system code generation produces code and a common dictionary so that when the code sends a log event message to the ground only an ID and a set of arguments are sent. The ground data system does an association of the ID with the message text via a dictionary to produce text messages that are human readable. The scheme is obviously meant to minimize information transmitted and save bandwidth. The technique seems pretty widely used today in various forms. So all that is currently in F prime is a Log Event Reporting system. There are no universally general Event mechanisms as you site above. If there is documentation that suggests otherwise could you provide some pointers to it for me.

[SterlingPeet]

Hello @ljreder, I had heard that you might have retired, and I hope that and your current endeavors are going well for you!

I realize that I did the math professor thing and adjusted the notation to try and clarify the concepts, and that may have been counterproductive. I think that this topic warrants some further conversation, but I will need to take some time to go gather my thoughts and resources to do so. The problem is not that F Prime is deficient, nor that the customer needs are not achievable, but rather that there is some common confusion generated in the developer user experience (UX). This type of problem is always hard to explain to folks who build a product, because it has to do with the learning process for someone not intimately familiar with the tool and the ways that problems may be solved within it.

I will go collect up my information and provide a better (probably new discussion topic) write-up about it.

[ljreder]

@SterlingPeet yes I retired from JPL in July 2022 but they hired me back for limited consulting work in Aug 2022 on a special program they have for retirees. Been mostly helping with an internal special technology project and reviewing technologies proposals over the past year. This year they asked if I could be involved with F′ a little bit. Mainly for reviews, testing, and participate here in discussions. My involvement is and will be very limited.

Yes go away and think about it and perhaps start a discussion that deals with Events. But also important is dealing with state-machine behaviors and algorithm execution flows in a structured way that can interact with state behaviors. There are control applications that are custom hacked which could benefit from tight integration into all the structural capability of the F prime framework.

[zimri-leisher]

When trying to find where events were implemented in the framework, I initially had trouble finding the right place because I was looking for a component with the word Event in its name, instead of one of the various Logger components. It is definitely a little confusing but after reading your answers I understand why this is the case.

Perhaps better than adding a new event type to the framework would be adding some prebuilt system for fault detection and fault response. Obviously the logic for both of these could only be mission specific, but there is probably some level of abstraction which guides users to the right system design without restricting them. Maybe it could be based on https://llis.nasa.gov/lesson/772

[SterlingPeet]

I'm glad that the conversation was helpful, it can be hard to find what you are looking for amongst arcane terminology that is not always consistent with the "center of mass" of computer science.

For your purposes, yes, a cookbook recipe for fault protection would be most helpful. For the use case of firing and catching events to support the common architecture of state machines (which may be used for implement fault protection, but are a distinct topic), some sort of supported signal event recipe is probably also needed as a separate cookbook recipe.

[timcanham]

Good discussion! I created a discussion issue specifically for a basic FDIR feature here: #2536. It has some thoughts that have been rattling around in my head for a while.

[ljreder]

Hello @ljreder, I had heard that you might have retired, and I hope that and your current endeavors are going well for you!

I realize that I did the math professor thing and adjusted the notation to try and clarify the concepts, and that may have been counterproductive. I think that this topic warrants some further conversation, but I will need to take some time to go gather my thoughts and resources to do so. The problem is not that F Prime is deficient, nor that the customer needs are not achievable, but rather that there is some common confusion generated in the developer user experience (UX). This type of problem is always hard to explain to folks who build a product, because it has to do with the learning process for someone not intimately familiar with the tool and the ways that problems may be solved within it.

I will go collect up my information and provide a better (probably new discussion topic) write-up about it.

[ljreder]

Sounds like this discussion has morphed in a FDIR feature of @timcanham and @SterlingPeet and @ljreder are thinking about a bit more general behavior/event driven things.