"Exiting with abort signal and core dump file" when using UART drivers and "startReadThread"

[mosa11aei]

I have a deployment (built for RPi running Ubuntu server, and my own Linux computer) ready that utilizes a UART-connected GPS module (NEO M9N, over USB). I have gotten F' to successfully "open" the serial address where the GPS is located, but when I try to run "startReadThread" on the corresponding serialDriver (instance of LinuxUartDriver), I get a seg fault. This is true for both my RPi and Linux computers, so I don't think my RPi or computer are "broken", per se.

I am running my binary like so:

# example for my Linux --> similar for RPi deployment
sudo build-artifacts/Linux/GpsTestDeployment/bin/GpsTestDeployment -a 127.0.0.1 -p 50000

I have used multiple debugging tools (i.e., gdb) to read the coredump generated, but it just gives me a massive runaround. I have commented out all of my implementations in my Component folder (that handles the GPS), so I am 99% certain that the issue is with "startReadThread". I have also independently ensured my GPS module is working properly. How can I go about solving this issue? Some code snippets are below:

deployment > Top > Toplogy.cpp

void configureTopology() {
    ...
    bool uart_success = GpsTestDeployment::serialDriver.open("/dev/ttyACM0",
            Drv::LinuxUartDriver::BAUD_9600,
            Drv::LinuxUartDriver::NO_FLOW,
            Drv::LinuxUartDriver::PARITY_NONE,
            true);

    if(uart_success){
        // I do reach here successfully
        logger.log("[INFO] UART opened successfully\n");
        GpsTestDeployment::serialDriver.startReadThread(); // where the error happens
    } else {
        logger.log("[ERROR] UART failed to open\n");
    }
...

deployment > instances.fpp

...
  instance gpsModule: Components.GpsModule base id 0x4B00 \
    queue size Default.QUEUE_SIZE \
    stack size Default.STACK_SIZE \
    priority 30

  instance serialDriver: Drv.LinuxUartDriver base id 0x6D00

deployment > topology.fpp

...
    instance serialDriver
    instance gpsModule
...
    connections UART {
      # gpsModule.UartBuffers -> serialDriver.$recv
      serialDriver.$recv -> gpsModule.UartRead
    }

Components > GpsModule > GpsModule.cpp

...
  void GpsModule ::
      UartRead_handler(
          const NATIVE_INT_TYPE portNum,
          Fw::Buffer &recvBuffer,
          const Drv::RecvStatus &recvStatus)
  {
    // char uMsg[recvBuffer.getSize() + 1];
    // char *bPtr = reinterpret_cast<char *>(recvBuffer.getData());

    // for (NATIVE_UINT_TYPE byte = 0; byte < recvBuffer.getSize(); byte++)
    // {
    //   uMsg[byte] = isalpha(bPtr[byte]) ? bPtr[byte] : '*';
    // }
    // uMsg[sizeof(uMsg) - 1] = 0;

    // this->m_uartReadBytes += recvBuffer.getSize();

    // recvBuffer.setSize(40);

    // this->UartBuffers_out(0, recvBuffer);
    // this->tlmWrite_UartRecvBytes(this->m_uartReadBytes);
  }
...

[mosa11aei]

I realize the core dump might also be useful (from valgrind)

Exiting with abort signal and core dump file.
==2694== 
==2694== Process terminating with default action of signal 6 (SIGABRT)
==2694==    at 0x48797D8: raise (raise.c:51)
==2694==    by 0x43B88F: Svc::FatalHandlerComponentImpl::FatalReceive_handler(int, unsigned int) (in /home/fprime-node1/GpsTestDeployment/aarch64-linux/GpsTestDeployment/bin/GpsTestDeployment)
==2694==    by 0x43BC47: Svc::FatalHandlerComponentBase::FatalReceive_handlerBase(int, unsigned int) (in /home/fprime-node1/GpsTestDeployment/aarch64-linux/GpsTestDeployment/bin/GpsTestDeployment)
==2694==    by 0x43BBDB: Svc::FatalHandlerComponentBase::m_p_FatalReceive_in(Fw::PassiveComponentBase*, int, unsigned int) (in /home/fprime-node1/GpsTestDeployment/aarch64-linux/GpsTestDeployment/bin/GpsTestDeployment)
==2694==    by 0x43BDEB: Svc::InputFatalEventPort::invoke(unsigned int) (in /home/fprime-node1/GpsTestDeployment/aarch64-linux/GpsTestDeployment/bin/GpsTestDeployment)
==2694==    by 0x43BFFB: Svc::OutputFatalEventPort::invoke(unsigned int) (in /home/fprime-node1/GpsTestDeployment/aarch64-linux/GpsTestDeployment/bin/GpsTestDeployment)
==2694==    by 0x417147: Svc::ActiveLoggerComponentBase::FatalAnnounce_out(int, unsigned int) (in /home/fprime-node1/GpsTestDeployment/aarch64-linux/GpsTestDeployment/bin/GpsTestDeployment)
==2694==    by 0x4143C3: Svc::ActiveLoggerImpl::LogRecv_handler(int, unsigned int, Fw::Time&, Fw::LogSeverity const&, Fw::LogBuffer&) (in /home/fprime-node1/GpsTestDeployment/aarch64-linux/GpsTestDeployment/bin/GpsTestDeployment)
==2694==    by 0x418D6F: Svc::ActiveLoggerComponentBase::LogRecv_handlerBase(int, unsigned int, Fw::Time&, Fw::LogSeverity const&, Fw::LogBuffer&) (in /home/fprime-node1/GpsTestDeployment/aarch64-linux/GpsTestDeployment/bin/GpsTestDeployment)
==2694==    by 0x418C97: Svc::ActiveLoggerComponentBase::m_p_LogRecv_in(Fw::PassiveComponentBase*, int, unsigned int, Fw::Time&, Fw::LogSeverity const&, Fw::LogBuffer&) (in /home/fprime-node1/GpsTestDeployment/aarch64-linux/GpsTestDeployment/bin/GpsTestDeployment)
==2694==    by 0x46F923: Fw::InputLogPort::invoke(unsigned int, Fw::Time&, Fw::LogSeverity const&, Fw::LogBuffer&) (in /home/fprime-node1/GpsTestDeployment/aarch64-linux/GpsTestDeployment/bin/GpsTestDeployment)
==2694==    by 0x46FC4F: Fw::OutputLogPort::invoke(unsigned int, Fw::Time&, Fw::LogSeverity const&, Fw::LogBuffer&) (in /home/fprime-node1/GpsTestDeployment/aarch64-linux/GpsTestDeployment/bin/GpsTestDeployment)
==2694== 
==2694== HEAP SUMMARY:
==2694==     in use at exit: 138,370 bytes in 138 blocks
==2694==   total heap usage: 148 allocs, 10 frees, 218,082 bytes allocated
==2694== 
==2694== LEAK SUMMARY:
==2694==    definitely lost: 0 bytes in 0 blocks
==2694==    indirectly lost: 0 bytes in 0 blocks
==2694==      possibly lost: 4,320 bytes in 15 blocks
==2694==    still reachable: 134,050 bytes in 123 blocks
==2694==         suppressed: 0 bytes in 0 blocks
==2694== Rerun with --leak-check=full to see details of leaked memory
==2694== 
==2694== For lists of detected and suppressed errors, rerun with: -s
==2694== ERROR SUMMARY: 0 errors from 0 contexts (suppressed: 0 from 0)

[mosa11aei]

I've done a bit more debugging, and I'm thinking it's the parameters of the startReadThread that are causing the seg fault. The call I have above has them all undefined, so the values for priority, stack size and cpu affinity are all -1. I don't believe they should be -1, but I also am unsure what values they should be instead.

I'm thinking priority can be a default priority of 30, that is seen in the topology file? Stack size 64 * 1024? But I am in the dark about CPU affinity; are there some default values I can start from to tune them to my specific component?

[mosa11aei]

On a similar note, I was able to locate where an assertion was failing in the process of starting the read thread (see picture below). It was located in the build folder > F-Prime > Fw > Buffer > BufferGetPortAc.cpp: FW_ASSERT(this->m_port != nullptr);

I've tried to backsearch for this->m_port with nothing very useful gained. Does this provide any insight into what might be going wrong?

[LeStarch]

Several things:

1. What version of F´ are you running?
2. • 1 for arguments means "use a default value" so those should be ok.
3. The segfault is coming from the fatal handler, which comes from the ASSERT you tripped, so it is the assert we should first debug.
4. The assert you see happens when a port is not connected. Is the port: $recv connected to something in the topology?

[LeStarch]

Apologies, I miss-read it. The Linux driver needs to be hooked up to a memory provider. Thus the allocate port should be hooked-up to a buffer manager or static memory instance. Otherwise it cannot get memory it needs to read the port.

[mosa11aei]

1. I'm running the latest version of F'; whatever is in the devel branch currently. Specific commit ID for my version is a17badf7. Most likely this means F' 3.2.0.
2. Yes it is defined in the topology. Here's the code I have for that at the end of my topology.fpp file:

    connections UART {
      # gpsModule.UartBuffers -> serialDriver.send 
      serialDriver.$recv -> gpsModule.UartRead 
    }

I have tried the build with the gpsModule.UartBuffers -> serialDriver.send line both commented and uncommented if that was a concern (though I'm not sure relevant in this case); still the same issue.

[Edit]: I see your comment now about the memory provider; I will look into that and let you know how I have implemented that in the component/topology.

[mosa11aei]

Apologies, I miss-read it. The Linux driver needs to be hooked up to a memory provider. Thus the allocate port should be hooked-up to a buffer manager or static memory instance. Otherwise it cannot get memory it needs to read the port.

This was great, thanks! No more seg fault issue, but now I'm running into an issue with [device] ran out of buffers. At this point, I have to go in and learn more about the buffer manager, but what is the overall structure of the implementation I need for the buffer manager? Is this something I can find in the docs, or in the F' GPS tutorial?

For reference, this is the (very very quick and crude) implementation of the buffer input port function that I made to get it to work.

  Fw::Buffer GpsModule ::
    UartMemManager_handler(
        const NATIVE_INT_TYPE portNum,
        U32 size
    )
  {
    Fw::Buffer value;
    return value;
  }

[LeStarch]

First, the GPS tutorial is severely outdated. In the latest documentation, I think we have removed it for this reason. Here is a better tutorial: https://github.com/fprime-community/fprime-workshop-led-blinker, reasonably updated for latest F´ and shipping with sample code.

Next, to provide memory we typically do so from existing memory components. BufferManager is a buffer-pool component. Using it involves:

1. Instantiating a manager in the topology (instances.fpp):

instance comBufferManager: Svc.BufferManager base id 0x4500

2. Wiring to it (topology.fpp):

      uartDriver.allocate -> bufferManager.bufferGetCallee. # Allocation
      gps.deallocate -> bufferManager.bufferSendIn. # Deallocation when finished

3. Configuring it for the buffer pool it manages:

    Svc::BufferManager::BufferBins bufferManagerBins;
    memset(&bufferManagerBins, 0, sizeof(bufferManagerBins));
    {
      bufferManagerBins.bins[0].bufferSize = 1024; # or any size
      bufferManagerBins.bins[0].numBuffers = 10;  # Or any number
      bufferManagerBins.setup(
          <some id>,
          0,
          Allocation::mallocator,
          bufferManagerBins
      );
    }

[mosa11aei]

Great, thanks for the resources!

[jsilveira1409]

Hi! I ran into the same issue. So there's no need to call
GpsTestDeployment::serialDriver.startReadThread(); // where the error happens
?

And what should be the <some id> ? I'm getting the following error:

Assert in file /Desktop/DPhi/DPhi-FS/DPhi/fprime/Svc/BufferManager/BufferMana, line 105: 65535 10
FATAL 16898 handled.

Which seems to be linked to the id ?

Thank you for the clarifications above!

[mosa11aei]

Sorry for asking so many questions! From this info, I can't off the bat tell you what I think the problem is. But I think I have an inkling. Is the error you're getting now the same as before? If you're able to send something, I'm thinking it's no longer a port issue.

If you wait 30 seconds before sending something over TCP, does the error still occur? I.e. I'm seeing if this is an implementation problem, or a setup problem.

On send to the server, do you send a response back to F' immediately? Do you know if the error is caused when the output is sent out of F', or when you're reading something?

A couple things I'm thinking could be the problem: you haven't allocated enough buffers/buffer size to the buffer manager, or there's a step that's different in TCP from UART that needs to be considered relating to memory management...

[mosa11aei]

Something that may be of use is the TcpClient doc. You've most likely seen it, but compare your code with it and see if the general process of what you're doing is the same: https://github.com/nasa/fprime/blob/master/Drv/TcpClient/docs/sdd.md

Also, you mentioned that you have a Topology.cpp file; you are using the latest version of F' correct? Just curious.

[jsilveira1409]

No problem at all, thank you for the patience ! So

• No error occurs after 30 seconds of idle
• I used to send the response back immediately, removed it to test if the error persisted and it did.
• When sending the data, the first message is correct, the following are empty, so i guess it gets disconnected after the first message.

I'll check the TcpClientComponent documentation to check what step I'm missing :( But thanks a lot for the health.

[LeStarch]

Is the cameraDriver deallocate port hooked up? This drallocate port should be hooked up to the buffer send in port of the buffer manager to return the buffers when finished.

[jsilveira1409]

@LeStarch it is hooked up now (was missing indeed). However, I'm getting the same error as I was getting for the Uart before implementing the BufferManager with the correct ID, so the :

Assert in file Svc/BufferManager/BufferManagerComponentImpl.cpp, line 105: 65535 1

Any ideas or leads ? The current topology is :

instance.fpp:

# Camera
instance cameraBufferManager : Svc.BufferManager base id 0x4D00
instance cameraDriver: Drv.ByteStreamDriverModel base id 0x4E00 \
    type "Drv::TcpClient" \ # type specified to select implementor of ByteStreamDriverModel
     at  "../../Drv/TcpClient/TcpClient.hpp" # location of above implementor must also be specified

topology.fpp

cameraDriver.$recv -> camera.DataIn
cameraDriver.allocate -> cameraBufferManager.bufferGetCallee    
cameraDriver.deallocate -> cameraBufferManager.bufferSendIn 
camera.DataOut -> cameraDriver.send

And Topology.cpp

Svc::BufferManager::BufferBins cameraBufferBins;
memset(&cameraBufferBins, 0, sizeof(cameraBufferBins));
cameraBufferBins.bins[0].bufferSize = CAMERA_BUFFER_MANAGER_STORE_SIZE;
cameraBufferBins.bins[0].numBuffers = CAMERA_BUFFER_MANAGER_QUEUE_SIZE;
cameraBufferManager.setup(CAMERA_BUFFER_MANAGER_ID, 0, mallocator, cameraBufferBins);

...

Drv::SocketIpStatus status = cameraDriver.configure("127.0.0.1", 5005, 1, 0);
Os::TaskString cameraTask("CameraTask");
cameraDriver.startSocketTask(cameraTask, true,COMM_PRIORITY,Default::STACK_SIZE);

The CAMERA_BUFFER_MANAGER_ID should be equal to the cameraDriver id by any chance ?