Slicer hangs after starting and stopping the simulation a few times

[lassoan]

After stopping and starting the simulation a few times, Slicer stops responding.

I loaded the RightLung sample mesh defined an ROI at the bottom of the model, and defined a moving point. After that, I used the soft tissue simulation module: started the simulation, moved the point, stopped the simulation. It seemed that moving the point persistently changed the model's shape, so next time the simulation was started, the effect of the previous point moving was still there. It was surprising, but OK. I repeated this a few times: start simulation, move the point, stop the simulation, move the point. This always resulted in Slicer hanging, with an output like this:

[ERROR]   [MechanicalObject(AttachedPoint)] Invalid vOp operation 1 (null(V_DERIV),0(0),null(V_DERIV),100)
[ERROR]   [MechanicalObject(AttachedPoint)] Invalid vOp operation 1 (null(V_DERIV),0(0),null(V_DERIV),100)
[ERROR]   [MechanicalObject(AttachedPoint)] Invalid vOp operation 1 (null(V_DERIV),0(0),null(V_DERIV),100)
[ERROR]   [MechanicalObject(AttachedPoint)] Invalid vOp operation 1 (null(V_DERIV),0(0),null(V_DERIV),100)
[ERROR]   [MechanicalObject(AttachedPoint)] Invalid vOp operation 1 (null(V_DERIV),0(0),null(V_DERIV),100)
[ERROR]   [MechanicalObject(AttachedPoint)] Invalid vOp operation 1 (null(V_DERIV),0(0),null(V_DERIV),100)
[ERROR]   [MechanicalObject(AttachedPoint)] Invalid vOp operation 1 (null(V_DERIV),0(0),null(V_DERIV),100)
[ERROR]   [MechanicalObject(AttachedPoint)] Invalid vOp operation 1 (null(V_DERIV),0(0),null(V_DERIV),100)
[ERROR]   [MechanicalObject(AttachedPoint)] Invalid vOp operation 1 (null(V_DERIV),0(0),null(V_DERIV),100)
[ERROR]   [MechanicalObject(AttachedPoint)] Invalid vOp operation 1 (null(V_DERIV),0(0),null(V_DERIV),100)
[ERROR]   [MechanicalObject(AttachedPoint)] Invalid vOp operation 1 (null(V_DERIV),0(0),null(V_DERIV),100)
[ERROR]   [MechanicalObject(AttachedPoint)] Invalid vOp operation 1 (null(V_DERIV),0(0),null(V_DERIV),100)
[ERROR]   [MechanicalObject(AttachedPoint)] Invalid vOp operation 1 (null(V_DERIV),0(0),null(V_DERIV),100)
[ERROR]   [MechanicalObject(AttachedPoint)] Invalid vOp operation 1 (null(V_DERIV),0(0),null(V_DERIV),100)
[ERROR]   [MechanicalObject(AttachedPoint)] Invalid vOp operation 1 (null(V_DERIV),0(0),null(V_DERIV),100)
[ERROR]   [MechanicalObject(AttachedPoint)] Invalid vOp operation 1 (null(V_DERIV),0(0),null(V_DERIV),100)
[ERROR]   [MechanicalObject(AttachedPoint)] Invalid vOp operation 1 (null(V_DERIV),0(0),null(V_DERIV),100)
[ERROR]   [MechanicalObject(AttachedPoint)] Invalid vOp operation 1 (null(V_DERIV),0(0),null(V_DERIV),100)
[ERROR]   [PluginManager] Plugin loading failed (C:/D/S4R/Slicer-build/slicer.org/Extensions-33142/SlicerSOFA/lib/Slicer-5.7/qt-scripted-modules/SofaEnvironment/../../../../plugins/SofaIGTLink/bin/SofaIGTLink.dll): function initExternalModule() not found. The following error was received: The specified procedure could not be found.

[ERROR]   [RequiredPlugin(SofaIGTLink)] Plugin loading failed (C:/D/S4R/Slicer-build/slicer.org/Extensions-33142/SlicerSOFA/lib/Slicer-5.7/qt-scripted-modules/SofaEnvironment/../../../../plugins/SofaIGTLink/bin/SofaIGTLink.dll): function initExternalModule() not found. The following error was received: The specified procedure could not be found.


Failed to load: ["SofaIGTLink"]
[WARNING] [TetrahedronSetTopologyContainer(Container)] Could not read value for data field position: None
[WARNING] [TetrahedronSetTopologyContainer(Container)] Could not read value for data field tetrahedra: None

########## SIG 11 - SIGSEGV: segfault ##########
  sofa::helper::BackTrace::dump
  sofa::helper::BackTrace::sig
  log2f
  log2f
  _C_specific_handler
  _chkstk
  RtlFindCharInUnicodeString
  KiUserExceptionDispatcher
  sofa::component::collision::geometry::TriangleCollisionModel<sofa::defaulttype::StdVectorTypes<sofa::type::Vec<3,double>,sofa::type::Vec<3,double>,double> >::updateNormals
  sofa::component::collision::geometry::TriangleCollisionModel<sofa::defaulttype::StdVectorTypes<sofa::type::Vec<3,double>,sofa::type::Vec<3,double>,double> >::init
  sofa::simulation::InitVisitor::processNodeTopDown
  sofa::simulation::graph::DAGNode::executeVisitorTopDown
  sofa::simulation::graph::DAGNode::executeVisitorTopDown
  sofa::simulation::graph::DAGNode::executeVisitorTopDown
  sofa::simulation::graph::DAGNode::doExecuteVisitor
  sofa::simulation::Node::executeVisitor
  sofa::simulation::node::init
  sofa::simulation::node::initRoot
  PyInit_Simulation
  PyInit_Simulation
  PyMemoryView_GetContiguous
  PyObject_MakeTpCall
  PyEval_EvalFrameDefault
  Py_CheckFunctionResult
  PyFunction_Vectorcall
  PyEval_SliceIndexNotNone
  PyEval_EvalFrameDefault
  PyEval_CallTracing
  PyFunction_Vectorcall
  PyEval_SliceIndexNotNone
  PyEval_EvalFrameDefault
  Py_CheckFunctionResult
  PyFunction_Vectorcall
  PyMethod_Self
  PyObject_MakeTpCall
  PyObject_CallObject
  PythonQtSignalTarget::call
  PythonQtSignalTarget::call
  PythonQtSignalTarget::isSame
  QObject::qt_static_metacall
  QAbstractButton::clicked
  QAbstractButton::click
  QAbstractButton::mouseReleaseEvent
  QWidget::event
  QApplicationPrivate::notify_helper
  QApplication::notify
  qSlicerApplication::notify
  QCoreApplication::notifyInternal2
  QApplicationPrivate::sendMouseEvent
  QSizePolicy::QSizePolicy
  QSizePolicy::QSizePolicy
  QApplicationPrivate::notify_helper
  QApplication::notify
  qSlicerApplication::notify
  QCoreApplication::notifyInternal2
  QGuiApplicationPrivate::processMouseEvent
  QWindowSystemInterface::sendWindowSystemEvents
  QEventDispatcherWin32::processEvents
  qt_plugin_query_metadata
  QEventLoop::exec
  QCoreApplication::exec
  qSlicerCoreApplication::exec
  qSlicerCoreApplication::exec
  qSlicerCoreApplication::exec
  BaseThreadInitThunk
  RtlUserThreadStart

I guess the mesh somehow became too irregular and the simulation failed. Having a simulation failing for a difficult input is OK. What is unacceptable is to bring down the entire application. Recover from a crashed application would take at least a minute, even if the application state was saved, and such delays would not be acceptable in many clinical use cases.

It would be nice to fix this crash. However, it may not be possible to fix all the crashes, because a simulation may require lots of computations and everything has to be fast. Preventing all possibility of a crash could be done in theory, but it may be impractical. Adding lots of checks everywhere may be computationally too expensive. Or things could be implemented more carefully, smarter, etc. but then development would require special skills, experience, and thus the development cost could be just too high. However, if we accept that crashes can always occur for somewhat unusual inputs, it would mean that the only practically achievable solution for clinical use would be to run the simulation in a process separate from the main GUI application.

[pieper]

We should check how data is being passed from sofa to slicer. For my experiments I re-run the simulation dozens of times and never get a crash, even if they simulation becomes unstable or is ill-posed, so I suspect there's a fixable memory ownership issue in the module code.

[RafaelPalomar]

@lassoan, I'll try to reproduce this issue and come back to you. As @pieper points out, these problems are usually due to inconsistencies between the internal simulation state and the data we pass from Slicer.

[RafaelPalomar]

@lassoan, I have been able to reproduce the bug. It seems it is in the component that makes the deformation on mouse picking. Simulations based only on force (i.e., not creating the moving point at all) don't have that problem. The problem will be either on the way we pass the data to the simulation, or the simulation component that makes the picker happen in SOFA. We'll continue investigating and solve the issue.

[lassoan]

Unfortunately, the hang is still reproducible. I used a factory build, so I don't have the stack trace, so I cannot confirm that it is the exact same issue, but the behavior was the same. Maybe it took a little more to make Slicer hang (not after 2-3 restarts but 6-8). I always use a point to move the surface (not changing the gravity vector).

[RafaelPalomar]

@lassoan, to isolate the problem, could you try not adding the fiducial point and just manipulate the gravity vector? I suspect it could have to do with the SOFA component itself