[BUG]: Silent failure if python object is destroyed but shared_ptr to CPP object is held

[Chekov2k]

Required prerequisites

• Make sure you've read the documentation. Your issue may be addressed there.
• Search the issue tracker and Discussions to verify that this hasn't already been reported. +1 or comment there if it has.
• Consider asking first in the Gitter chat room or in a Discussion.

What version (or hash if on master) of pybind11 are you using?

2.10.4

Problem description

We have come across an issue concerning the combination of Python and CPP object lifetimes. If a Cpp class is exposed via pybind11 using a shared_ptr as a holder, it is possible to destroy the python object while retaining a shared_ptr to the Cpp object. However, the Cpp object behaviour reverts to the Cpp base class and all python behaviour is silently dropped.

Is the silent drop expected? We would either expect an exception when the object is used or to retain full functionality. We attached some simple example code below:

Reproducible example code

On the Cpp side:

#include <gtest/gtest.h>
#include <pybind11/embed.h>
namespace py = pybind11;

#include <memory>

class Parent {
 public:
  Parent() = default;
  virtual ~Parent() = default;

  virtual int Return() const { return 1; }
};
using ParentPtr = std::shared_ptr<Parent>;

class Child : public Parent {
 public:
  Child() = default;
  virtual ~Child() = default;

  int Return() const override { return 2; }
};
using ChildPtr = std::shared_ptr<Child>;

template <typename Base = Parent>
class PyTrampoline : public Base {
 public:
  using Base::Base;
  virtual ~PyTrampoline() = default;

  int Return() const override { PYBIND11_OVERLOAD(int, Base, Return, ); }
};

PYBIND11_EMBEDDED_MODULE(duality, module) {
  py::class_<Parent, PyTrampoline<>, ParentPtr>(module, "Parent")  //
      .def(py::init<>())                                           //
      .def("Return", &Parent::Return);
  py::class_<Child, Parent, PyTrampoline<Child>, ChildPtr>(module, "Child")  //
      .def(py::init<>());
}

void checkPythonResult(const std::string& pClassName, int pPythonExpected,
                       int pCppExpected) {
  auto dualityModule = py::module_::import("Duality");
  auto python = dualityModule.attr(pClassName.c_str())();

  auto result = python.attr("Return")();
  EXPECT_EQ(result.cast<int>(), pPythonExpected);

  auto parentPtr = python.cast<ParentPtr>();
  ASSERT_NE(parentPtr, nullptr);
  EXPECT_EQ(parentPtr->Return(), pPythonExpected);

  ChildPtr childPtr;
  try {
    childPtr = python.cast<ChildPtr>();
    ASSERT_NE(childPtr, nullptr);
    EXPECT_EQ(childPtr->Return(), pPythonExpected);
  } catch (std::exception&) {
  }

  if (childPtr != nullptr) EXPECT_EQ(childPtr->Return(), pPythonExpected);

  python = py::none();
  EXPECT_EQ(parentPtr->Return(), pCppExpected);
  if (childPtr != nullptr) EXPECT_EQ(childPtr->Return(), pCppExpected);
}

TEST(Utilities, Duality) {
  EXPECT_EQ(std::make_shared<Parent>()->Return(), 1);
  EXPECT_EQ(std::make_shared<Child>()->Return(), 2);

  py::initialize_interpreter(false);

  checkPythonResult("Parent", 1, 1);
  checkPythonResult("Child", 2, 2);
  checkPythonResult("PythonParent", 3, 1);
  checkPythonResult("PythonChild", 4, 2);

  py::finalize_interpreter();
}

On the Python side

#!/usr/bin/env python3
# -*- coding: utf-8 -*-

from duality import Parent, Child


class PythonParent(Parent):
    def __init__(self) -> None:
        Parent.__init__(self)

    def Return(self) -> int:
        return 3


class PythonChild(Child):
    def __init__(self) -> None:
        Child.__init__(self)

    def Return(self) -> int:
        return 4

And finally a CMakeLists.txt file to build

cmake_minimum_required(VERSION 3.4)
project(Duality)

set(CMAKE_CXX_STANDARD 14)

include(FetchContent)
FetchContent_Declare(googletest GIT_REPOSITORY "https://github.com/google/googletest" GIT_TAG "v1.13.0" GIT_SHALLOW ON TEST_BEFORE_INSTALL ON)
FetchContent_MakeAvailable(googletest)

find_package(pybind11 REQUIRED)

add_executable(DualityTest DualityTest.cpp)
target_link_libraries(DualityTest PRIVATE pybind11::embed)
target_link_libraries(DualityTest PRIVATE GTest::gtest_main)

Is this a regression? Put the last known working version here if it is.

Not a regression

[Skylion007]

@rwgk Does using smart_holders fix this behavior? Or is it a deeper bug?

[rwgk]

At first glance it looks like a duplicate of #1333.

Could you please try your reproducer with:

• git checkout smart_holder
• #include <pybind11/smart_holder.h>
• Insert PYBIND11_SMART_HOLDER_TYPE_CASTERS(Parent) and PYBIND11_SMART_HOLDER_TYPE_CASTERS(Child), right before the PYBIND11_EMBEDDED_MODULE line.
• Replace class_ with classh?

I think that's all you need. For more background see https://github.com/pybind/pybind11/blob/smart_holder/README_smart_holder.rst

[Chekov2k]

Will do!

[Chekov2k]

I can confirm that the smart_holder branch fixes the silent failure, thank you :-) Any indication on when that branch will be merged?