debugger with attach method does not stop at a breakpoint

[hellt]

Environment data

• debugpy version: 1.8.12
• OS and version: Debian 12
• Python version (& distribution if applicable, e.g. Anaconda): Python 3.11 (cpython)
• Using VS Code or Visual Studio: VS Code

Actual behavior

I need to debug a remote python process (CLI). To do so, I have the following configuration:

{
    "version": "0.2.0",
    "configurations": [
        {
            "name": "Remote Debugger: srl",
            "type": "debugpy",
            "request": "attach",
            "connect": {
                "host": "srl",
                "port": 55678
            },
            "justMyCode": false,
            "pathMappings": [
                {
                    "localRoot": "${workspaceFolder}",
                    "remoteRoot": "/etc/opt/srlinux/cli/plugins"
                }
            ]
        }
    ]
}

In the CLI plugin code that I need to debug I install the following:

try:
    import debugpy

    DEBUGPY_AVAILABLE = True
except ImportError:
    DEBUGPY_AVAILABLE = False

class Plugin(CLIPlugin):
    def load(self, cli: CliLoader, **_kwargs):
        if DEBUGPY_AVAILABLE:
            debugpy.log_to("/etc/opt/srlinux/")
            debugpy.listen(("0.0.0.0", 55678))
            debugpy.wait_for_client()

Then later in the plugin code I set a breakpoint with breakpoint().

I can successfully connect to the DAP and see some threads in the callstack:

But my breakpoint never triggers the debugger...

In the debugpy.server log I see:

I+00000.083: Adapter is accepting incoming client connections on 0.0.0.0:55678

D+00000.083: pydevd.settrace(*(), **{'host': '127.0.0.1', 'port': 45377, 'wait_for_ready_to_run': False, 'block_until_connected': True, 'access_token': '8961aa6cb386ed7e42061fefb5a2ddbf5faf6d6d149aa677a05b3b1a6e14ec5c', 'suspend': False, 'patch_multiprocessing': True, 'dont_trace_start_patterns': ('/opt/srlinux/python/virtual-env/lib/python3.11/site-packages/debugpy',), 'dont_trace_end_patterns': ('debugpy_launcher.py',)})

I+00000.192: pydevd is connected to adapter at 127.0.0.1:45377

D+00000.193: wait_for_client()

D+00303.869: breakpoint()

D+00303.870: pydevd.settrace(*(), **{'suspend': True, 'trace_only_current_thread': True, 'patch_multiprocessing': False, 'stop_at_frame': <frame at 0xffff3901bd00, file '<string>', line 84, code _fetch_state>})

In the pydev.d I see many of:

0.00s - Unable to load helper lib to set tracing to all threads (unsupported python vm).

In the adapter log I see the breakpoints from the UI correctly verified:

D+00002.839: Server[1] --> {
                 "pydevd_cmd_id": 502,
                 "seq": 30,
                 "type": "response",
                 "request_seq": 6,
                 "success": true,
                 "command": "setBreakpoints",
                 "body": {
                     "breakpoints": [
                         {
                             "verified": true,
                             "id": 0,
                             "source": {
                                 "name": "uptime.py",
                                 "path": "/home/romandodin/projects/srl-labs/custom-cli/uptime/uptime.py"
                             },
                             "line": 47
                         },
                         {
                             "verified": true,
                             "id": 1,
                             "source": {
                                 "name": "uptime.py",
                                 "path": "/home/romandodin/projects/srl-labs/custom-cli/uptime/uptime.py"
                             },
                             "line": 76
                         },
                         {
                             "verified": true,
                             "id": 2,
                             "source": {
                                 "name": "uptime.py",
                                 "path": "/home/romandodin/projects/srl-labs/custom-cli/uptime/uptime.py"
                             },
                             "line": 77
                         },
                         {
                             "verified": true,
                             "id": 3,
                             "source": {
                                 "name": "uptime.py",
                                 "path": "/home/romandodin/projects/srl-labs/custom-cli/uptime/uptime.py"
                             },
                             "line": 78
                         },
                         {
                             "verified": true,
                             "id": 4,
                             "source": {
                                 "name": "uptime.py",
                                 "path": "/home/romandodin/projects/srl-labs/custom-cli/uptime/uptime.py"
                             },
                             "line": 79
                         }
                     ]
                 }
             }

Expected behavior

Debugger stops at a breakpoint

[rchiodo]

Try setting this as an environment variable.

DEBUG_PYDEVD_PATHS_TRANSLATION=True

The pydevd.log should then have more to say about path translation. That's usually the problem with setting breakpoints in source, they don't map to the actual file.

[hellt]

The pydevd.log should then have more to say about path translation. That's usually the problem with setting breakpoints in source, they don't map to the actual file.

thanks @rchiodo for the suggestion, but if I had the breakpoint set in the source code via breakpoint() func call, shouldn't it had direct effect?

ADD1:
Some additional info that might be relevant.

The source code file I bind mount to the container, so it is line-by-line compatible. Then I also tried to play with the pathMapping to ensure UI do not mark files as "not existing"

[hellt]

Here is the full pydevd log after adding the env var like this:

os.environ["DEBUG_PYDEVD_PATHS_TRANSLATION"] = "True"
debugpy.log_to("/etc/opt/srlinux/")
debugpy.listen(("0.0.0.0", 55678))
debugpy.wait_for_client()

https://gist.github.com/hellt/c22eebcb69c9fe6a0558b1ed3c6dd150

[rchiodo]

I don't see the debuggee getting the breakpoints request.

There should be something like this in the pydevd.log

0.00s - Process SetBreakpointsRequest: {
    "arguments": {
        "breakpoints": [
            {
                "line": 7
            }
        ],
        "lines": [
            7
        ],
        "source": {
            "name": "test_debugattach.py",
            "path": "c:\\Users\\rchiodo\\source\\testing\\test_pylance\\test_debugattach.py"
        },
        "sourceModified": false
    },
    "command": "setBreakpoints",
    "seq": 6,
    "type": "request"
}

0.00s - Request for breakpoint in: c:\Users\rchiodo\source\testing\test_pylance\test_debugattach.py line: 7
0.00s - Breakpoint (after path translation) in: c:\Users\rchiodo\source\testing\test_pylance\test_debugattach.py line: 7
0.00s - File traced: c:\Users\rchiodo\source\testing\test_pylance\test_debugattach.py
0.00s - Added breakpoint:c:\users\rchiodo\source\testing\test_pylance\test_debugattach.py - line:7 - func_name:None

[hellt]

Could it be because of this in the beginning of the log?

0.00s - Error loading: /opt/srlinux/python/virtual-env/lib/python3.11/site-packages/debugpy/_vendored/pydevd/pydevd_attach_to_process/attach_linux_amd64.so
Traceback (most recent call last):
  File "/opt/srlinux/python/virtual-env/lib/python3.11/site-packages/debugpy/_vendored/pydevd/pydevd_tracing.py", line 285, in _load_python_helper_lib_uncached
    lib = ctypes.pydll.LoadLibrary(filename)
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.11/ctypes/__init__.py", line 454, in LoadLibrary
    return self._dlltype(name)
           ^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.11/ctypes/__init__.py", line 376, in __init__
    self._handle = _dlopen(self._name, mode)
                   ^^^^^^^^^^^^^^^^^^^^^^^^^
OSError: /opt/srlinux/python/virtual-env/lib/python3.11/site-packages/debugpy/_vendored/pydevd/pydevd_attach_to_process/attach_linux_amd64.so: cannot open shared object file: No such file or directory

[hellt]

Sorry, I think the breakpoints got wiped after I recreated the environment. I have opened the log on one side and monitored it while I am adding a breakpoint in the UI

Here is the full log https://gist.github.com/hellt/077be35d958d75ecc408e5cb3b272805

The request is on ln 1756

[rchiodo]

That dlopen exception is handled, but it might be on attach we can't turn on sys.trace without that dll. I wasn't sure that was the case, but it explains your situation. If we can't trace the functions with the breakpoints in them, the breakpoints will never get hit.

I assume the remote machine is ARM64?

Can you try this same thing with python 3.13 instead? We use a different method for injecting the trace in 3.12 and above.

[hellt]

I don't control the python env, so can't switch from py 3.11.
Will repeat the same test on the amd64 machine with the exat same venv...

[hellt]

@rchiodo repeated on the amd64 system in the same environment. The dlopen exception is no more, the paths look accurate, but still no breakpoint action.

https://gist.github.com/hellt/428e35d11fb0d485bb0439ad51d02988

[rchiodo]

Are you sure you're hitting the code with the breakpoint in it? You have a breakpoint in a file called uptime.py.

I never see a trace that goes through that file. Lines like this:

0.00s - Set tracing of frame: /opt/srlinux/python/virtual-env/lib/python3.11/dist-packages/srlinux/mgmt/cli/command_node.py

In fact, I never see tracing of anything in this directory:

/etc/opt/srlinux/cli/plugins

How is the code for your plugin being loaded in the remote machine?

[hellt]

In fact, I never see tracing of anything in this directory:

/etc/opt/srlinux/cli/plugins

How is the code for your plugin being loaded in the remote machine?

Good call... I think maybe the dynamic nature of the way my file is loaded causes these issues...

My py skills are not meeting the requirement to understand the loader code, but maybe indeed it loads the plugin in mem so that the original file is never actually called...

[rchiodo]

I think it's loaded here:

        old_plugin_id = id(globals().get("Plugin"))
        source = open(self.path).read()
        exec(source, globals(), globals()) <-- This line here is execing the code as a string. 
        new_plugin_id = id(globals().get("Plugin"))

Not sure if you own the plugin code or not, but if that was changed to exec a code object, it might work.

Something like so:

code_object = compile(source, self.path, 'exec')
exec(code_object, globals(), globals())

I believe the code-object is still tied to its original file, but you'd have to try it to be sure.

[hellt]

Not sure if you own the plugin code or not, but if that was changed to exec a code object, it might work.

yes, I can control and edit the loader src code, will try this. Thanks for the hint, appreciate!

I'll post the results when I get them.

[hellt]

Oh wow, that worked on amd64. Thanks a lot @rchiodo, you are the best!
And it worked on arm64 despite the issue with the dll load.

[rchiodo]

Yay! Glad I could help.