[Nightly rolling test] [UT] XPU OP UT terminate when running test_meta_xpu.py

[RUIJIEZHONG66166]

🐛 Describe the bug

Issue log as below, test would terminate and I found it would fail in different cases, but all the nightly rolling test jobs will be terminated when the entire "Run XPU OP UT"stage reaches 2h46m40s.
Detail: https://github.com/intel/torch-xpu-ops/actions/runs/12482377278/job/34836378515
Other nightly test job(terminated when the entire "Run XPU OP UT"stage reaches 2h46m40s):
https://github.com/intel/torch-xpu-ops/actions/runs/12583237633/job/35070365286

2024-12-25T00:02:47.4994936Z test_meta_xpu.py::TestMetaXPU::test_meta_outplace_nn_functional_adaptive_avg_pool1d_xpu_float32 PASSED [ 93%]
2024-12-25T00:02:47.5061604Z test_meta_xpu.py::TestMetaXPU::test_meta_outplace_nn_functional_adaptive_avg_pool1d_xpu_float64 PASSED [ 93%]
2024-12-25T00:02:47.5143864Z test_meta_xpu.py::TestMetaXPU::test_meta_outplace_nn_functional_adaptive_avg_pool2d_xpu_bfloat16 PASSED [ 93%]
2024-12-25T00:02:47.5229145Z test_meta_xpu.py::TestMetaXPU::test_meta_outplace_nn_functional_adaptive_avg_pool2d_xpu_float16 PASSED [ 93%]
2024-12-25T00:02:47.5309120Z test_meta_xpu.py::TestMetaXPU::test_meta_outplace_nn_functional_adaptive_avg_pool2d_xpu_float32 PASSED [ 93%]
2024-12-25T00:02:47.5391016Z test_meta_xpu.py::TestMetaXPU::test_meta_outplace_nn_functional_adaptive_avg_pool2d_xpu_float64 PASSED [ 93%]
2024-12-25T00:02:47.5493136Z test_meta_xpu.py::TestMetaXPU::test_meta_outplace_nn_functional_adaptive_avg_pool3d_xpu_bfloat16 PASSED [ 93%]
2024-12-25T00:02:47.5593975Z test_meta_xpu.py::TestMetaXPU::test_meta_outplace_nn_functional_adaptive_avg_pool3d_xpu_float16 PASSED [ 93%]
2024-12-25T00:02:47.5690967Z test_meta_xpu.py::TestMetaXPU::test_meta_outplace_nn_functional_adaptive_avg_pool3d_xpu_float32 PASSED [ 93%]
2024-12-25T00:02:47.5793076Z test_meta_xpu.py::TestMetaXPU::test_meta_outplace_nn_functional_adaptive_avg_pool3d_xpu_float64 PASSED [ 93%]
2024-12-25T00:02:47.5873581Z test_meta_xpu.py::TestMetaXPU::test_meta_outplace_nn_functional_adaptive_max_pool1d_xpu_bfloat16 PASSED [ 93%]
2024-12-25T00:02:47.5953083Z test_meta_xpu.py::TestMetaXPU::test_meta_outplace_nn_functional_adaptive_max_pool1d_xpu_float16 PASSED [ 93%]
2024-12-25T00:02:47.6034579Z test_meta_xpu.py::TestMetaXPU::test_meta_outplace_nn_functional_adaptive_max_pool1d_xpu_float32 PASSED [ 93%]
2024-12-25T00:02:47.6115595Z test_meta_xpu.py::TestMetaXPU::test_meta_outplace_nn_functional_adaptive_max_pool1d_xpu_float64 PASSED [ 93%]
2024-12-25T00:02:47.6289890Z test_meta_xpu.py::TestMetaXPU::test_meta_outplace_nn_functional_adaptive_max_pool2d_xpu_bfloat16 PASSED [ 93%]
2024-12-25T00:02:47.6465805Z test_meta_xpu.py::TestMetaXPU::test_meta_outplace_nn_functional_adaptive_max_pool2d_xpu_float16 PASSED [ 93%]
2024-12-25T00:02:47.6639865Z test_meta_xpu.py::TestMetaXPU::test_meta_outplace_nn_functional_adaptive_max_pool2d_xpu_float32 PASSED [ 93%]
2024-12-25T00:02:47.6815464Z test_meta_xpu.py::TestMetaXPU::test_meta_outplace_nn_functional_adaptive_max_pool2d_xpu_float64 PASSED [ 93%]
2024-12-25T00:02:47.6962400Z test_meta_xpu.py::TestMetaXPU::test_meta_outplace_nn_functional_adaptive_max_pool3d_xpu_bfloat16 PASSED [ 93%]
2024-12-25T00:02:47.7076809Z *** longjmp causes uninitialized stack frame ***: terminated
2024-12-25T00:02:47.7081019Z Fatal Python error: Aborted
2024-12-25T00:02:47.7084674Z
2024-12-25T00:02:47.7088841Z Thread 0x00007fef3259e640 (most recent call first):
2024-12-25T00:02:47.7092966Z
2024-12-25T00:02:47.7095090Z
2024-12-25T00:02:47.7095440Z Thread 0x00007fef32d9f640 (most recent call first):
2024-12-25T00:02:47.7095804Z
2024-12-25T00:02:47.7095945Z
2024-12-25T00:02:47.7096077Z Thread 0x00007fef3159c640 (most recent call first):
2024-12-25T00:02:47.7096377Z
2024-12-25T00:02:47.7096511Z
2024-12-25T00:02:47.7096633Z Thread 0x00007fef335a0640 (most recent call first):
2024-12-25T00:02:47.7096922Z
2024-12-25T00:02:47.7097278Z
2024-12-25T00:02:47.7097454Z Thread 0x00007fef31d9d640 (most recent call first):
2024-12-25T00:02:47.7097739Z
2024-12-25T00:02:47.7097935Z
2024-12-25T00:02:47.7098057Z Thread 0x00007fef33da1640 (most recent call first):
2024-12-25T00:02:47.7098344Z
2024-12-25T00:02:47.7098467Z
2024-12-25T00:02:47.7098586Z Thread 0x00007fef345a2640 (most recent call first):
2024-12-25T00:02:47.7098893Z
2024-12-25T00:02:47.7099015Z
2024-12-25T00:02:47.7099138Z Thread 0x00007fef34da3640 (most recent call first):
2024-12-25T00:02:47.7099418Z
2024-12-25T00:02:47.7099548Z
2024-12-25T00:02:47.7099667Z Thread 0x00007ff1df57f640 (most recent call first):
2024-12-25T00:02:47.7100137Z File "/home/sdp/miniforge3/envs/xpu_op_/lib/python3.10/socket.py", line 293 in accept
2024-12-25T00:02:47.7100859Z File "/home/sdp/miniforge3/envs/xpu_op_/lib/python3.10/site-packages/pytest_rerunfailures.py", line 433 in run_server
2024-12-25T00:02:47.7101535Z File "/home/sdp/miniforge3/envs/xpu_op_/lib/python3.10/threading.py", line 953 in run
2024-12-25T00:02:47.7102130Z File "/home/sdp/miniforge3/envs/xpu_op_/lib/python3.10/threading.py", line 1016 in bootstrap_inner
2024-12-25T00:02:47.7102749Z File "/home/sdp/miniforge3/envs/xpu_op/lib/python3.10/threading.py", line 973 in bootstrap
2024-12-25T00:02:47.7103099Z
2024-12-25T00:02:47.7103254Z Current thread 0x00007ff1e2046740 (most recent call first):
2024-12-25T00:02:47.7103743Z File "/home/sdp/miniforge3/envs/xpu_op/lib/python3.10/linecache.py", line 72 in checkcache
2024-12-25T00:02:47.7104326Z File "/home/sdp/miniforge3/envs/xpu_op_/lib/python3.10/traceback.py", line 379 in extract
2024-12-25T00:02:47.7104907Z File "/home/sdp/miniforge3/envs/xpu_op_/lib/python3.10/traceback.py", line 227 in extract_stack
2024-12-25T00:02:47.7105511Z File "/home/sdp/miniforge3/envs/xpu_op_/lib/python3.10/traceback.py", line 213 in format_stack
2024-12-25T00:02:47.7106198Z File "/home/sdp/miniforge3/envs/xpu_op_/lib/python3.10/site-packages/torch/cuda/init.py", line 257 in lazy_call
2024-12-25T00:02:47.7107080Z File "/home/sdp/miniforge3/envs/xpu_op/lib/python3.10/site-packages/torch/cuda/random.py", line 127 in manual_seed_all
2024-12-25T00:02:47.7107839Z File "/home/sdp/miniforge3/envs/xpu_op_/lib/python3.10/site-packages/torch/random.py", line 46 in manual_seed
2024-12-25T00:02:47.7108573Z File "/home/sdp/miniforge3/envs/xpu_op_/lib/python3.10/site-packages/torch/_dynamo/eval_frame.py", line 751 in fn
2024-12-25T00:02:47.7109299Z File "/home/sdp/miniforge3/envs/xpu_op/lib/python3.10/site-packages/torch/compile.py", line 32 in inner
2024-12-25T00:02:47.7110085Z File "/home/sdp/miniforge3/envs/xpu_op/lib/python3.10/site-packages/torch/testing/internal/common_utils.py", line 2278 in set_rng_seed
2024-12-25T00:02:47.7110971Z File "/home/sdp/miniforge3/envs/xpu_op/lib/python3.10/site-packages/torch/testing/internal/common_utils.py", line 412 in next
2024-12-25T00:02:47.7112043Z File "/home/sdp/actions-runner/work/torch-xpu-ops/pytorch/third_party/torch-xpu-ops/test/xpu/../../../../test/test_meta.py", line 1170 in test_meta_outplace
2024-12-25T00:02:47.7112985Z File "/home/sdp/miniforge3/envs/xpu_op/lib/python3.10/site-packages/torch/testing/internal/common_utils.py", line 2249 in wrapper
2024-12-25T00:02:47.7113841Z File "/home/sdp/miniforge3/envs/xpu_op/lib/python3.10/site-packages/torch/testing/internal/common_utils.py", line 1548 in wrapper
2024-12-25T00:02:47.7114735Z File "/home/sdp/miniforge3/envs/xpu_op/lib/python3.10/site-packages/torch/testing/internal/common_device_type.py", line 1162 in test_wrapper
2024-12-25T00:02:47.7115629Z File "/home/sdp/miniforge3/envs/xpu_op/lib/python3.10/site-packages/torch/testing/internal/common_utils.py", line 1620 in wrapper
2024-12-25T00:02:47.7116527Z File "/home/sdp/miniforge3/envs/xpu_op/lib/python3.10/site-packages/torch/testing/internal/common_device_type.py", line 460 in instantiated_test
2024-12-25T00:02:47.7117499Z File "/home/sdp/miniforge3/envs/xpu_op/lib/python3.10/site-packages/torch/testing/internal/common_utils.py", line 3107 in wrapper
2024-12-25T00:02:47.7118292Z File "/home/sdp/miniforge3/envs/xpu_op/lib/python3.10/unittest/case.py", line 549 in callTestMethod
2024-12-25T00:02:47.7118905Z File "/home/sdp/miniforge3/envs/xpu_op/lib/python3.10/unittest/case.py", line 591 in run
2024-12-25T00:02:47.7119625Z File "/home/sdp/miniforge3/envs/xpu_op/lib/python3.10/site-packages/torch/testing/internal/common_utils.py", line 3214 in run_custom
2024-12-25T00:02:47.7120475Z File "/home/sdp/miniforge3/envs/xpu_op/lib/python3.10/site-packages/torch/testing/internal/common_utils.py", line 3242 in run
2024-12-25T00:02:47.7121321Z File "/home/sdp/miniforge3/envs/xpu_op/lib/python3.10/site-packages/torch/testing/internal/common_device_type.py", line 551 in run
2024-12-25T00:02:47.7122045Z File "/home/sdp/miniforge3/envs/xpu_op/lib/python3.10/unittest/case.py", line 650 in call
2024-12-25T00:02:47.7122694Z File "/home/sdp/miniforge3/envs/xpu_op/lib/python3.10/site-packages/pytest/unittest.py", line 333 in runtest
2024-12-25T00:02:47.7123443Z File "/home/sdp/miniforge3/envs/xpu_op/lib/python3.10/site-packages/pytest/runner.py", line 169 in pytest_runtest_call
2024-12-25T00:02:47.7124190Z File "/home/sdp/miniforge3/envs/xpu_op/lib/python3.10/site-packages/pluggy/_callers.py", line 103 in multicall
2024-12-25T00:02:47.7124898Z File "/home/sdp/miniforge3/envs/xpu_op/lib/python3.10/site-packages/pluggy/_manager.py", line 120 in hookexec
2024-12-25T00:02:47.7125595Z File "/home/sdp/miniforge3/envs/xpu_op/lib/python3.10/site-packages/pluggy/hooks.py", line 513 in call
2024-12-25T00:02:47.7126293Z File "/home/sdp/miniforge3/envs/xpu_op/lib/python3.10/site-packages/pytest/runner.py", line 262 in
2024-12-25T00:02:47.7126983Z File "/home/sdp/miniforge3/envs/xpu_op/lib/python3.10/site-packages/pytest/runner.py", line 341 in from_call
2024-12-25T00:02:47.7127707Z File "/home/sdp/miniforge3/envs/xpu_op/lib/python3.10/site-packages/pytest/runner.py", line 261 in call_runtest_hook
2024-12-25T00:02:47.7128457Z File "/home/sdp/miniforge3/envs/xpu_op/lib/python3.10/site-packages/pytest/runner.py", line 222 in call_and_report
2024-12-25T00:02:47.7129205Z File "/home/sdp/miniforge3/envs/xpu_op/lib/python3.10/site-packages/pytest/runner.py", line 133 in runtestprotocol
2024-12-25T00:02:47.7129972Z File "/home/sdp/miniforge3/envs/xpu_op/lib/python3.10/site-packages/pytest/runner.py", line 114 in pytest_runtest_protocol
2024-12-25T00:02:47.7130721Z File "/home/sdp/miniforge3/envs/xpu_op/lib/python3.10/site-packages/pluggy/_callers.py", line 103 in multicall
2024-12-25T00:02:47.7131432Z File "/home/sdp/miniforge3/envs/xpu_op/lib/python3.10/site-packages/pluggy/_manager.py", line 120 in hookexec
2024-12-25T00:02:47.7132115Z File "/home/sdp/miniforge3/envs/xpu_op/lib/python3.10/site-packages/pluggy/hooks.py", line 513 in call
2024-12-25T00:02:47.7132875Z File "/home/sdp/miniforge3/envs/xpu_op/lib/python3.10/site-packages/pytest/main.py", line 348 in pytest_runtestloop
2024-12-25T00:02:47.7133623Z File "/home/sdp/miniforge3/envs/xpu_op/lib/python3.10/site-packages/pluggy/_callers.py", line 103 in multicall
2024-12-25T00:02:47.7134311Z File "/home/sdp/miniforge3/envs/xpu_op/lib/python3.10/site-packages/pluggy/_manager.py", line 120 in hookexec
2024-12-25T00:02:47.7134985Z File "/home/sdp/miniforge3/envs/xpu_op/lib/python3.10/site-packages/pluggy/hooks.py", line 513 in call
2024-12-25T00:02:47.7135645Z File "/home/sdp/miniforge3/envs/xpu_op/lib/python3.10/site-packages/_pytest/main.py", line 323 in main
2024-12-25T00:02:47.7136309Z File "/home/sdp/miniforge3/envs/xpu_op/lib/python3.10/site-packages/pytest/main.py", line 269 in wrap_session
2024-12-25T00:02:47.7137017Z File "/home/sdp/miniforge3/envs/xpu_op/lib/python3.10/site-packages/pytest/main.py", line 316 in pytest_cmdline_main
2024-12-25T00:02:47.7137732Z File "/home/sdp/miniforge3/envs/xpu_op/lib/python3.10/site-packages/pluggy/_callers.py", line 103 in multicall
2024-12-25T00:02:47.7138447Z File "/home/sdp/miniforge3/envs/xpu_op/lib/python3.10/site-packages/pluggy/_manager.py", line 120 in hookexec
2024-12-25T00:02:47.7139149Z File "/home/sdp/miniforge3/envs/xpu_op/lib/python3.10/site-packages/pluggy/hooks.py", line 513 in call
2024-12-25T00:02:47.7139841Z File "/home/sdp/miniforge3/envs/xpu_op/lib/python3.10/site-packages/pytest/config/init.py", line 166 in main
2024-12-25T00:02:47.7140568Z File "/home/sdp/miniforge3/envs/xpu_op/lib/python3.10/site-packages/pytest/config/init.py", line 189 in console_main
2024-12-25T00:02:47.7141181Z File "/home/sdp/miniforge3/envs/xpu_op/bin/pytest", line 8 in
2024-12-25T00:02:47.7141455Z
2024-12-25T00:02:47.7155580Z Extension modules: yaml._yaml, numpy.core._multiarray_umath, numpy.core._multiarray_tests, numpy.linalg._umath_linalg, numpy.fft._pocketfft_internal, numpy.random._common, numpy.random.bit_generator, numpy.random._bounded_integers, numpy.random._mt19937, numpy.random.mtrand, numpy.random._philox, numpy.random._pcg64, numpy.random._sfc64, numpy.random._generator, torch._C, torch._C._dynamo.autograd_compiler, torch._C._dynamo.eval_frame, torch._C._dynamo.guards, torch._C._dynamo.utils, torch._C._fft, torch._C._linalg, torch._C._nested, torch._C._nn, torch._C._sparse, torch._C._special, multidict._multidict, yarl._quoting_c, propcache._helpers_c, aiohttp._http_writer, aiohttp._http_parser, aiohttp._websocket.mask, aiohttp._websocket.reader_c, frozenlist._frozenlist, charset_normalizer.md, requests.packages.charset_normalizer.md, requests.packages.chardet.md, thriftpy2.transport.cybase, thriftpy2.transport.buffered.cybuffered, thriftpy2.transport.framed.cyframed, thriftpy2.transport.memory.cymemory, thriftpy2.transport.sasl.cysasl, thriftpy2.protocol.cybin.cybin, cython.cimports.libc.math, Cython.Utils, Cython.Plex.Actions, Cython.Plex.Transitions, Cython.Plex.Machines, Cython.Plex.DFA, Cython.Plex.Scanners, Cython.Compiler.Scanning, Cython.StringIOTree, Cython.Compiler.Code, numba.core.typeconv._typeconv, numba._helperlib, numba._dynfunc, numba._dispatcher, numba.core.runtime._nrt_python, numba.np.ufunc._internal, scipy._lib._ccallback_c, numba.mviewbuf, psutil._psutil_linux, psutil._psutil_posix, scipy._lib._uarray._uarray, scipy.special._ufuncs_cxx, scipy.special._ufuncs, scipy.special._specfun, scipy.special._comb, scipy.linalg._fblas, scipy.linalg._flapack, scipy.linalg._cythonized_array_utils, scipy.linalg._flinalg, scipy.linalg._solve_toeplitz, scipy.linalg._matfuncs_sqrtm_triu, scipy.linalg.cython_lapack, scipy.linalg.cython_blas, scipy.linalg._matfuncs_expm, scipy.linalg._decomp_update, scipy.sparse._sparsetools, _csparsetools, scipy.sparse._csparsetools, scipy.sparse.linalg._isolve._iterative, scipy.sparse.linalg._dsolve._superlu, scipy.sparse.linalg._eigen.arpack._arpack, scipy.sparse.csgraph._tools, scipy.sparse.csgraph._shortest_path, scipy.sparse.csgraph._traversal, scipy.sparse.csgraph._min_spanning_tree, scipy.sparse.csgraph._flow, scipy.sparse.csgraph._matching, scipy.sparse.csgraph._reordering, scipy.special._ellip_harm_2, scipy.signal._sigtools, scipy.signal._max_len_seq_inner, scipy.signal._upfirdn_apply, scipy.signal._spline, scipy.optimize._minpack2, scipy.optimize._group_columns, scipy._lib.messagestream, scipy.optimize._trlib._trlib, numpy.linalg.lapack_lite, scipy.optimize._lbfgsb, _moduleTNC, scipy.optimize._moduleTNC, scipy.optimize._cobyla, scipy.optimize._slsqp, scipy.optimize._minpack, scipy.optimize._lsq.givens_elimination, scipy.optimize._zeros, scipy.optimize.__nnls, scipy.optimize._highs.cython.src._highs_wrapper, scipy.optimize._highs._highs_wrapper, scipy.optimize._highs.cython.src._highs_constants, scipy.optimize._highs._highs_constants, scipy.linalg._interpolative, scipy.optimize._bglu_dense, scipy.optimize._lsap, scipy.spatial._ckdtree, scipy.spatial._qhull, scipy.spatial._voronoi, scipy.spatial._distance_wrap, scipy.spatial._hausdorff, scipy.spatial.transform._rotation, scipy.optimize._direct, scipy.integrate._odepack, scipy.integrate._quadpack, scipy.integrate._vode, scipy.integrate._dop, scipy.integrate._lsoda, scipy.interpolate._fitpack, scipy.interpolate.dfitpack, scipy.interpolate._bspl, scipy.interpolate._ppoly, scipy.interpolate.interpnd, scipy.interpolate._rbfinterp_pythran, scipy.interpolate._rgi_cython, scipy.signal._sosfilt, scipy.ndimage._nd_image, _ni_label, scipy.ndimage._ni_label, scipy.signal._spectral, scipy.special.cython_special, scipy.stats._stats, scipy.stats.beta_ufunc, scipy.stats._boost.beta_ufunc, scipy.stats.binom_ufunc, scipy.stats._boost.binom_ufunc, scipy.stats.nbinom_ufunc, scipy.stats._boost.nbinom_ufunc, scipy.stats.hypergeom_ufunc, scipy.stats._boost.hypergeom_ufunc, scipy.stats.ncf_ufunc, scipy.stats._boost.ncf_ufunc, scipy.stats.ncx2_ufunc, scipy.stats._boost.ncx2_ufunc, scipy.stats.nct_ufunc, scipy.stats._boost.nct_ufunc, scipy.stats.skewnorm_ufunc, scipy.stats._boost.skewnorm_ufunc, scipy.stats.invgauss_ufunc, scipy.stats._boost.invgauss_ufunc, scipy.stats._biasedurn, scipy.stats._levy_stable.levyst, scipy.stats._stats_pythran, scipy.stats._statlib, scipy.stats._mvn, scipy.stats._sobol, scipy.stats._qmc_cy, scipy.stats._rcont.rcont, scipy.signal._peak_finding_utils (total: 169)
2024-12-25T00:02:48.0705108Z test_meta_xpu.py::TestMetaXPU::test_meta_outplace_nn_functional_adaptive_max_pool3d_xpu_float16
2024-12-25T00:02:48.0755047Z ##[error]Process completed with exit code 124.
2024-12-25T00:02:48.1036162Z Post job cleanup.
2024-12-25T00:02:48.2293017Z [command]/usr/bin/git version
2024-12-25T00:02:48.2330373Z git version 2.34.1
2024-12-25T00:02:48.2376930Z Temporarily overriding HOME='/home/sdp/actions-runner/_work/_temp/0681b850-bd77-4a4b-81c8-e92fb315632f' before making global git config changes
2024-12-25T00:02:48.2379661Z Adding repository directory to the temporary git global config as a safe directory
2024-12-25T00:02:48.2382185Z [command]/usr/bin/git config --global --add safe.directory /home/sdp/actions-runner/_work/torch-xpu-ops/torch-xpu-ops
2024-12-25T00:02:48.2425103Z [command]/usr/bin/git config --local --name-only --get-regexp core.sshCommand
2024-12-25T00:02:48.2456037Z [command]/usr/bin/git submodule foreach --recursive sh -c "git config --local --name-only --get-regexp 'core.sshCommand' && git config --local --unset-all 'core.sshCommand' || :"
2024-12-25T00:02:48.2809718Z [command]/usr/bin/git config --local --name-only --get-regexp http.https://github.com/.extraheader
2024-12-25T00:02:48.2837985Z http.https://github.com/.extraheader
2024-12-25T00:02:48.2852791Z [command]/usr/bin/git config --local --unset-all http.https://github.com/.extraheader
2024-12-25T00:02:48.2888810Z [command]/usr/bin/git submodule foreach --recursive sh -c "git config --local --name-only --get-regexp 'http.https://github.com/.extraheader' && git config --local --unset-all 'http.https://github.com/.extraheader' || :"
2024-12-25T00:02:48.3475463Z Cleaning up orphan processes

Versions

Pytorch main branch
Driver: rolling driver