Non-reproducibility in PF GPU clustering in 12834.42[23] workflows

[makortel]

PR GPU tests in #47226 (comment) showed differences in workflow 12834.422 in ParticleFlow/PFClusterV such as

And in workflow 12834.423 in ParticleFlow/PFClusterV as in above and also in ParticleFlow/pfClusterHBHEAlpakaV such as

[makortel]

assign heterogeneous

[makortel]

@jsamudio

[cmsbuild]

New categories assigned: heterogeneous

@fwyzard,@makortel you have been requested to review this Pull request/Issue and eventually sign? Thanks

[cmsbuild]

cms-bot internal usage

[cmsbuild]

A new Issue was created by @makortel.

@Dr15Jones, @antoniovilela, @makortel, @mandrenguyen, @rappoccio, @sextonkennedy, @smuzaffar can you please review it and eventually sign/assign? Thanks.

cms-bot commands are listed here

[makortel]

#47227 (comment) showed in 12834.422 ParticleFlow/PFClusterV different differences

[jsamudio]

I believe the plots in ParticleFlow/pfClusterHBHEAlpakaV look about as expected between the legacy format and the Alpaka version. IIRC and if the configuration in github is to be believed then this is the comparison it is doing. See slide 7 in http://cds.cern.ch/record/2898660. For plots in ParticleFlow/PFClusterV, I am not so sure.

[makortel]

I believe the plots in ParticleFlow/pfClusterHBHEAlpakaV look about as expected between the legacy format and the Alpaka version

The problem is that these plots are different between different executions (whatever that meant in the tests of those PRs). The RelMon doesn't seem to show very well what the differences really are for 2D plots, such as pfCluster_RecHitMultiplicity_GPUvsCPU, but at least the number of entries seems to be different (1376 vs 1161).

[jsamudio]

I believe the plots in ParticleFlow/pfClusterHBHEAlpakaV look about as expected between the legacy format and the Alpaka version

The problem is that these plots are different between different executions (whatever that meant in the tests of those PRs). The RelMon doesn't seem to show very well what the differences really are for 2D plots, such as pfCluster_RecHitMultiplicity_GPUvsCPU, but at least the number of entries seems to be different (1376 vs 1161).

Okay I see now. I think I need to do some testing.