MultiShift, Masked shift and masked interleave #2431

mazimkhan · 2025-01-06T16:11:14Z

Introduces the following operations for generic_ops-inl.h and arm_sve-inl.h where the use of sve intrinsics will provide a performance gain:

MultiShift - An arbitrary set of shifts performed on each byte in each 64-bit wide lane of a vector. A fuller description can be found in g3doc/quick_reference.md.
Masked shifts - shifting operations where a mask is applied for alternative behaviour where the mask is false.
Masked interleave operations which interleave either the even or odd elements of the two input vectors, unless the mask is false for a given lane. In this case, the lane is set to zero.

Tests have been introduced for all the listed operations.

google-cla · 2025-01-06T16:11:18Z

Thanks for your pull request! It looks like this may be your first contribution to a Google open source project. Before we can look at your pull request, you'll need to sign a Contributor License Agreement (CLA).

View this failed invocation of the CLA check for more information.

For the most up to date status, view the checks section at the bottom of the pull request.

johnplatts · 2025-01-06T17:01:26Z

AVX3_DL has _mm_multishift_epi64_epi8, _mm256_multishift_epi64_epi8, and _mm512_multishift_epi64_epi8 intrinsics that can be used to implement MultiShift on AVX3_DL.

jan-wassenberg

Nice implementation of MultiShift :) As John mentions, please add a TODO for the corresponding x86 _mm512_multishift so we can track that.

jan-wassenberg · 2025-01-08T18:23:08Z

g3doc/quick_reference.md

+
+*   `V`: `{u,i}64`, `VI`: `{u,i}8` \
+    <code>V **MultiShift**(V vals, VI indices)</code>: returns a
+    vector with `(vals[i] >> indices[i*8+j]) & 0xff` in byte `j` of `r[i]` for each


Let's define r, for example "vector r".

jan-wassenberg · 2025-01-08T18:24:25Z

g3doc/quick_reference.md

+    for(size_t i = 0; i < N; i++) {
+      uint64_t shift_result = 0;
+      for(int j = 0; j < 8; j++) {
+        uint64_t rot_result = (v[i] >> indices[i*8+j]) | (v[i] << (64 - indices[i*8+j]));


Because we're only using the lower 8 bits, we can just talk about the right-shift and not a rotate, right?

Unfortunately not, in the cases where indices[x] are greater than 55 you start to get the lower bits wrapping round.

i.e. For indices[x] = 59 the output bits are: 2, 1, 0, 63, 62, 61, 60, 59

Oh, I see - the x86 definition has & 63. Do we want to call it MultiRotateRight then?

jan-wassenberg · 2025-01-08T18:25:19Z

g3doc/quick_reference.md

+
+#### Masked Shifts
+*   `V`: `{u,i}` \
+    <code>V **MaskedShiftLeftOrZero**&lt;int&gt;(M mask, V a)</code> returns `a[i] << int` or `0` if


As before, let's drop the OrZero suffix?

jan-wassenberg · 2025-01-08T18:26:34Z

g3doc/quick_reference.md

@@ -2081,6 +2124,12 @@ Ops in this section are only available if `HWY_TARGET != HWY_SCALAR`:
    `InterleaveOdd(d, a, b)` is usually more efficient than `OddEven(b,
    DupOdd(a))`.

+*   <code>V **InterleaveEvenOrZero**(M m, V a, V b)</code>: Performs the same


Add Masked prefix and remove OrZero suffix? I notice this doesn't have an SVE implementation yet?

I've removed this, we'll add it to a future PR once we have an x86 implementation.

jan-wassenberg · 2025-01-08T18:27:50Z

hwy/ops/generic_ops-inl.h

@@ -574,6 +592,27 @@ HWY_API V MaskedSatSubOr(V no, M m, V a, V b) {
 }
 #endif  // HWY_NATIVE_MASKED_ARITH

+// ------------------------------ MaskedShift
+template <int kshift, class V, class M>


Nit: please rename kshift -> kShift.

jan-wassenberg · 2025-01-08T18:30:15Z

hwy/ops/generic_ops-inl.h

@@ -7299,6 +7338,62 @@ HWY_API V BitShuffle(V v, VI idx) {

 #endif  // HWY_NATIVE_BITSHUFFLE

+// ------------------------------ MultiShift (Rol)


The ops in parentheses are ops that the implementation of the one in this section uses, for purposes of sorting the ops in the source file. I think this is a copy-paste remnant? We can remove it because this implementation does not seem to use any ops defined in generic_ops-inl.h.

Ah, that makes sense. I've corrected this.

jan-wassenberg · 2025-01-08T18:33:20Z

hwy/ops/generic_ops-inl.h

+  const auto extract_mask = Dup128VecFromValues(du8, 0, 2, 4, 6, 8, 10, 12, 14,
+                                                0, 0, 0, 0, 0, 0, 0, 0);
+  const auto even_lanes =
+      BitCast(d64, TableLookupBytes(extracted_even_bytes, extract_mask));


I think we could use ConcatEven here?

That's a better way to do it. I've made that change.

Remove OrZero suffixes Remove masked interleaves. Will be in a future PR with x86 specialisations Correct template naming Correct multishift header comment Optimise MultiShift Add x86 TODOs for multishift Improve MultiShift docs

jan-wassenberg · 2025-01-31T14:37:53Z

g3doc/quick_reference.md

+    for(size_t i = 0; i < N; i++) {
+      uint64_t shift_result = 0;
+      for(int j = 0; j < 8; j++) {
+        uint64_t rot_result = (v[i] >> indices[i*8+j]) | (v[i] << (64 - indices[i*8+j]));


Oh, I see - the x86 definition has & 63. Do we want to call it MultiRotateRight then?

jan-wassenberg · 2025-01-31T14:38:48Z

g3doc/quick_reference.md

+    `mask[i]` is false.
+
+*   `V`: `{u,i}` \
+    <code>V **MaskedShiftRightOr**&lt;int&gt;(V no, M mask, V a)</code> returns `a[i] >> int` or `no` if


Minor clarification: no[i] here and below?

jan-wassenberg · 2025-01-31T14:39:37Z

hwy/ops/arm_sve-inl.h

+
+#undef HWY_SVE_SHIFT_Z
+
+// ------------------------------ MaskedShiftRightSameOr


Stale comment, it's now called MaskedShiftRightOr?

jan-wassenberg requested changes Jan 8, 2025

View reviewed changes

jan-wassenberg reviewed Jan 8, 2025

View reviewed changes

wbb-ccl force-pushed the cc_up_shift_and_interleave branch from 70d6581 to c4a44dd Compare January 31, 2025 11:01

mazimkhan and others added 3 commits January 31, 2025 12:21

MultiShift, Masked shift and masked interleave

f978890

Fix review comments

5047921

Remove OrZero suffixes Remove masked interleaves. Will be in a future PR with x86 specialisations Correct template naming Correct multishift header comment Optimise MultiShift Add x86 TODOs for multishift Improve MultiShift docs

Correct incorrect op

1e907b0

wbb-ccl force-pushed the cc_up_shift_and_interleave branch from 37e7368 to 1e907b0 Compare January 31, 2025 12:21

jan-wassenberg requested changes Jan 31, 2025

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

MultiShift, Masked shift and masked interleave #2431

MultiShift, Masked shift and masked interleave #2431

mazimkhan commented Jan 6, 2025

google-cla bot commented Jan 6, 2025

johnplatts commented Jan 6, 2025

jan-wassenberg left a comment

jan-wassenberg Jan 8, 2025

wbb-ccl Jan 31, 2025

jan-wassenberg Jan 8, 2025

wbb-ccl Jan 31, 2025

jan-wassenberg Jan 31, 2025

jan-wassenberg Jan 8, 2025

wbb-ccl Jan 31, 2025

jan-wassenberg Jan 8, 2025

wbb-ccl Jan 31, 2025

jan-wassenberg Jan 8, 2025

wbb-ccl Jan 31, 2025

jan-wassenberg Jan 8, 2025

wbb-ccl Jan 31, 2025

jan-wassenberg Jan 8, 2025

wbb-ccl Jan 31, 2025

jan-wassenberg Jan 31, 2025

jan-wassenberg Jan 31, 2025

jan-wassenberg Jan 31, 2025

		@@ -7299,6 +7338,62 @@ HWY_API V BitShuffle(V v, VI idx) {

		#endif // HWY_NATIVE_BITSHUFFLE

		// ------------------------------ MultiShift (Rol)


		#undef HWY_SVE_SHIFT_Z

		// ------------------------------ MaskedShiftRightSameOr

MultiShift, Masked shift and masked interleave #2431

Are you sure you want to change the base?

MultiShift, Masked shift and masked interleave #2431

Conversation

mazimkhan commented Jan 6, 2025

google-cla bot commented Jan 6, 2025

johnplatts commented Jan 6, 2025

jan-wassenberg left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment