Broken reference completion w/ trailing dot inside complex types {} / []

[radeksimko]

Context

Reference completion often includes trailing dot, which is the step separator. The trailing dot is not included in the AST by HCL, which we already account for in the obvious cases:

hcl-lang/decoder/expr_reference_completion.go

Lines 71 to 82 in bd545f4

	editRng := eType.Range()
	if !editRng.ContainsPos(pos) {
	// account for trailing character(s) which doesn't appear in AST
	// such as dot, opening bracket etc.
	editRng.End = pos
	}
	prefixRng := hcl.Range{
	Filename: eType.Range().Filename,
	Start: eType.Range().Start,
	End: pos,
	}
	prefix := string(prefixRng.SliceBytes(file.Bytes))

However we do not seem to account for similar cases involving references with trailing dot inside complex types (list, set, map, maybe object as well):

Debugging Details

For list specifically I was able to narrow down the codepath to this condition, which compares the element range with the given position, which is not contained (it's off by one due to the dot that isn't included in the range):

hcl-lang/decoder/expr_list_completion.go

Lines 77 to 79 in bd545f4

	if elemExpr.Range().ContainsPos(pos) || elemExpr.Range().End.Byte == pos.Byte {
	return newExpression(list.pathCtx, elemExpr, list.cons.Elem).CompletionAtPos(ctx, pos)
	}

It is likely that other complex types have the same root cause.

Proposal

Fix the bug by providing completion of nested references after the trailing dot.

Ideas

We could leverage the recovery mechanism to recover the whole expression:

hcl-lang/decoder/expression.go

Lines 226 to 253 in bd545f4

	// recoverLeftBytes seeks left from given pos in given slice of bytes
	// and recovers all bytes up until f matches, including that match.
	// This allows recovery of incomplete configuration which is not
	// present in the parsed AST during completion.
	//
	// Zero bytes is returned if no match was found.
	func recoverLeftBytes(b []byte, pos hcl.Pos, f func(byteOffset int, r rune) bool) []byte {
	firstRune, size := utf8.DecodeLastRune(b[:pos.Byte])
	offset := pos.Byte - size
	// check for early match
	if f(pos.Byte, firstRune) {
	return b[offset:pos.Byte]
	}
	for offset > 0 {
	nextRune, size := utf8.DecodeLastRune(b[:offset])
	if f(offset, nextRune) {
	// record the matched offset
	// and include the matched last rune
	startByte := offset - size
	return b[startByte:pos.Byte]
	}
	offset -= size
	}
	return []byte{}
	}

The only extra complexity is that we'd have to pass around those recovered bytes (maybe as prefix?) in addition to the AST (hcl.Expression) between all expression types.

[dbanck]

Here is a test case for objects

func TestCompletionAtPos_exprObject_references(t *testing.T) {
	testCases := []struct {
		testName           string
		attrSchema         map[string]*schema.AttributeSchema
		refTargets         reference.Targets
		cfg                string
		pos                hcl.Pos
		expectedCandidates lang.Candidates
	}{
		{
			"single-line element with trailing dot",
			map[string]*schema.AttributeSchema{
				"attr": {
					Constraint: schema.Object{
						Attributes: schema.ObjectAttributes{
							"foo": {
								IsOptional: true,
								Constraint: schema.Reference{OfScopeId: lang.ScopeId("variable")},
							},
						},
					},
				},
			},
			reference.Targets{
				{
					Addr: lang.Address{
						lang.RootStep{Name: "var"},
						lang.AttrStep{Name: "bar"},
					},
					RangePtr: &hcl.Range{
						Filename: "variables.tf",
						Start:    hcl.Pos{Line: 2, Column: 1, Byte: 17},
						End:      hcl.Pos{Line: 2, Column: 3, Byte: 19},
					},
					ScopeId: lang.ScopeId("variable"),
				},
			},
			`attr = { foo = var. }
`,
			hcl.Pos{Line: 1, Column: 20, Byte: 19},
			lang.CompleteCandidates([]lang.Candidate{
				{
					Label:  "var.bar",
					Detail: "reference",
					Kind:   lang.TraversalCandidateKind,
					TextEdit: lang.TextEdit{
						NewText: "var.bar",
						Snippet: "var.bar",
						Range: hcl.Range{
							Filename: "test.tf",
							Start:    hcl.Pos{Line: 1, Column: 16, Byte: 15},
							End:      hcl.Pos{Line: 1, Column: 20, Byte: 19},
						},
					},
				},
			}),
		},
	}

	for i, tc := range testCases {
		t.Run(fmt.Sprintf("%d-%s", i, tc.testName), func(t *testing.T) {
			bodySchema := &schema.BodySchema{
				Attributes: tc.attrSchema,
			}

			f, _ := hclsyntax.ParseConfig([]byte(tc.cfg), "test.tf", hcl.InitialPos)
			d := testPathDecoder(t, &PathContext{
				Schema: bodySchema,
				Files: map[string]*hcl.File{
					"test.tf": f,
				},
				ReferenceTargets: tc.refTargets,
			})

			ctx := context.Background()
			candidates, err := d.CandidatesAtPos(ctx, "test.tf", tc.pos)
			if err != nil {
				t.Fatal(err)
			}

			if diff := cmp.Diff(tc.expectedCandidates, candidates); diff != "" {
				t.Logf("position: %#v in config: %s", tc.pos, tc.cfg)
				t.Fatalf("unexpected candidates: %s", diff)
			}
		})
	}
}