bug: Can't set up an aggregate UDF with Any as input and return type

[sergei3000]

What happened?

I'm trying to register an aggregate UDF like so:

from typing import Any
import ibis

@ibis.udf.agg.builtin
def latest(_: Any) -> Any:
    """Get the latest value in the column, just like Druids LATEST clause."""

and am getting this error:

E   ibis.common.annotations.SignatureValidationError: ValueOf(typing.Any) has failed due to the following errors:
E     `dtype`: typing.Any is not either None or coercible to a DataType
E   
E   Expected signature: ValueOf(dtype: Optional[DataType] = None)

LATEST is a Druid SQL clause and it can take and return any type, so it seems that Ibis UDF's should be able to accept Any in type annotations.

What version of ibis are you using?

ibis=9.5.0
python=3.11
pydruid=0.6.9

What backend(s) are you using, if any?

Apache Druid

Relevant log output

@ibis.udf.agg.builtin
/opt/miniconda3/envs/test-sergey-ibis/lib/python3.11/site-packages/ibis/expr/operations/udf.py:634: in builtin
    return _wrap(
/opt/miniconda3/envs/test-sergey-ibis/lib/python3.11/site-packages/ibis/expr/operations/udf.py:85: in _wrap
    return wrap(fn) if fn is not None else wrap
/opt/miniconda3/envs/test-sergey-ibis/lib/python3.11/site-packages/ibis/expr/operations/udf.py:82: in wrap
    deferrable(wrapper(input_type, fn, **kwargs)), fn
/opt/miniconda3/envs/test-sergey-ibis/lib/python3.11/site-packages/ibis/expr/operations/udf.py:161: in _make_wrapper
    node = cls._make_node(fn, input_type, **kwargs)
/opt/miniconda3/envs/test-sergey-ibis/lib/python3.11/site-packages/ibis/expr/operations/udf.py:122: in _make_node
    fields = {
/opt/miniconda3/envs/test-sergey-ibis/lib/python3.11/site-packages/ibis/expr/operations/udf.py:124: in <dictcomp>
    pattern=rlz.ValueOf(annotations.get(arg_name)),
/opt/miniconda3/envs/test-sergey-ibis/lib/python3.11/site-packages/ibis/common/bases.py:72: in __call__
    return cls.__create__(*args, **kwargs)
/opt/miniconda3/envs/test-sergey-ibis/lib/python3.11/site-packages/ibis/common/grounds.py:119: in __create__
    kwargs = cls.__signature__.validate(cls, args, kwargs)
/opt/miniconda3/envs/test-sergey-ibis/lib/python3.11/site-packages/ibis/common/annotations.py:501: in validate
    raise SignatureValidationError(
E   ibis.common.annotations.SignatureValidationError: ValueOf(typing.Any) has failed due to the following errors:
E     `dtype`: typing.Any is not either None or coercible to a DataType
E   
E   Expected signature: ValueOf(dtype: Optional[DataType] = None)

Code of Conduct

• I agree to follow this project's Code of Conduct

[cpcloud]

Thanks for opening an issue about this.

For the input, you can simply leave off the annotation, and that will behave as if it were Any.

For the output, there's currently no way to spell that in Ibis, because Ibis requires type information about expressions it produces, and Any has no meaningful type information.

I would recommended annotating the return type with the type you plan to use latest with.

[sergei3000]

I would recommended annotating the return type with the type you plan to use latest with

thing is, I plan to use latest with different types in the same module

[cpcloud]

Currently we don't have a way to overload any UDF signature, so you'll have to create a separate function for each type, like latest_str, latest_date, etc.

Eventually we want make this kind of thing transparent to the user, but it hasn't been implemented yet.

[sergei3000]

And how do I tell ibis that those should be translated to Druid's LATEST? I tried this:

@ibis.udf.agg.builtin
def latest_float(expr, name="latest") -> float:
    """Get the latest value in the column, just like Druids LATEST clause."""

But got this error:

>           raise exceptions.ProgrammingError(msg)
E           pydruid.db.exceptions.ProgrammingError: druidException (INVALID_INPUT): No match found for function signature LATEST_FLOAT(<NUMERIC>, <CHARACTER>) (line [1], column [51])

/opt/miniconda3/envs/test-sergey-ibis/lib/python3.11/site-packages/pydruid/db/api.py:386: ProgrammingError

[sergei3000]

Never mind, I've found my error, I should have used name= in the decorator