You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Note: I don't mean glob patterns here, but go ahead and read on.
Instead of configuring a file path like:
./bla/some/foo_23.csv
One should be able to configure a file path using a format-pattern like:
./bla/{whatever}/foo_{myid}.csv
This could work like the following:
First, if params are provided as a dict parameter in get_dataframe(), we would replace the {}-patters in a similar way to how it currently works with sql, as a way to deal with files in a directory more dynamically. This would also be useful for the file data sink.
For {}s for which no param is be provided, just match * where {} stands and append every resulting {}-string as a column to the end of the returned data frame (think: unification).
Appending it as a column would also solve the question of what to do if the path with un-parametrized {}s match more than one file: We would simply concatenate the dataframes, but with the {}-determined column the user code would be able to differentiate between files/params if needed. Maybe this should be made an optional parameter either in get_dataframe() or in the datasource config.
The text was updated successfully, but these errors were encountered:
Note: I don't mean glob patterns here, but go ahead and read on.
Instead of configuring a file path like:
One should be able to configure a file path using a format-pattern like:
This could work like the following:
either in get_dataframe() orin the datasource config.The text was updated successfully, but these errors were encountered: