TeslaFi Import fails if one line is mailformed

[dbcm]

Is there an existing issue for this?

• I have searched the existing issues

What happened?

I'm trying to import teslaFi data, import process fails. Tested with latest and edge releases.

Console logs (removed coordinates):

teslamate | 2025-01-05 21:38:24.662 [error] GenStateMachine :import_Fly terminating
teslamate | ** (MatchError) no match of right hand side value: {:error, #Ecto.Changeset<action: :insert, changes: %{date: ~U[2024-11-23 20:29:43.000000Z], power: 1, latitude: Decimal.new("XXX"), longitude: Decimal.new("XXX"), odometer: 73381.499485, ideal_battery_range_km: Decimal.new("109.48"), outside_temp: Decimal.new("17.5"), rated_battery_range_km: Decimal.new("109.48"), usable_battery_level: 28, driver_temp_setting: Decimal.new("21.0"), est_battery_range_km: Decimal.new("111.37"), fan_status: 0, inside_temp: Decimal.new("19.7"), passenger_temp_setting: Decimal.new("21.0")}, errors: [battery_level: {"is invalid", [type: :integer, validation: :cast]}], data: #TeslaMate.Log.Position<>, valid?: false, ...>}
teslamate | (teslamate 1.32.1-dev) lib/teslamate/vehicles/vehicle.ex:766: anonymous fn/4 in TeslaMate.Vehicles.Vehicle.handle_event/4
teslamate | (ecto_sql 3.12.1) lib/ecto/adapters/sql.ex:1400: anonymous fn/3 in Ecto.Adapters.SQL.checkout_or_transaction/4
teslamate | (db_connection 2.7.0) lib/db_connection.ex:1756: DBConnection.run_transaction/4
teslamate | (teslamate 1.32.1-dev) lib/teslamate/vehicles/vehicle.ex:758: TeslaMate.Vehicles.Vehicle.handle_event/4
teslamate | (stdlib 5.2.3.1) gen_statem.erl:1397: :gen_statem.loop_state_callback/11
teslamate | (stdlib 5.2.3.1) proc_lib.erl:241: :proc_lib.init_p_do_apply/3
teslamate | 2025-01-05 21:38:24.664 [warning] Import failed: {{:badmatch,
teslamate | {:error,
teslamate | #Ecto.Changeset<
teslamate | action: :insert,
teslamate | changes: %{
teslamate | date: ~U[2024-11-23 20:29:43.000000Z],
teslamate | power: 1,
teslamate | latitude: Decimal.new("XXX"),
teslamate | longitude: Decimal.new("XXX"),
teslamate | odometer: 73381.499485,
teslamate | ideal_battery_range_km: Decimal.new("109.48"),
teslamate | outside_temp: Decimal.new("17.5"),
teslamate | rated_battery_range_km: Decimal.new("109.48"),
teslamate | usable_battery_level: 28,
teslamate | driver_temp_setting: Decimal.new("21.0"),
teslamate | est_battery_range_km: Decimal.new("111.37"),
teslamate | fan_status: 0,
teslamate | inside_temp: Decimal.new("19.7"),
teslamate | passenger_temp_setting: Decimal.new("21.0")
teslamate | },
teslamate | errors: [
teslamate | battery_level: {"is invalid", [type: :integer, validation: :cast]}
teslamate | ],
teslamate | data: #TeslaMate.Log.Position<>,
teslamate | valid?: false,
teslamate | ...
teslamate | >}},
teslamate | [
teslamate | {TeslaMate.Vehicles.Vehicle, :"-handle_event/4-fun-3-", 4,
teslamate | [file: ~c"lib/teslamate/vehicles/vehicle.ex", line: 766]},
teslamate | {Ecto.Adapters.SQL, :"-checkout_or_transaction/4-fun-0-", 3,
teslamate | [file: ~c"lib/ecto/adapters/sql.ex", line: 1400]},
teslamate | {DBConnection, :run_transaction, 4,
teslamate | [file: ~c"lib/db_connection.ex", line: 1756]},
teslamate | {TeslaMate.Vehicles.Vehicle, :handle_event, 4,
teslamate | [file: ~c"lib/teslamate/vehicles/vehicle.ex", line: 758]},
teslamate | {:gen_statem, :loop_state_callback, 11,
teslamate | [file: ~c"gen_statem.erl", line: 1397]},
teslamate | {:proc_lib, :init_p_do_apply, 3, [file: ~c"proc_lib.erl", line: 241]}
teslamate | ]}
teslamate | 2025-01-05 21:42:18.409 [info] GET /import

Web error:

{{:badmatch, {:error, #Ecto.Changeset< action: :insert, changes: %{ date: ~U[2024-11-23 20:29:43.000000Z], power: 1, latitude: Decimal.new("XXX"), longitude: Decimal.new("XXX"), odometer: 73381.499485, ideal_battery_range_km: Decimal.new("109.48"), outside_temp: Decimal.new("17.5"), rated_battery_range_km: Decimal.new("109.48"), usable_battery_level: 28, driver_temp_setting: Decimal.new("21.0"), est_battery_range_km: Decimal.new("111.37"), fan_status: 0, inside_temp: Decimal.new("19.7"), passenger_temp_setting: Decimal.new("21.0") }, errors: [ battery_level: {"is invalid", [type: :integer, validation: :cast]} ], data: #TeslaMate.Log.Position<>, valid?: false, ... >}}, [ {TeslaMate.Vehicles.Vehicle, :"-handle_event/4-fun-3-", 4, [file: ~c"lib/teslamate/vehicles/vehicle.ex", line: 766]}, {Ecto.Adapters.SQL, :"-checkout_or_transaction/4-fun-0-", 3, [file: ~c"lib/ecto/adapters/sql.ex", line: 1400]}, {DBConnection, :run_transaction, 4, [file: ~c"lib/db_connection.ex", line: 1756]}, {TeslaMate.Vehicles.Vehicle, :handle_event, 4, [file: ~c"lib/teslamate/vehicles/vehicle.ex", line: 758]}, {:gen_statem, :loop_state_callback, 11, [file: ~c"gen_statem.erl", line: 1397]}, {:proc_lib, :init_p_do_apply, 3, [file: ~c"proc_lib.erl", line: 241]} ]}

CVS line:

682739,"2024-11-23 20:29:43",,1,"1227 95 2 0 0 0 62",Fly,"1 0 0 High",,,LRWYGCFS0PC551608,IEC,"17.499999608845 17073.890",0,"3.1 3.1 3.05 3.05",online,,0.0,16,0,,,0,,68.03,0,100,16,Engaged,,0,0,,28,,,,28.01,0.0,1,,,68.03,,NoPower,0,1,,1732406400,69.2,0.0,1,16,1,0.0,,0.0,19.7,XXX,250,,XXX,,,0,,0,21.0,0,21.0,17.5,0,0,,,0,0,0,0,,,,,,,0,,0,,,0,45597.149823,,0,off,,,0,,0,,,,,,0,,0,,,,,0,,,Apollo19MetallicShad,0,,,"2024.38.7 f78dd5c34b",0,,67,,63,,4456,0,2510,0,,0,0,0,248.91,,,0,,,0,1,0,0,0,74.2

Expected Behavior

Should import the rest of the file

Steps To Reproduce

create a cvs file with that line

Relevant log output

teslamate  | 2025-01-05 21:38:24.662 [error] GenStateMachine :import_Fly terminating
teslamate  | ** (MatchError) no match of right hand side value: {:error, #Ecto.Changeset<action: :insert, changes: %{date: ~U[2024-11-23 20:29:43.000000Z], power: 1, latitude: Decimal.new("XXX"), longitude: Decimal.new("XXX"), odometer: 73381.499485, ideal_battery_range_km: Decimal.new("109.48"), outside_temp: Decimal.new("17.5"), rated_battery_range_km: Decimal.new("109.48"), usable_battery_level: 28, driver_temp_setting: Decimal.new("21.0"), est_battery_range_km: Decimal.new("111.37"), fan_status: 0, inside_temp: Decimal.new("19.7"), passenger_temp_setting: Decimal.new("21.0")}, errors: [battery_level: {"is invalid", [type: :integer, validation: :cast]}], data: #TeslaMate.Log.Position<>, valid?: false, ...>}
teslamate  |     (teslamate 1.32.1-dev) lib/teslamate/vehicles/vehicle.ex:766: anonymous fn/4 in TeslaMate.Vehicles.Vehicle.handle_event/4
teslamate  |     (ecto_sql 3.12.1) lib/ecto/adapters/sql.ex:1400: anonymous fn/3 in Ecto.Adapters.SQL.checkout_or_transaction/4
teslamate  |     (db_connection 2.7.0) lib/db_connection.ex:1756: DBConnection.run_transaction/4
teslamate  |     (teslamate 1.32.1-dev) lib/teslamate/vehicles/vehicle.ex:758: TeslaMate.Vehicles.Vehicle.handle_event/4
teslamate  |     (stdlib 5.2.3.1) gen_statem.erl:1397: :gen_statem.loop_state_callback/11
teslamate  |     (stdlib 5.2.3.1) proc_lib.erl:241: :proc_lib.init_p_do_apply/3
teslamate  | 2025-01-05 21:38:24.664 [warning] Import failed: {{:badmatch,
teslamate  |   {:error,
teslamate  |    #Ecto.Changeset<
teslamate  |      action: :insert,
teslamate  |      changes: %{
teslamate  |        date: ~U[2024-11-23 20:29:43.000000Z],
teslamate  |        power: 1,
teslamate  |        latitude: Decimal.new("XXX"),
teslamate  |        longitude: Decimal.new("XXX"),
teslamate  |        odometer: 73381.499485,
teslamate  |        ideal_battery_range_km: Decimal.new("109.48"),
teslamate  |        outside_temp: Decimal.new("17.5"),
teslamate  |        rated_battery_range_km: Decimal.new("109.48"),
teslamate  |        usable_battery_level: 28,
teslamate  |        driver_temp_setting: Decimal.new("21.0"),
teslamate  |        est_battery_range_km: Decimal.new("111.37"),
teslamate  |        fan_status: 0,
teslamate  |        inside_temp: Decimal.new("19.7"),
teslamate  |        passenger_temp_setting: Decimal.new("21.0")
teslamate  |      },
teslamate  |      errors: [
teslamate  |        battery_level: {"is invalid", [type: :integer, validation: :cast]}
teslamate  |      ],
teslamate  |      data: #TeslaMate.Log.Position<>,
teslamate  |      valid?: false,
teslamate  |      ...
teslamate  |    >}},
teslamate  |  [
teslamate  |    {TeslaMate.Vehicles.Vehicle, :"-handle_event/4-fun-3-", 4,
teslamate  |     [file: ~c"lib/teslamate/vehicles/vehicle.ex", line: 766]},
teslamate  |    {Ecto.Adapters.SQL, :"-checkout_or_transaction/4-fun-0-", 3,
teslamate  |     [file: ~c"lib/ecto/adapters/sql.ex", line: 1400]},
teslamate  |    {DBConnection, :run_transaction, 4,
teslamate  |     [file: ~c"lib/db_connection.ex", line: 1756]},
teslamate  |    {TeslaMate.Vehicles.Vehicle, :handle_event, 4,
teslamate  |     [file: ~c"lib/teslamate/vehicles/vehicle.ex", line: 758]},
teslamate  |    {:gen_statem, :loop_state_callback, 11,
teslamate  |     [file: ~c"gen_statem.erl", line: 1397]},
teslamate  |    {:proc_lib, :init_p_do_apply, 3, [file: ~c"proc_lib.erl", line: 241]}
teslamate  |  ]}
teslamate  | 2025-01-05 21:42:18.409 [info] GET /import
teslamate  | 2025-01-05 21:42:18.414 [info] Sent 200 in 4ms

Screenshots

No response

Additional data

No response

Type of installation

Docker

Version

1.32 and edge

[Xapier]

FYI: I have similar issues when migrating from TeslaFi as TeslaFi battery_level column is decimal instead of integer since 22 Nov 2024 - every monthly export file one would like to import is throwing errors.

[jaggernod]

The same happened to me.
Apparently, at that time, the battery level started to be reported as a double (number with a fraction) instead of an int (natural number).

What you can do before the import is to open the CSV file, find the battery_level column and convert all the values in that column to a natural number (all spreadsheet editors should have this option).
After saving the file it was imported properly for me.

[Xapier]

The same happened to me. Apparently, at that time, the battery level started to be reported as a double (number with a fraction) instead of an int (natural number).

What you can do before the import is to open the CSV file, find the battery_level column and convert all the values in that column to a natural number (all spreadsheet editors should have this option). After saving the file it was imported properly for me.

I tried this already but failed. Can you tell me which editor worked for you? Excel import destroys several column formats so I was struggling to fix the battery_level numbers (to integer) and import the data to teslamate after transforming it.