Issue with ADTs during cycling

[Dooruk]

I keep encountering an issue while cycling with a 2/3-deg setup we have where it ends up crashing after 3 weeks of simulation time. I tried multiple GEOS settings but after taking a closer look at the increments I figured it might be SOCA related. I'm at a bit of a loss here though.

For these particular plots, I'm using a 24-hr DA window, 2/3-deg tripolar grid, ocean-only assimilation (surface observations only), and using GDAS YAMLs.

I'm calling "unhealthy" increments a the ones behave as spikes in the J_o plots.

A healthy increment (2021-06-09):

An unhealthy increment (2021-06-10) (notice the difference in color scale):

I compared hofx_0 and hofx_1 vs. obs value plots to take a closer look

2021-06-09 (left, healthy) 2021-06-10(right, unhealthy)

So hofx_1 in the unhealthy plot shows analysis is having 2-3m difference and this behavior eventually leads to blow up.

One clear solution is perhaps activating an obs - analysis filter in the ADT YAMLs but I would like to ensure this is not a systemic issue first. It is just puzzling to me this doesn't happen during 1/4-deg runs but I notice some spikes there as well. I will investigate them closely next.

Our first instinct was some sort of a land mask issue but that doesn't seem to be the case, you can see the location of the issue.

I also suspect the correlation lengths might be the culprit here. I used StaticB BUMP for these results and I just used the same correlation lengths from the 1/4-deg setup, obviously had to create new BUMP files for the 2/3-deg grid.

However I also looked at the ADT ufo operator and I was wondering if this might have an impact on the analysis?

https://github.com/JCSDA-internal/ufo/blob/d7a8918003ca5da787c4e459f2928a87e6e50882/src/ufo/operators/marine/adt/ObsADT.cc#L77

P.S.: I see same outliers with SST observations as well but maybe the obs. count prevent the same issue from happening (272k for SST vs. ~9k with ADTs)

FYI @guillaumevernieres

[guillaumevernieres]

Thanks for sharing @Dooruk . We recently changed the adt obs yamls and the way we assimilate the rads retrieved adt. I'm not sure this is related to your problem though, but here's what changed:
1 - We're no longer blacklisting adt obs in the southern ocean, instead we compute the unbalanced term of ssh at every cycle.
2 - We only have one obs space for adt. This was done to improve the reference that we estimate in the ufo.

A large ssh increment translates into a large temperature and salinity increment. Did you look at those? I suspect something is wrong in your background error for temperature.

[travissluka]

A couple of related things.

1. Guillaume mentions background error. @Dooruk you're using the BkgErrGodas filter for generating parametric background error, right? It's next on my list to rewrite that filter into a proper saber block (move BkgErrGodas to a saber outer block #1004) and while I'm at it have it optionally save the calculated error so we can look at it.
2. The line in the ADT operator you note is a problem, but I don't think it's the problem here. An outlier ob would just shift the global bias slightly.

Since the adt hofx_0 looks fine, but hofx_1 is weird, I suspect it is not because of the ADT obs but rather one of the other surface obs, as Guillaume said you should look at what the T/S increments look like at that location

[Dooruk]

@guillaumevernieres

We're no longer blacklisting adt obs in the southern ocean, instead we compute the unbalanced term of ssh at every cycle.

Sorry, calculating the unbalanced part and incorporating it back is not clear to me. Is this something that can be done via the BalanceSOCA operator? Or by changing h/ssh in the increment file? you already responded to this, calculated offline, I remember now!

Combining ADTs is something I will try, at some point..

@travissluka
Yes, I use the BkgErrGodas. Sounds good on saving errors.

The line in the ADT operator you note is a problem, but I don't think it's the problem here. An outlier ob would just shift the global bias slightly.

Yeah, that makes sense.. I will look at increments closely and conduct more dirac tests while at it. Thanks for the responses, I will look into this issue again in the near future.