3363 decommission kibana elasticsearch

[raftmsohani]

Summary of Changes

Provide a brief summary of changes
Pull request closes #3363_

How to Test

NOTE: Tests are passing but have not tested deployment yet. Thought it makes sense to get an early review since this is such a big PR

1. Open http://localhost:3000/ and sign in.
2. Proceed with functional tests as described herein.
3. Test steps should be captured in the demo GIF(s) and/or screenshots below.

Demo GIF(s) and screenshots for testing procedure

Deliverables

More details on how deliverables herein are assessed included here.

Deliverable 1: Accepted Features

Checklist of ACs:

• [insert ACs here]
• lfrohlich and/or adpennington confirmed that ACs are met.

Deliverable 2: Tested Code

• Are all areas of code introduced in this PR meaningfully tested?
  • If this PR introduces backend code changes, are they meaningfully tested?
  • If this PR introduces frontend code changes, are they meaningfully tested?
• Are code coverage minimums met?
  • Frontend coverage: [insert coverage %] (see CodeCov Report comment in PR)
  • Backend coverage: [insert coverage %] (see CodeCov Report comment in PR)

Deliverable 3: Properly Styled Code

• Are backend code style checks passing on CircleCI?
• Are frontend code style checks passing on CircleCI?
• Are code maintainability principles being followed?

Deliverable 4: Accessible

• Does this PR complete the epic?
• Are links included to any other gov-approved PRs associated with epic?
• Does PR include documentation for Raft's a11y review?
• Did automated and manual testing with iamjolly and ttran-hub using Accessibility Insights reveal any errors introduced in this PR?

Deliverable 5: Deployed

• Was the code successfully deployed via automated CircleCI process to development on Cloud.gov?

Deliverable 6: Documented

• Does this PR provide background for why coding decisions were made?
• If this PR introduces backend code, is that code easy to understand and sufficiently documented, both inline and overall?
• If this PR introduces frontend code, is that code easy to understand and sufficiently documented, both inline and overall?
• If this PR introduces dependencies, are their licenses documented?
• Can reviewer explain and take ownership of these elements presented in this code review?

Deliverable 7: Secure

• Does the OWASP Scan pass on CircleCI?
• Do manual code review and manual testing detect any new security issues?
• If new issues detected, is investigation and/or remediation plan documented?

Deliverable 8: User Research

Research product(s) clearly articulate(s):

• the purpose of the research
• methods used to conduct the research
• who participated in the research
• what was tested and how
• impact of research on TDP
• (if applicable) final design mockups produced for TDP development

[codecov]

Codecov Report

Attention: Patch coverage is 93.75000% with 2 lines in your changes missing coverage. Please review.

Project coverage is 91.62%. Comparing base (d227065) to head (c67ed54).
Report is 1 commits behind head on develop.

Files with missing lines	Patch %	Lines
...h_indexes/management/commands/clean_and_reparse.py	0.00%	2 Missing ⚠️

Additional details and impacted files

@@             Coverage Diff             @@
##           develop    #3446      +/-   ##
===========================================
+ Coverage    91.24%   91.62%   +0.38%     
===========================================
  Files          303      300       -3     
  Lines         8734     8458     -276     
  Branches       650      636      -14     
===========================================
- Hits          7969     7750     -219     
+ Misses         645      591      -54     
+ Partials       120      117       -3     

Flag	Coverage Δ
dev-backend	91.52% <93.75%> (+0.41%)	⬆️
dev-frontend	92.37% <ø> (+0.14%)	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

Files with missing lines	Coverage Δ
tdrs-backend/tdpservice/parsers/aggregates.py	98.03% <100.00%> (ø)
...d/tdpservice/parsers/case_consistency_validator.py	96.65% <100.00%> (ø)
...rs-backend/tdpservice/parsers/duplicate_manager.py	94.11% <100.00%> (ø)
tdrs-backend/tdpservice/parsers/parse.py	85.44% <100.00%> (+2.74%)	⬆️
tdrs-backend/tdpservice/parsers/row_schema.py	96.58% <100.00%> (ø)
...s-backend/tdpservice/parsers/schema_defs/header.py	100.00% <ø> (ø)
...s-backend/tdpservice/parsers/schema_defs/ssp/m1.py	100.00% <ø> (ø)
...s-backend/tdpservice/parsers/schema_defs/ssp/m2.py	100.00% <ø> (ø)
...s-backend/tdpservice/parsers/schema_defs/ssp/m3.py	100.00% <ø> (ø)
...s-backend/tdpservice/parsers/schema_defs/ssp/m4.py	100.00% <ø> (ø)
... and 30 more

... and 1 file with indirect coverage changes

---

Continue to review full report in Codecov by Sentry.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 8955b3b...c67ed54. Read the comment docs.

[elipe17]

This is very confusing. I think the entire documents directory in the search_indexes app should be deleted. It doesn't make sense to have reference to documents since we no longer have Elastic. Why can't we just use the model directly like we were prior the elastic bulk create stuff? i.e. model=SSP_M1 where SSP_M1 is the django model in search_indexes.models.ssp.

[raftmsohani]

I agree, we can definitely do that now.

[elipe17]

If you implement the suggestion below from m1.py this line could be simply become: record = self.model().

[raftmsohani]

Tested deployment in RAFT, everything works as normal.

Need to decommission ES services in tf but this will be done once all the apps have decommissioned ES.