From f9d0a4acb4de078fc069bdd9d7818572e7b7de46 Mon Sep 17 00:00:00 2001 From: Marie Backman Date: Tue, 6 Aug 2024 14:36:16 -0400 Subject: [PATCH 1/7] docs like communication flow diagram --- docs/developer/design/overview.rst | 131 ++++++++++++++++++++++ docs/developer/index.rst | 1 + docs/developer/instruction/deployment.rst | 97 ++++++++++++++++ 3 files changed, 229 insertions(+) create mode 100644 docs/developer/design/overview.rst diff --git a/docs/developer/design/overview.rst b/docs/developer/design/overview.rst new file mode 100644 index 00000000..e408d51e --- /dev/null +++ b/docs/developer/design/overview.rst @@ -0,0 +1,131 @@ +Design Overview +=============== + +ActiveMQ Communication Flow +--------------------------- + +This sequence diagram describes the communication flow through the ActiveMQ message broker as a run +gets processed in WebMon. + +Note that the post-processing workflow is configurable in the database table ``Task`` - the +sequence presented here is representative for the majority of the instruments. + +.. mermaid:: + + sequenceDiagram + participant Translation Service + participant Workflow Manager + participant Autoreducer + participant ONCat + + + Translation Service->>Workflow Manager: POSTPROCESS.DATA_READY + Workflow Manager->>Autoreducer: CATALOG.ONCAT.DATA_READY + Autoreducer->>Workflow Manager: CATALOG.ONCAT.STARTED + Autoreducer->>ONCat: pyoncat + Autoreducer->>Workflow Manager: CATALOG.ONCAT.COMPLETE + Workflow Manager->>Autoreducer: REDUCTION.DATA_READY + Autoreducer->>Workflow Manager: REDUCTION.STARTED + Autoreducer->>Workflow Manager: REDUCTION.COMPLETE + Workflow Manager->>Autoreducer: REDUCTION_CATALOG.DATA_READY + Autoreducer->>Workflow Manager: REDUCTION_CATALOG.STARTED + Autoreducer->>ONCat: pyoncat + Autoreducer->>Workflow Manager: REDUCTION_CATALOG.COMPLETE + + +.. mermaid:: + + sequenceDiagram + participant Translation Service + participant Workflow Manager + participant Autoreducer + participant ONCat + + + Translation Service->>Workflow Manager: POSTPROCESS.DATA_READY + Workflow Manager->>Autoreducer: CATALOG.ONCAT.DATA_READY + Autoreducer->>Workflow Manager: REDUCTION.STARTED + Autoreducer->>Workflow Manager: REDUCTION.COMPLETE + + +.. list-table:: Reduction Plan - Main Fields + :widths: 20 20 20 20 20 + :header-rows: 1 + + * - Field + - Type + - Value Origin + - Additional validation + - Mandatory + * - Reduction Plan + - String + - + - filepath usage + - yes + * - Instrument + - String + - predefined choices from available instrument + - + - yes + * - IPTS + - Integer + - + - valid/existing filepath + - yes + * - Run Ranges + - Comma-separated numbers and number ranges + - + - valid/existing filepath + - yes + * - Wavelength + - Float + - default value from instrument configuration + - positive + - yes + * - Grouping + - String + - predefined choices from instrument configuration + - + - yes + + + +WebMon interaction with Live Data Server +---------------------------------------- + +WebMon has two modes of interaction with Live Data Server: publish (save) plots to the Live Data +Server database and display (fetch) plots from the database. + +Publish to Live Data Server +........................... + +The instrument-specific autoreduction script can optionally publish plots (in either JSON format +or HTML div) to Live Data Server. + +.. mermaid:: + + sequenceDiagram + participant WebMon + participant Autoreducer + participant Live Data Server + + WebMon->>Autoreducer: REDUCTION.DATA_READY + Autoreducer->>Live Data Server: publish_plot + Note over Live Data Server: Store plot in DB + +Fetch plot from Live Data Server +................................ + +Run overview pages (monitor.sns.gov/report///) will query the Live +Data Server for a plot for that instrument and run number and display it if available. + +.. mermaid:: + + sequenceDiagram + participant WebMon + participant Live Data Server + + WebMon->>Live Data Server: HTTP GET + loop Every 60 s + WebMon->>Live Data Server: HTTP GET + end diff --git a/docs/developer/index.rst b/docs/developer/index.rst index 73fae528..26f81a39 100644 --- a/docs/developer/index.rst +++ b/docs/developer/index.rst @@ -7,6 +7,7 @@ Developer Guide .. toctree:: :maxdepth: 1 + design/overview instruction/build instruction/test_fixture instruction/docker diff --git a/docs/developer/instruction/deployment.rst b/docs/developer/instruction/deployment.rst index 4070ad5a..244d46bc 100644 --- a/docs/developer/instruction/deployment.rst +++ b/docs/developer/instruction/deployment.rst @@ -117,3 +117,100 @@ You need to make sure the following variables match: * ``DATABASE_*`` in **web-monitor**, **workflow-mgr** and **testfixtures**, and **POSTGRES_*** in database * ``ICAT_USER`` and ``ICAT_PASS`` in **web-monitor**, **workflow-mgr**, **catalog** and **testfixtures**, and **amqbroker** (``icat_activemq.xml``) and **autoreducer** (``post_processing.conf``) * ``WORKFLOW_USER`` and ``WORKFLOW_PASS`` in **workflow-mgr** and in **amqbroker** (``icat_activemq.xml``) + +Production servers +------------------ + ++-----------------------------+--------------------------------+---------------------------------------------------+----------------------------------------------------------------------+ +| Hostname | Application | Source code | Deployment | ++=============================+================================+===================================================+======================================================================+ +| amqbroker.sns.gov | ActiveMQ broker | | Handled by Linux sysadmins | ++-----------------------------+--------------------------------+---------------------------------------------------+----------------------------------------------------------------------+ +| autoreducer[1-4].sns.gov | Post-processing agent | https://github.com/neutrons/post_processing_agent | Manual installation of RPM | ++-----------------------------+--------------------------------+---------------------------------------------------+----------------------------------------------------------------------+ +| scse-livedata-prod1.sns.gov | Live Data Server app | https://github.com/neutrons/live_data_server | https://code.ornl.gov/sns-hfir-scse/deployments/livedata-deploy | +| +--------------------------------+---------------------------------------------------+ | +| | Reverse proxy (nginx) | | | +| +--------------------------------+---------------------------------------------------+ | +| | PostgreSQL DB | | | ++-----------------------------+--------------------------------+---------------------------------------------------+----------------------------------------------------------------------+ +| scse-webmon-prod1.sns.gov | WebMon app | https://github.com/neutrons/data_workflow | https://code.ornl.gov/sns-hfir-scse/deployments/web-monitor-deploy | +| +--------------------------------+---------------------------------------------------+ | +| | Reverse proxy (nginx) | | | ++-----------------------------+--------------------------------+---------------------------------------------------+----------------------------------------------------------------------+ +| scse-wkfdb-prod1.sns.gov | PostgreSQL DB for Workflow mgr | | https://code.ornl.gov/sns-hfir-scse/deployments/workflow-db-deploy | ++-----------------------------+--------------------------------+---------------------------------------------------+----------------------------------------------------------------------+ +| scse-wkfmgr-prod1.sns.gov | Dasmon listener | https://github.com/neutrons/data_workflow | https://code.ornl.gov/sns-hfir-scse/deployments/workflow-mgr-deploy | +| +--------------------------------+---------------------------------------------------+ | +| | Workflow manager | https://github.com/neutrons/data_workflow | | ++-----------------------------+--------------------------------+---------------------------------------------------+----------------------------------------------------------------------+ + +.. list-table:: Reduction Plan - Main Fields + :widths: 20 20 20 20 20 + :header-rows: 1 + + * - Field + - Type + - Value Origin + - Additional validation + - Mandatory + * - Reduction Plan + - String + - + - filepath usage + - yes + * - Instrument + - String + - predefined choices from available instrument + - + - yes + * - IPTS + - Integer + - + - valid/existing filepath + - yes + * - Run Ranges + - Comma-separated numbers and number ranges + - + - valid/existing filepath + - yes + * - Wavelength + - Float + - default value from instrument configuration + - positive + - yes + * - Grouping + - String + - predefined choices from instrument configuration + - + - yes + * - UB Matrix + - String + - + - valid/existing filepath + - yes + * - Flux + - String + - + - valid/existing filepath + - no + * - Solid Angle + - String + - + - valid/existing filepath + - no + * - Mask + - String + - + - valid/existing filepath + - no + * - Background + - String + - + - valid/existing filepath + - no + * - Reduction Plan file + - String + - + - valid/non-existing filepath + - yes From 1f38a758ffabd51d645b1c76bc340d93e4da9ab0 Mon Sep 17 00:00:00 2001 From: Marie Backman Date: Tue, 20 Aug 2024 13:22:58 -0400 Subject: [PATCH 2/7] more documentation of communication flow --- conda_environment.yml | 1 + docs/conf.py | 1 + docs/developer/design/communication_flows.rst | 185 ++++++++++++++++++ docs/developer/design/overview.rst | 130 +----------- docs/developer/instruction/autoreduction.rst | 2 + 5 files changed, 192 insertions(+), 127 deletions(-) create mode 100644 docs/developer/design/communication_flows.rst diff --git a/conda_environment.yml b/conda_environment.yml index 790ca87f..7ef9e0d0 100644 --- a/conda_environment.yml +++ b/conda_environment.yml @@ -20,6 +20,7 @@ dependencies: - versioningit~=1.1 - pyoncat - sphinx_rtd_theme=1.2.* # readthedocs use this env file, and we need to install this theme here + - sphinxcontrib-mermaid - pip - pip: - django-auth-ldap==4.1.0 diff --git a/docs/conf.py b/docs/conf.py index 63d45f9f..deba9873 100644 --- a/docs/conf.py +++ b/docs/conf.py @@ -55,6 +55,7 @@ "sphinx.ext.ifconfig", "sphinx.ext.viewcode", "sphinx.ext.githubpages", + "sphinxcontrib.mermaid", ] # Add any paths that contain templates here, relative to this directory. diff --git a/docs/developer/design/communication_flows.rst b/docs/developer/design/communication_flows.rst new file mode 100644 index 00000000..afd044f7 --- /dev/null +++ b/docs/developer/design/communication_flows.rst @@ -0,0 +1,185 @@ +ActiveMQ Communication Flows +============================ + +Autoreducers +------------ + +Run post-processing +................... + +The sequence diagram below describes the communication flow as a run gets post-processed. +The post-processing workflow is triggered when the Translation Service has finished translating and +sends a message on the queue ``POSTPROCESS.DATA_READY`` specifying the instrument, IPTS, run number and +location of the Nexus file. + +The post-processing workflow for the instrument is configurable in the database table ``Task``. +The diagram shows the three post-processing steps that are available: autoreduction, cataloging of +raw data in `ONCat `_ and cataloging of reduced data in +`ONCat `_. + +.. + .. mermaid:: + + sequenceDiagram + participant Translation Service + participant Workflow Manager + participant Autoreducer + participant ONCat + + Translation Service->>Workflow Manager: POSTPROCESS.DATA_READY + opt Cataloging + Workflow Manager->>Autoreducer: CATALOG.ONCAT.DATA_READY + Autoreducer->>Workflow Manager: CATALOG.ONCAT.STARTED + Autoreducer->>ONCat: pyoncat + Autoreducer->>Workflow Manager: CATALOG.ONCAT.COMPLETE + end + opt Autoreduction + Workflow Manager->>Autoreducer: REDUCTION.DATA_READY + Autoreducer->>Workflow Manager: REDUCTION.STARTED + Note over Autoreducer: Execute autoreduction script + Autoreducer->>Workflow Manager: REDUCTION.COMPLETE + end + opt Reduced data cataloging + Workflow Manager->>Autoreducer: REDUCTION_CATALOG.DATA_READY + Autoreducer->>Workflow Manager: REDUCTION_CATALOG.STARTED + Autoreducer->>ONCat: pyoncat + Autoreducer->>Workflow Manager: REDUCTION_CATALOG.COMPLETE + end + +.. mermaid:: + + sequenceDiagram + participant Translation Service + participant Workflow Manager + participant Autoreducer + + Translation Service->>Workflow Manager: POSTPROCESS.DATA_READY + opt Cataloging + Workflow Manager->>Autoreducer: CATALOG.ONCAT.DATA_READY + Autoreducer->>Workflow Manager: CATALOG.ONCAT.STARTED + Note over Autoreducer: Ingest in ONCat + Autoreducer->>Workflow Manager: CATALOG.ONCAT.COMPLETE + end + opt Autoreduction + Workflow Manager->>Autoreducer: REDUCTION.DATA_READY + Autoreducer->>Workflow Manager: REDUCTION.STARTED + Note over Autoreducer: Execute autoreduction script + Autoreducer->>Workflow Manager: REDUCTION.COMPLETE + end + opt Reduced data cataloging + Workflow Manager->>Autoreducer: REDUCTION_CATALOG.DATA_READY + Autoreducer->>Workflow Manager: REDUCTION_CATALOG.STARTED + Note over Autoreducer: Ingest in ONCat + Autoreducer->>Workflow Manager: REDUCTION_CATALOG.COMPLETE + end + + +Configuring the autoreduction +............................. + +In addition to run post-processing, the autoreducers handle updating instrument reduction script +parameters for instruments that have implemented +:doc:`autoreduction parameter configuration<../instruction/autoreduction>` at +`monitor.sns.gov/reduction// `_. + +.. mermaid:: + + sequenceDiagram + actor Instrument Scientist + participant WebMon + participant Autoreducer + + Instrument Scientist->>WebMon: Submit form with parameter values + WebMon->>Autoreducer: REDUCTION.CREATE_SCRIPT + Note over Autoreducer: Update parameter values in
instrument reduction script + +DASMON +------ +DASMON, from Data Acquisition System (DAS) Monitor, provides process variable (PV) updates from the +beamlines to WebMon. Due to the high volume of PV updates, DASMON writes directly to the database. + +.. mermaid:: + + sequenceDiagram + participant DASMON + participant Workflow DB + participant Dasmon listener + DASMON->>Workflow DB: PV update + Workflow DB->>Dasmon listener: PV update + DASMON->>Dasmon listener: Heartbeats + +Dasmon listener +--------------- + +Streaming Message Service (SMS) +............................... + +.. mermaid:: + + sequenceDiagram + participant SMS + participant Dasmon listener + participant Workflow DB + SMS->>Dasmon listener: Run started + Dasmon listener->>Workflow DB: Create new run + SMS->>Dasmon listener: Run stopped + SMS->>Dasmon listener: Translation succeeded + +Heartbeats from services +........................ + +Dasmon listener subscribes to heartbeats from the other services. There is a mechanism for alerting +admins by email when a service has missed heartbeats (needs to be verified that this still works). + +.. mermaid:: + + sequenceDiagram + participant Other services + participant Dasmon listener + participant Workflow DB + actor Subscribed users + Other services->>Dasmon listener: Heartbeat + Dasmon listener->>Workflow DB: Status update + opt There are missed heartbeats + Dasmon listener->>Subscribed users: Email + end + +WebMon interaction with Live Data Server +---------------------------------------- + +WebMon has two modes of interaction with Live Data Server: publish (save) plots to the Live Data +Server database and display (fetch) plots from the database. + +Publish to Live Data Server +........................... + +The instrument-specific autoreduction script can optionally publish plots (in either JSON format +or HTML div) to Live Data Server. + +.. mermaid:: + + sequenceDiagram + participant WebMon + participant Autoreducer + participant Live Data Server + + WebMon->>Autoreducer: REDUCTION.DATA_READY + Autoreducer->>Live Data Server: publish_plot + Note over Live Data Server: Store plot in DB + +Display plot from Live Data Server +................................ + +Run overview pages (monitor.sns.gov/report///) will query the Live +Data Server for a plot for that instrument and run number and display it if available. + +.. mermaid:: + + sequenceDiagram + participant WebMon + participant Live Data Server + + WebMon->>Live Data Server: HTTP GET + loop Every 60 s + WebMon->>Live Data Server: HTTP GET + end diff --git a/docs/developer/design/overview.rst b/docs/developer/design/overview.rst index e408d51e..f175fc1f 100644 --- a/docs/developer/design/overview.rst +++ b/docs/developer/design/overview.rst @@ -1,131 +1,7 @@ Design Overview =============== -ActiveMQ Communication Flow ---------------------------- +.. toctree:: + :maxdepth: 1 -This sequence diagram describes the communication flow through the ActiveMQ message broker as a run -gets processed in WebMon. - -Note that the post-processing workflow is configurable in the database table ``Task`` - the -sequence presented here is representative for the majority of the instruments. - -.. mermaid:: - - sequenceDiagram - participant Translation Service - participant Workflow Manager - participant Autoreducer - participant ONCat - - - Translation Service->>Workflow Manager: POSTPROCESS.DATA_READY - Workflow Manager->>Autoreducer: CATALOG.ONCAT.DATA_READY - Autoreducer->>Workflow Manager: CATALOG.ONCAT.STARTED - Autoreducer->>ONCat: pyoncat - Autoreducer->>Workflow Manager: CATALOG.ONCAT.COMPLETE - Workflow Manager->>Autoreducer: REDUCTION.DATA_READY - Autoreducer->>Workflow Manager: REDUCTION.STARTED - Autoreducer->>Workflow Manager: REDUCTION.COMPLETE - Workflow Manager->>Autoreducer: REDUCTION_CATALOG.DATA_READY - Autoreducer->>Workflow Manager: REDUCTION_CATALOG.STARTED - Autoreducer->>ONCat: pyoncat - Autoreducer->>Workflow Manager: REDUCTION_CATALOG.COMPLETE - - -.. mermaid:: - - sequenceDiagram - participant Translation Service - participant Workflow Manager - participant Autoreducer - participant ONCat - - - Translation Service->>Workflow Manager: POSTPROCESS.DATA_READY - Workflow Manager->>Autoreducer: CATALOG.ONCAT.DATA_READY - Autoreducer->>Workflow Manager: REDUCTION.STARTED - Autoreducer->>Workflow Manager: REDUCTION.COMPLETE - - -.. list-table:: Reduction Plan - Main Fields - :widths: 20 20 20 20 20 - :header-rows: 1 - - * - Field - - Type - - Value Origin - - Additional validation - - Mandatory - * - Reduction Plan - - String - - - - filepath usage - - yes - * - Instrument - - String - - predefined choices from available instrument - - - - yes - * - IPTS - - Integer - - - - valid/existing filepath - - yes - * - Run Ranges - - Comma-separated numbers and number ranges - - - - valid/existing filepath - - yes - * - Wavelength - - Float - - default value from instrument configuration - - positive - - yes - * - Grouping - - String - - predefined choices from instrument configuration - - - - yes - - - -WebMon interaction with Live Data Server ----------------------------------------- - -WebMon has two modes of interaction with Live Data Server: publish (save) plots to the Live Data -Server database and display (fetch) plots from the database. - -Publish to Live Data Server -........................... - -The instrument-specific autoreduction script can optionally publish plots (in either JSON format -or HTML div) to Live Data Server. - -.. mermaid:: - - sequenceDiagram - participant WebMon - participant Autoreducer - participant Live Data Server - - WebMon->>Autoreducer: REDUCTION.DATA_READY - Autoreducer->>Live Data Server: publish_plot - Note over Live Data Server: Store plot in DB - -Fetch plot from Live Data Server -................................ - -Run overview pages (monitor.sns.gov/report///) will query the Live -Data Server for a plot for that instrument and run number and display it if available. - -.. mermaid:: - - sequenceDiagram - participant WebMon - participant Live Data Server - - WebMon->>Live Data Server: HTTP GET - loop Every 60 s - WebMon->>Live Data Server: HTTP GET - end + communication_flows diff --git a/docs/developer/instruction/autoreduction.rst b/docs/developer/instruction/autoreduction.rst index ff0e874c..2ed61ecc 100644 --- a/docs/developer/instruction/autoreduction.rst +++ b/docs/developer/instruction/autoreduction.rst @@ -1,3 +1,5 @@ +.. autoreduction-parameter-configuration + How to Modify an Instrument Autoreduction Configuration Page ============================================================ From c41defb7bc7a444bfa8a75999e0b32b09c24279c Mon Sep 17 00:00:00 2001 From: Marie Backman Date: Tue, 20 Aug 2024 13:29:13 -0400 Subject: [PATCH 3/7] remove garbage --- docs/developer/instruction/deployment.rst | 97 ----------------------- 1 file changed, 97 deletions(-) diff --git a/docs/developer/instruction/deployment.rst b/docs/developer/instruction/deployment.rst index 244d46bc..4070ad5a 100644 --- a/docs/developer/instruction/deployment.rst +++ b/docs/developer/instruction/deployment.rst @@ -117,100 +117,3 @@ You need to make sure the following variables match: * ``DATABASE_*`` in **web-monitor**, **workflow-mgr** and **testfixtures**, and **POSTGRES_*** in database * ``ICAT_USER`` and ``ICAT_PASS`` in **web-monitor**, **workflow-mgr**, **catalog** and **testfixtures**, and **amqbroker** (``icat_activemq.xml``) and **autoreducer** (``post_processing.conf``) * ``WORKFLOW_USER`` and ``WORKFLOW_PASS`` in **workflow-mgr** and in **amqbroker** (``icat_activemq.xml``) - -Production servers ------------------- - -+-----------------------------+--------------------------------+---------------------------------------------------+----------------------------------------------------------------------+ -| Hostname | Application | Source code | Deployment | -+=============================+================================+===================================================+======================================================================+ -| amqbroker.sns.gov | ActiveMQ broker | | Handled by Linux sysadmins | -+-----------------------------+--------------------------------+---------------------------------------------------+----------------------------------------------------------------------+ -| autoreducer[1-4].sns.gov | Post-processing agent | https://github.com/neutrons/post_processing_agent | Manual installation of RPM | -+-----------------------------+--------------------------------+---------------------------------------------------+----------------------------------------------------------------------+ -| scse-livedata-prod1.sns.gov | Live Data Server app | https://github.com/neutrons/live_data_server | https://code.ornl.gov/sns-hfir-scse/deployments/livedata-deploy | -| +--------------------------------+---------------------------------------------------+ | -| | Reverse proxy (nginx) | | | -| +--------------------------------+---------------------------------------------------+ | -| | PostgreSQL DB | | | -+-----------------------------+--------------------------------+---------------------------------------------------+----------------------------------------------------------------------+ -| scse-webmon-prod1.sns.gov | WebMon app | https://github.com/neutrons/data_workflow | https://code.ornl.gov/sns-hfir-scse/deployments/web-monitor-deploy | -| +--------------------------------+---------------------------------------------------+ | -| | Reverse proxy (nginx) | | | -+-----------------------------+--------------------------------+---------------------------------------------------+----------------------------------------------------------------------+ -| scse-wkfdb-prod1.sns.gov | PostgreSQL DB for Workflow mgr | | https://code.ornl.gov/sns-hfir-scse/deployments/workflow-db-deploy | -+-----------------------------+--------------------------------+---------------------------------------------------+----------------------------------------------------------------------+ -| scse-wkfmgr-prod1.sns.gov | Dasmon listener | https://github.com/neutrons/data_workflow | https://code.ornl.gov/sns-hfir-scse/deployments/workflow-mgr-deploy | -| +--------------------------------+---------------------------------------------------+ | -| | Workflow manager | https://github.com/neutrons/data_workflow | | -+-----------------------------+--------------------------------+---------------------------------------------------+----------------------------------------------------------------------+ - -.. list-table:: Reduction Plan - Main Fields - :widths: 20 20 20 20 20 - :header-rows: 1 - - * - Field - - Type - - Value Origin - - Additional validation - - Mandatory - * - Reduction Plan - - String - - - - filepath usage - - yes - * - Instrument - - String - - predefined choices from available instrument - - - - yes - * - IPTS - - Integer - - - - valid/existing filepath - - yes - * - Run Ranges - - Comma-separated numbers and number ranges - - - - valid/existing filepath - - yes - * - Wavelength - - Float - - default value from instrument configuration - - positive - - yes - * - Grouping - - String - - predefined choices from instrument configuration - - - - yes - * - UB Matrix - - String - - - - valid/existing filepath - - yes - * - Flux - - String - - - - valid/existing filepath - - no - * - Solid Angle - - String - - - - valid/existing filepath - - no - * - Mask - - String - - - - valid/existing filepath - - no - * - Background - - String - - - - valid/existing filepath - - no - * - Reduction Plan file - - String - - - - valid/non-existing filepath - - yes From eae3bd2e466df7fc28f36e91e30998e20b92da50 Mon Sep 17 00:00:00 2001 From: Marie Backman Date: Thu, 5 Sep 2024 17:13:28 -0400 Subject: [PATCH 4/7] add high-level architecture diagram --- docs/developer/design/communication_flows.rst | 46 +++++-- docs/developer/design/overview.rst | 112 ++++++++++++++++++ 2 files changed, 149 insertions(+), 9 deletions(-) diff --git a/docs/developer/design/communication_flows.rst b/docs/developer/design/communication_flows.rst index afd044f7..28f077e6 100644 --- a/docs/developer/design/communication_flows.rst +++ b/docs/developer/design/communication_flows.rst @@ -1,6 +1,10 @@ +.. _communication_flows: + ActiveMQ Communication Flows ============================ +.. contents:: :local: + Autoreducers ------------ @@ -95,7 +99,7 @@ parameters for instruments that have implemented DASMON ------ -DASMON, from Data Acquisition System (DAS) Monitor, provides process variable (PV) updates from the +DASMON, from Data Acquisition (DAQ) System Monitor, provides process variable (PV) updates from the beamlines to WebMon. Due to the high volume of PV updates, DASMON writes directly to the database. .. mermaid:: @@ -105,13 +109,12 @@ beamlines to WebMon. Due to the high volume of PV updates, DASMON writes directl participant Workflow DB participant Dasmon listener DASMON->>Workflow DB: PV update - Workflow DB->>Dasmon listener: PV update DASMON->>Dasmon listener: Heartbeats Dasmon listener --------------- -Streaming Message Service (SMS) +Stream Management Service (SMS) ............................... .. mermaid:: @@ -138,12 +141,35 @@ admins by email when a service has missed heartbeats (needs to be verified that participant Dasmon listener participant Workflow DB actor Subscribed users - Other services->>Dasmon listener: Heartbeat - Dasmon listener->>Workflow DB: Status update - opt There are missed heartbeats + loop Every N s + Other services->>Dasmon listener: Heartbeat + Dasmon listener->>Workflow DB: Status update + end + opt Service has 3 missed heartbeats Dasmon listener->>Subscribed users: Email end + +.. mermaid:: + + flowchart LR + SMS["SMS (per beamline)"] + PVSD["PVSD (per beamline)"] + DASMON["DASMON (per beamline)"] + STC + Autoreducers + DasmonListener + WorkflowDB[(DB)] + SMS-->|heartbeat|DasmonListener + PVSD-->|heartbeat|DasmonListener + DASMON-->|heartbeat|DasmonListener + STC-->|heartbeat|DasmonListener + Autoreducers-->|heartbeat|DasmonListener + WorkflowManager-->|heartbeat|DasmonListener + DasmonListener-->|heartbeat|DasmonListener + DasmonListener-->WorkflowDB + DasmonListener-.->|if missed 3 heartbeats|InstrumentScientist + WebMon interaction with Live Data Server ---------------------------------------- @@ -164,13 +190,15 @@ or HTML div) to Live Data Server. participant Live Data Server WebMon->>Autoreducer: REDUCTION.DATA_READY - Autoreducer->>Live Data Server: publish_plot - Note over Live Data Server: Store plot in DB + opt Publish plot + Autoreducer->>Live Data Server: publish_plot + Note over Live Data Server: Store plot in DB + end Display plot from Live Data Server ................................ -Run overview pages (monitor.sns.gov/report///) will query the Live +Run overview pages (``monitor.sns.gov/report///``) will query the Live Data Server for a plot for that instrument and run number and display it if available. .. mermaid:: diff --git a/docs/developer/design/overview.rst b/docs/developer/design/overview.rst index f175fc1f..695984ac 100644 --- a/docs/developer/design/overview.rst +++ b/docs/developer/design/overview.rst @@ -5,3 +5,115 @@ Design Overview :maxdepth: 1 communication_flows + + +High-level architecture +----------------------- + +The diagram below describes the high-level architecture of WebMon. The gray box labeled "DAS" are +services owned by the Data Acquisition System team that feed information to WebMon but are not part +of WebMon. The services communicate mainly through an ActiveMQ message broker. The communication +flow is described in :ref:`communication_flows`. + +.. mermaid:: + + flowchart LR + FileArchive[("`SNS/HFIR + File archive`")] + subgraph DAS + DASMON + TranslationService["`Streaming + Translation + Client + (STC)`"] + SMS["`Stream + Management + Service + (SMS)`"] + PVSD + end + subgraph "Workflow Manager" + WorkflowManager[Workflow Manager] + DasmonListener[Dasmon listener] + Database[(DB)] + end + subgraph Autoreduction + Autoreducers-->|cataloging|ONCat + LiveDataServer<-->LiveDataDB[(DB)] + LiveReduce + end + subgraph Frontend + WebMon["`WebMon + monitor.sns.gov`"] + end + SMS-->|live data stream|LiveReduce + TranslationService-->|trigger reduction|WorkflowManager + DASMON-->|status|DasmonListener + DASMON-->|PV updates|Database + WorkflowManager<-->|instructions|Autoreducers + Autoreducers-->|publish plots|LiveDataServer + Database-->WebMon + ONCat-->|run metadata|WebMon + LiveDataServer-->|plots|WebMon + LiveReduce-->|publish plots|LiveDataServer + DasmonListener-->Database + WorkflowManager-->Database + TranslationService-->|NeXus file|FileArchive + FileArchive<-->Autoreducers + style DAS fill:#D3D3D3, stroke-dasharray: 5 5 + + +High-level architecture +----------------------- + +The diagram below describes the high-level architecture of WebMon. The gray box labeled "DAS" are +services owned by the Data Acquisition System team that feed information to WebMon, e.g. to trigger +autoreduction when a run is finished. The services mainly communicate through an ActiveMQ message +broker. The communication flow is described in :ref:`communication_flows`. + +.. mermaid:: + + flowchart LR + FileArchive[("`SNS/HFIR + File archive`")] + subgraph DAS + DASMON + TranslationService["`Streaming + Translation + Client + (STC)`"] + SMS["`Stream + Management + Service + (SMS)`"] + PVSD + end + subgraph "Workflow Manager" + WorkflowManager[Workflow Manager] + DasmonListener[Dasmon listener] + Database[(DB)] + end + subgraph Autoreduction + Autoreducers-->|REST API|ONCat + LiveDataServer<-->LiveDataDB[(DB)] + LiveReduce + end + subgraph Frontend + WebMon["`WebMon + monitor.sns.gov`"] + end + SMS-->LiveReduce + TranslationService-.->|ActiveMQ|WorkflowManager + DASMON-.->|ActiveMQ|DasmonListener + DASMON-->Database + WorkflowManager<-.->|ActiveMQ|Autoreducers + Autoreducers-->|REST API|LiveDataServer + Database-->WebMon + ONCat-->|REST API|WebMon + LiveDataServer-->|REST API|WebMon + LiveReduce-->|REST API|LiveDataServer + DasmonListener-->Database + WorkflowManager-->Database + TranslationService-->FileArchive + FileArchive<-->Autoreducers + style DAS fill:#D3D3D3, stroke-dasharray: 5 5 From 3ce28fbd3f4bcec072ce9ad1eb3624674226d714 Mon Sep 17 00:00:00 2001 From: Marie Backman Date: Fri, 6 Sep 2024 11:53:17 -0400 Subject: [PATCH 5/7] address comments --- docs/developer/design/communication_flows.rst | 133 ++++++++++------- docs/developer/design/overview.rst | 141 ++++++++---------- 2 files changed, 138 insertions(+), 136 deletions(-) diff --git a/docs/developer/design/communication_flows.rst b/docs/developer/design/communication_flows.rst index 28f077e6..d29b43b8 100644 --- a/docs/developer/design/communication_flows.rst +++ b/docs/developer/design/communication_flows.rst @@ -12,15 +12,38 @@ Run post-processing ................... The sequence diagram below describes the communication flow as a run gets post-processed. -The post-processing workflow is triggered when the Translation Service has finished translating and -sends a message on the queue ``POSTPROCESS.DATA_READY`` specifying the instrument, IPTS, run number and -location of the Nexus file. +The post-processing workflow is triggered when the Translation Service has finished translating the +data stream to NeXus and sends a message on the queue ``POSTPROCESS.DATA_READY`` specifying the +instrument, IPTS, run number and location of the NeXus file. -The post-processing workflow for the instrument is configurable in the database table ``Task``. +The post-processing workflow for the instrument is configurable in the database table +``report_task``. The diagram shows the three post-processing steps that are available: autoreduction, cataloging of raw data in `ONCat `_ and cataloging of reduced data in `ONCat `_. +.. mermaid:: + + sequenceDiagram + participant Translation Service + participant Workflow Manager + participant Autoreducer + participant ONCat + + Translation Service->>Workflow Manager: POSTPROCESS.DATA_READY + Workflow Manager->>Autoreducer: CATALOG.ONCAT.DATA_READY + Autoreducer->>Workflow Manager: CATALOG.ONCAT.STARTED + Autoreducer->>ONCat: pyoncat + Autoreducer->>Workflow Manager: CATALOG.ONCAT.COMPLETE + Workflow Manager->>Autoreducer: REDUCTION.DATA_READY + Autoreducer->>Workflow Manager: REDUCTION.STARTED + Autoreducer->>Autoreducer: run reduction + Autoreducer->>Workflow Manager: REDUCTION.COMPLETE + Workflow Manager->>Autoreducer: REDUCTION_CATALOG.DATA_READY + Autoreducer->>Workflow Manager: REDUCTION_CATALOG.STARTED + Autoreducer->>ONCat: pyoncat + Autoreducer->>Workflow Manager: REDUCTION_CATALOG.COMPLETE + .. .. mermaid:: @@ -28,13 +51,12 @@ raw data in `ONCat `_ and cataloging of reduced data in participant Translation Service participant Workflow Manager participant Autoreducer - participant ONCat Translation Service->>Workflow Manager: POSTPROCESS.DATA_READY opt Cataloging Workflow Manager->>Autoreducer: CATALOG.ONCAT.DATA_READY Autoreducer->>Workflow Manager: CATALOG.ONCAT.STARTED - Autoreducer->>ONCat: pyoncat + Note over Autoreducer: Ingest in ONCat Autoreducer->>Workflow Manager: CATALOG.ONCAT.COMPLETE end opt Autoreduction @@ -46,37 +68,10 @@ raw data in `ONCat `_ and cataloging of reduced data in opt Reduced data cataloging Workflow Manager->>Autoreducer: REDUCTION_CATALOG.DATA_READY Autoreducer->>Workflow Manager: REDUCTION_CATALOG.STARTED - Autoreducer->>ONCat: pyoncat + Note over Autoreducer: Ingest in ONCat Autoreducer->>Workflow Manager: REDUCTION_CATALOG.COMPLETE end -.. mermaid:: - - sequenceDiagram - participant Translation Service - participant Workflow Manager - participant Autoreducer - - Translation Service->>Workflow Manager: POSTPROCESS.DATA_READY - opt Cataloging - Workflow Manager->>Autoreducer: CATALOG.ONCAT.DATA_READY - Autoreducer->>Workflow Manager: CATALOG.ONCAT.STARTED - Note over Autoreducer: Ingest in ONCat - Autoreducer->>Workflow Manager: CATALOG.ONCAT.COMPLETE - end - opt Autoreduction - Workflow Manager->>Autoreducer: REDUCTION.DATA_READY - Autoreducer->>Workflow Manager: REDUCTION.STARTED - Note over Autoreducer: Execute autoreduction script - Autoreducer->>Workflow Manager: REDUCTION.COMPLETE - end - opt Reduced data cataloging - Workflow Manager->>Autoreducer: REDUCTION_CATALOG.DATA_READY - Autoreducer->>Workflow Manager: REDUCTION_CATALOG.STARTED - Note over Autoreducer: Ingest in ONCat - Autoreducer->>Workflow Manager: REDUCTION_CATALOG.COMPLETE - end - Configuring the autoreduction ............................. @@ -92,24 +87,28 @@ parameters for instruments that have implemented actor Instrument Scientist participant WebMon participant Autoreducer + participant HFIR/SNS File archive Instrument Scientist->>WebMon: Submit form with parameter values WebMon->>Autoreducer: REDUCTION.CREATE_SCRIPT - Note over Autoreducer: Update parameter values in
instrument reduction script + Autoreducer->>HFIR/SNS File archive: Update instrument reduction script DASMON ------ -DASMON, from Data Acquisition (DAQ) System Monitor, provides process variable (PV) updates from the -beamlines to WebMon. Due to the high volume of PV updates, DASMON writes directly to the database. +DASMON, from Data Acquisition (DAQ) System Monitor, provides instrument status and process variable +(PV) updates from the beamlines to WebMon. DASMON connects to the WebMon message broker to pass +status information, for example the current run number and count rate, to Dasmon listener. Due to +the high volume of PV updates, DASMON writes PV:s directly to the PostgreSQL database. .. mermaid:: sequenceDiagram participant DASMON - participant Workflow DB participant Dasmon listener + participant Workflow DB DASMON->>Workflow DB: PV update - DASMON->>Dasmon listener: Heartbeats + DASMON->>Dasmon listener: Instrument status + DASMON->>Workflow DB: Instrument status Dasmon listener --------------- @@ -134,20 +133,21 @@ Heartbeats from services Dasmon listener subscribes to heartbeats from the other services. There is a mechanism for alerting admins by email when a service has missed heartbeats (needs to be verified that this still works). -.. mermaid:: +.. + .. mermaid:: - sequenceDiagram - participant Other services - participant Dasmon listener - participant Workflow DB - actor Subscribed users - loop Every N s - Other services->>Dasmon listener: Heartbeat - Dasmon listener->>Workflow DB: Status update - end - opt Service has 3 missed heartbeats - Dasmon listener->>Subscribed users: Email - end + sequenceDiagram + participant Other services + participant Dasmon listener + participant Workflow DB + actor Subscribed users + loop Every N s + Other services->>Dasmon listener: Heartbeat + Dasmon listener->>Workflow DB: Status update + end + opt Service has 3 missed heartbeats + Dasmon listener->>Subscribed users: Email + end .. mermaid:: @@ -170,14 +170,14 @@ admins by email when a service has missed heartbeats (needs to be verified that DasmonListener-->WorkflowDB DasmonListener-.->|if missed 3 heartbeats|InstrumentScientist -WebMon interaction with Live Data Server +Live Data Server ---------------------------------------- WebMon has two modes of interaction with Live Data Server: publish (save) plots to the Live Data Server database and display (fetch) plots from the database. -Publish to Live Data Server -........................... +Publish to Live Data Server from autoreduction script +..................................................... The instrument-specific autoreduction script can optionally publish plots (in either JSON format or HTML div) to Live Data Server. @@ -192,9 +192,30 @@ or HTML div) to Live Data Server. WebMon->>Autoreducer: REDUCTION.DATA_READY opt Publish plot Autoreducer->>Live Data Server: publish_plot - Note over Live Data Server: Store plot in DB end +Publish to Live Data Server from live data stream +................................................. + +Livereduce (https://github.com/mantidproject/livereduce/) allows scientists to reduce +data from an ongoing experiment, i.e. before translation to NeXus, by connecting to the live data +stream from the Stream Management Service (SMS). The instrument-specific processing +script can make the results available in WebMon by publishing plots to Live Data Server. + +.. mermaid:: + + sequenceDiagram + participant SMS + participant Livereduce + participant Live Data Server + + SMS->>Livereduce: data stream + loop Every N minutes + Livereduce->>Livereduce: run processing script + Livereduce->>Live Data Server: publish plot + end + + Display plot from Live Data Server ................................ diff --git a/docs/developer/design/overview.rst b/docs/developer/design/overview.rst index 695984ac..b5e1df00 100644 --- a/docs/developer/design/overview.rst +++ b/docs/developer/design/overview.rst @@ -10,10 +10,12 @@ Design Overview High-level architecture ----------------------- -The diagram below describes the high-level architecture of WebMon. The gray box labeled "DAS" are -services owned by the Data Acquisition System team that feed information to WebMon but are not part -of WebMon. The services communicate mainly through an ActiveMQ message broker. The communication -flow is described in :ref:`communication_flows`. +The diagram below describes the high-level architecture of WebMon, including some external resources +that WebMon interacts with. The gray box labeled "DAS" are services owned by the Data Acquisition +System team that feed information to WebMon. `ONCat `_ is the experiment +data catalog, which the autoreducers catalog runs to and the frontend fetches run metadata from. +The autoreducers access instrument-specific reduction scripts and experiment data files on the +HFIR/SNS file archive. The autoreducers also write reduced data files and logs to the file archive. .. mermaid:: @@ -30,90 +32,69 @@ flow is described in :ref:`communication_flows`. Management Service (SMS)`"] - PVSD end - subgraph "Workflow Manager" - WorkflowManager[Workflow Manager] - DasmonListener[Dasmon listener] - Database[(DB)] - end - subgraph Autoreduction - Autoreducers-->|cataloging|ONCat - LiveDataServer<-->LiveDataDB[(DB)] - LiveReduce - end - subgraph Frontend - WebMon["`WebMon - monitor.sns.gov`"] - end - SMS-->|live data stream|LiveReduce - TranslationService-->|trigger reduction|WorkflowManager - DASMON-->|status|DasmonListener - DASMON-->|PV updates|Database - WorkflowManager<-->|instructions|Autoreducers - Autoreducers-->|publish plots|LiveDataServer + WorkflowManager[Workflow Manager] + DasmonListener[Dasmon listener] + Database[(Workflow DB)] + Autoreducers-->ONCat + LiveDataServer-->WebMon + LiveDataServer<-->LiveDataDB[(LiveData DB)] + LiveReduce + WebMon["`WebMon + frontend`"] + SMS-->LiveReduce + TranslationService-->WorkflowManager + DASMON-->DasmonListener + DASMON-->Database + WorkflowManager-->Database + WorkflowManager<-->Autoreducers + Autoreducers-->LiveDataServer Database-->WebMon - ONCat-->|run metadata|WebMon - LiveDataServer-->|plots|WebMon - LiveReduce-->|publish plots|LiveDataServer + ONCat-->WebMon + LiveReduce-->LiveDataServer DasmonListener-->Database - WorkflowManager-->Database - TranslationService-->|NeXus file|FileArchive + TranslationService-->FileArchive FileArchive<-->Autoreducers style DAS fill:#D3D3D3, stroke-dasharray: 5 5 + classDef webMonStyle fill:#FFFFE0 + class WorkflowManager,DasmonListener,Database,Autoreducers,LiveDataServer,LiveReduce,WebMon,LiveDataDB webMonStyle + subgraph Legend + direction LR + External["External resource"] + Internal["Internal resource"] + External ~~~ Internal + end + LiveReduce ~~~ External + style Legend fill:#FFFFFF,stroke:#000000 + class Internal webMonStyle -High-level architecture ------------------------ -The diagram below describes the high-level architecture of WebMon. The gray box labeled "DAS" are -services owned by the Data Acquisition System team that feed information to WebMon, e.g. to trigger -autoreduction when a run is finished. The services mainly communicate through an ActiveMQ message -broker. The communication flow is described in :ref:`communication_flows`. +Message broker +-------------- + +WebMon uses an `ActiveMQ `_ message broker for communication between +services. The message broker also serves as a load balancer by distributing post-processing jobs to +the available autoreducers in a round-robin fashion. + +Service communication flows are described in :ref:`communication_flows`. .. mermaid:: - flowchart LR - FileArchive[("`SNS/HFIR - File archive`")] - subgraph DAS - DASMON - TranslationService["`Streaming - Translation - Client - (STC)`"] - SMS["`Stream - Management - Service - (SMS)`"] - PVSD - end - subgraph "Workflow Manager" - WorkflowManager[Workflow Manager] - DasmonListener[Dasmon listener] - Database[(DB)] - end - subgraph Autoreduction - Autoreducers-->|REST API|ONCat - LiveDataServer<-->LiveDataDB[(DB)] - LiveReduce - end - subgraph Frontend - WebMon["`WebMon - monitor.sns.gov`"] - end - SMS-->LiveReduce - TranslationService-.->|ActiveMQ|WorkflowManager - DASMON-.->|ActiveMQ|DasmonListener - DASMON-->Database - WorkflowManager<-.->|ActiveMQ|Autoreducers - Autoreducers-->|REST API|LiveDataServer - Database-->WebMon - ONCat-->|REST API|WebMon - LiveDataServer-->|REST API|WebMon - LiveReduce-->|REST API|LiveDataServer - DasmonListener-->Database - WorkflowManager-->Database - TranslationService-->FileArchive - FileArchive<-->Autoreducers - style DAS fill:#D3D3D3, stroke-dasharray: 5 5 + flowchart TB + TranslationService["`Streaming + Translation + Client + (STC)`"] + SMS["`Stream + Management + Service + (SMS)`"] + Broker[ActiveMQ broker] + Broker<-->Autoreducers + Broker<-->WorkflowManager[Workflow Manager] + Broker<-->DasmonListener[Dasmon listener] + Broker<-->DASMON + Broker<-->PVSD + Broker<-->TranslationService + Broker<-->SMS From b6aa20ce22bbab8d0e459124f7c6fb5cec9d2e1c Mon Sep 17 00:00:00 2001 From: Marie Backman Date: Fri, 13 Sep 2024 09:59:23 -0400 Subject: [PATCH 6/7] reorganize communication flows by function --- .../communication_flows.rst | 261 ++++++++---------- docs/developer/architecture/index.rst | 9 + .../{design => architecture}/overview.rst | 27 +- docs/developer/index.rst | 13 +- 4 files changed, 149 insertions(+), 161 deletions(-) rename docs/developer/{design => architecture}/communication_flows.rst (62%) create mode 100644 docs/developer/architecture/index.rst rename docs/developer/{design => architecture}/overview.rst (75%) diff --git a/docs/developer/design/communication_flows.rst b/docs/developer/architecture/communication_flows.rst similarity index 62% rename from docs/developer/design/communication_flows.rst rename to docs/developer/architecture/communication_flows.rst index d29b43b8..dbacc79c 100644 --- a/docs/developer/design/communication_flows.rst +++ b/docs/developer/architecture/communication_flows.rst @@ -1,20 +1,67 @@ .. _communication_flows: -ActiveMQ Communication Flows -============================ +Communication Flows +=================== + +This section presents communication sequences organized by WebMon functionality. .. contents:: :local: -Autoreducers ------------- +Experiment monitoring +--------------------- + +Instrument status and PV updates +................................ + +DASMON, from Data Acquisition (DAQ) System Monitor, provides instrument status and process variable +(PV) updates from the beamlines to WebMon. DASMON connects to the WebMon message broker to pass +status information, for example the current run number and count rate, to Dasmon listener. Due to +the high volume of PV updates, DASMON writes PV:s directly to the PostgreSQL database. + +.. mermaid:: -Run post-processing -................... + sequenceDiagram + participant DASMON + participant Dasmon listener + participant Workflow DB + par + DASMON->>Workflow DB: PV update + and + DASMON->>Dasmon listener: Instrument status + Dasmon listener->>Workflow DB: Instrument status + end + +Run status updates +.................. + +The Stream Management Service (SMS) posts messages on the queue ``APP.SMS`` at run start, run stop +and when the Streaming Translation Client (STC) completes translation to NeXus. + +.. mermaid:: + + sequenceDiagram + participant SMS + participant Dasmon listener + participant Workflow DB + SMS->>Dasmon listener: Run started + Dasmon listener->>Workflow DB: Add new data run + Dasmon listener->>Workflow DB: Run status + SMS->>Dasmon listener: Run stopped + Dasmon listener->>Workflow DB: Run status + SMS->>Dasmon listener: Translation succeeded + Dasmon listener->>Workflow DB: Run status + + +Experiment data post-processing +------------------------------- + +Autoreduction and cataloging +............................ The sequence diagram below describes the communication flow as a run gets post-processed. -The post-processing workflow is triggered when the Translation Service has finished translating the -data stream to NeXus and sends a message on the queue ``POSTPROCESS.DATA_READY`` specifying the -instrument, IPTS, run number and location of the NeXus file. +The post-processing workflow is triggered when the Streaming Translation Client (STC) has finished +translating the data stream to NeXus and sends a message on the queue ``POSTPROCESS.DATA_READY`` +specifying the instrument, IPTS, run number and location of the NeXus file. The post-processing workflow for the instrument is configurable in the database table ``report_task``. @@ -25,54 +72,26 @@ raw data in `ONCat `_ and cataloging of reduced data in .. mermaid:: sequenceDiagram - participant Translation Service + participant STC participant Workflow Manager participant Autoreducer participant ONCat + participant HFIR/SNS File Archive - Translation Service->>Workflow Manager: POSTPROCESS.DATA_READY + STC->>Workflow Manager: POSTPROCESS.DATA_READY Workflow Manager->>Autoreducer: CATALOG.ONCAT.DATA_READY Autoreducer->>Workflow Manager: CATALOG.ONCAT.STARTED Autoreducer->>ONCat: pyoncat Autoreducer->>Workflow Manager: CATALOG.ONCAT.COMPLETE Workflow Manager->>Autoreducer: REDUCTION.DATA_READY Autoreducer->>Workflow Manager: REDUCTION.STARTED - Autoreducer->>Autoreducer: run reduction + Autoreducer->>HFIR/SNS File Archive: reduced data, reduction log Autoreducer->>Workflow Manager: REDUCTION.COMPLETE Workflow Manager->>Autoreducer: REDUCTION_CATALOG.DATA_READY Autoreducer->>Workflow Manager: REDUCTION_CATALOG.STARTED Autoreducer->>ONCat: pyoncat Autoreducer->>Workflow Manager: REDUCTION_CATALOG.COMPLETE -.. - .. mermaid:: - - sequenceDiagram - participant Translation Service - participant Workflow Manager - participant Autoreducer - - Translation Service->>Workflow Manager: POSTPROCESS.DATA_READY - opt Cataloging - Workflow Manager->>Autoreducer: CATALOG.ONCAT.DATA_READY - Autoreducer->>Workflow Manager: CATALOG.ONCAT.STARTED - Note over Autoreducer: Ingest in ONCat - Autoreducer->>Workflow Manager: CATALOG.ONCAT.COMPLETE - end - opt Autoreduction - Workflow Manager->>Autoreducer: REDUCTION.DATA_READY - Autoreducer->>Workflow Manager: REDUCTION.STARTED - Note over Autoreducer: Execute autoreduction script - Autoreducer->>Workflow Manager: REDUCTION.COMPLETE - end - opt Reduced data cataloging - Workflow Manager->>Autoreducer: REDUCTION_CATALOG.DATA_READY - Autoreducer->>Workflow Manager: REDUCTION_CATALOG.STARTED - Note over Autoreducer: Ingest in ONCat - Autoreducer->>Workflow Manager: REDUCTION_CATALOG.COMPLETE - end - - Configuring the autoreduction ............................. @@ -93,129 +112,54 @@ parameters for instruments that have implemented WebMon->>Autoreducer: REDUCTION.CREATE_SCRIPT Autoreducer->>HFIR/SNS File archive: Update instrument reduction script -DASMON ------- -DASMON, from Data Acquisition (DAQ) System Monitor, provides instrument status and process variable -(PV) updates from the beamlines to WebMon. DASMON connects to the WebMon message broker to pass -status information, for example the current run number and count rate, to Dasmon listener. Due to -the high volume of PV updates, DASMON writes PV:s directly to the PostgreSQL database. - -.. mermaid:: +Live data visualization +-------------------------- - sequenceDiagram - participant DASMON - participant Dasmon listener - participant Workflow DB - DASMON->>Workflow DB: PV update - DASMON->>Dasmon listener: Instrument status - DASMON->>Workflow DB: Instrument status +Live Data Server (https://github.com/neutrons/live_data_server) is a service that serves plots to +the WebMon frontend. It provides a REST API with endpoints to create/update to and retrieve plots +from the Live Data Server database. -Dasmon listener ---------------- +Publish to Live Data Server from live data stream +................................................. -Stream Management Service (SMS) -............................... +Livereduce (https://github.com/mantidproject/livereduce/) allows scientists to reduce +data from an ongoing experiment, i.e. before translation to NeXus, by connecting to the live data +stream from the Stream Management Service (SMS). The instrument-specific livereduce processing +script can make the results available in WebMon by publishing plots to Live Data Server. .. mermaid:: sequenceDiagram participant SMS - participant Dasmon listener - participant Workflow DB - SMS->>Dasmon listener: Run started - Dasmon listener->>Workflow DB: Create new run - SMS->>Dasmon listener: Run stopped - SMS->>Dasmon listener: Translation succeeded - -Heartbeats from services -........................ - -Dasmon listener subscribes to heartbeats from the other services. There is a mechanism for alerting -admins by email when a service has missed heartbeats (needs to be verified that this still works). - -.. - .. mermaid:: - - sequenceDiagram - participant Other services - participant Dasmon listener - participant Workflow DB - actor Subscribed users - loop Every N s - Other services->>Dasmon listener: Heartbeat - Dasmon listener->>Workflow DB: Status update - end - opt Service has 3 missed heartbeats - Dasmon listener->>Subscribed users: Email - end - - -.. mermaid:: - - flowchart LR - SMS["SMS (per beamline)"] - PVSD["PVSD (per beamline)"] - DASMON["DASMON (per beamline)"] - STC - Autoreducers - DasmonListener - WorkflowDB[(DB)] - SMS-->|heartbeat|DasmonListener - PVSD-->|heartbeat|DasmonListener - DASMON-->|heartbeat|DasmonListener - STC-->|heartbeat|DasmonListener - Autoreducers-->|heartbeat|DasmonListener - WorkflowManager-->|heartbeat|DasmonListener - DasmonListener-->|heartbeat|DasmonListener - DasmonListener-->WorkflowDB - DasmonListener-.->|if missed 3 heartbeats|InstrumentScientist - -Live Data Server ----------------------------------------- + participant Livereduce + participant Live Data Server -WebMon has two modes of interaction with Live Data Server: publish (save) plots to the Live Data -Server database and display (fetch) plots from the database. + SMS->>Livereduce: data stream + loop Every N minutes + Livereduce->>Livereduce: run processing script + Livereduce->>Live Data Server: HTTP POST + end Publish to Live Data Server from autoreduction script ..................................................... -The instrument-specific autoreduction script can optionally publish plots (in either JSON format -or HTML div) to Live Data Server. +The instrument-specific autoreduction script can include a step to publish plots (in either JSON +format or HTML div) to Live Data Server. The Post-processing Agent repository includes some +convenience functions for generating and publishing plots in `publish_plot.py +`_. .. mermaid:: sequenceDiagram - participant WebMon + participant Workflow Manager participant Autoreducer participant Live Data Server - WebMon->>Autoreducer: REDUCTION.DATA_READY + Workflow Manager->>Autoreducer: REDUCTION.DATA_READY opt Publish plot - Autoreducer->>Live Data Server: publish_plot + Autoreducer->>Live Data Server: HTTP POST end -Publish to Live Data Server from live data stream -................................................. - -Livereduce (https://github.com/mantidproject/livereduce/) allows scientists to reduce -data from an ongoing experiment, i.e. before translation to NeXus, by connecting to the live data -stream from the Stream Management Service (SMS). The instrument-specific processing -script can make the results available in WebMon by publishing plots to Live Data Server. - -.. mermaid:: - - sequenceDiagram - participant SMS - participant Livereduce - participant Live Data Server - - SMS->>Livereduce: data stream - loop Every N minutes - Livereduce->>Livereduce: run processing script - Livereduce->>Live Data Server: publish plot - end - - Display plot from Live Data Server ................................ @@ -232,3 +176,38 @@ Data Server for a plot for that instrument and run number and display it if avai loop Every 60 s WebMon->>Live Data Server: HTTP GET end + +System diagnostics +------------------ + +WebMon displays system diagnostics information on https://monitor.sns.gov/dasmon/common/diagnostics/ +and diagnostics for DASMON and PVSD at the beamline at +`https://monitor.sns.gov/dasmon//diagnostics/ +`_. +Diagnostics information is primarily collected by Dasmon listener. + +Heartbeats from services +........................ + +Dasmon listener subscribes to heartbeats from the other services. There is a mechanism for alerting +admins by email when a service has missed heartbeats (needs to be verified that this still works). + +.. mermaid:: + + flowchart LR + SMS["SMS (per beamline)"] + PVSD["PVSD (per beamline)"] + DASMON["DASMON (per beamline)"] + STC + Autoreducers + DasmonListener + WorkflowDB[(DB)] + SMS-->|heartbeat|DasmonListener + PVSD-->|heartbeat|DasmonListener + DASMON-->|heartbeat|DasmonListener + STC-->|heartbeat|DasmonListener + Autoreducers-->|heartbeat|DasmonListener + WorkflowManager-->|heartbeat|DasmonListener + DasmonListener-->|heartbeat|DasmonListener + DasmonListener-->WorkflowDB + DasmonListener-.->|if missed 3 heartbeats|InstrumentScientist diff --git a/docs/developer/architecture/index.rst b/docs/developer/architecture/index.rst new file mode 100644 index 00000000..0fedd477 --- /dev/null +++ b/docs/developer/architecture/index.rst @@ -0,0 +1,9 @@ +Architecture +============ + +.. toctree:: + :maxdepth: 1 + :caption: Index + + overview + communication_flows diff --git a/docs/developer/design/overview.rst b/docs/developer/architecture/overview.rst similarity index 75% rename from docs/developer/design/overview.rst rename to docs/developer/architecture/overview.rst index b5e1df00..750ae324 100644 --- a/docs/developer/design/overview.rst +++ b/docs/developer/architecture/overview.rst @@ -1,21 +1,17 @@ -Design Overview -=============== - -.. toctree:: - :maxdepth: 1 - - communication_flows - +Overview +======== High-level architecture ----------------------- The diagram below describes the high-level architecture of WebMon, including some external resources -that WebMon interacts with. The gray box labeled "DAS" are services owned by the Data Acquisition -System team that feed information to WebMon. `ONCat `_ is the experiment -data catalog, which the autoreducers catalog runs to and the frontend fetches run metadata from. -The autoreducers access instrument-specific reduction scripts and experiment data files on the -HFIR/SNS file archive. The autoreducers also write reduced data files and logs to the file archive. +that WebMon interacts with. The direction of the arrows shows the direction of data flow, e.g. +Dasmon listener gets data from DASMON. The gray box labeled "DAS" are services owned by the Data +Acquisition System team that feed information to WebMon. `ONCat `_ is the +experiment data catalog, which the autoreducers catalog runs to and the frontend fetches run +metadata from. The autoreducers access instrument-specific reduction scripts and experiment data +files on the HFIR/SNS file archive. The autoreducers also write reduced data files and reduction log +files to the file archive. .. mermaid:: @@ -56,7 +52,7 @@ HFIR/SNS file archive. The autoreducers also write reduced data files and logs t TranslationService-->FileArchive FileArchive<-->Autoreducers style DAS fill:#D3D3D3, stroke-dasharray: 5 5 - classDef webMonStyle fill:#FFFFE0 + classDef webMonStyle fill:#FFFFE0, stroke:#B8860B class WorkflowManager,DasmonListener,Database,Autoreducers,LiveDataServer,LiveReduce,WebMon,LiveDataDB webMonStyle subgraph Legend @@ -69,6 +65,7 @@ HFIR/SNS file archive. The autoreducers also write reduced data files and logs t style Legend fill:#FFFFFF,stroke:#000000 class Internal webMonStyle +The section :ref:`communication_flows` provides sequence diagrams to show how the services interact. Message broker -------------- @@ -77,8 +74,6 @@ WebMon uses an `ActiveMQ `_ message broker for com services. The message broker also serves as a load balancer by distributing post-processing jobs to the available autoreducers in a round-robin fashion. -Service communication flows are described in :ref:`communication_flows`. - .. mermaid:: flowchart TB diff --git a/docs/developer/index.rst b/docs/developer/index.rst index 26f81a39..76447424 100644 --- a/docs/developer/index.rst +++ b/docs/developer/index.rst @@ -1,13 +1,21 @@ Developer documentation ======================= +Architecture +------------ + +.. toctree:: + :maxdepth: 1 + + architecture/overview + architecture/communication_flows + Developer Guide --------------- .. toctree:: :maxdepth: 1 - design/overview instruction/build instruction/test_fixture instruction/docker @@ -25,15 +33,12 @@ The web-monitor contains three independent Django applications * :py:mod:`webmon `: user facing web interface, visit the production version at `monitor.sns.gov`_. * :py:mod:`workflow`: backend manager. -and a mocked catalog services. - .. toctree:: :maxdepth: 1 dasmon/modules webmon/modules workflow/modules - catalog/modules .. _monitor.sns.gov: https://monitor.sns.gov/ From 46ceebfd5062f938cac8e71f8b64498d87a016f1 Mon Sep 17 00:00:00 2001 From: Marie Backman Date: Mon, 16 Sep 2024 10:59:27 -0400 Subject: [PATCH 7/7] address reviewer comments --- .../architecture/communication_flows.rst | 2 ++ docs/developer/architecture/overview.rst | 35 +++++++++++-------- 2 files changed, 22 insertions(+), 15 deletions(-) diff --git a/docs/developer/architecture/communication_flows.rst b/docs/developer/architecture/communication_flows.rst index dbacc79c..6fad9165 100644 --- a/docs/developer/architecture/communication_flows.rst +++ b/docs/developer/architecture/communication_flows.rst @@ -68,6 +68,8 @@ The post-processing workflow for the instrument is configurable in the database The diagram shows the three post-processing steps that are available: autoreduction, cataloging of raw data in `ONCat `_ and cataloging of reduced data in `ONCat `_. +Note that the sequence in the diagram is one possible workflow, but there are variations in the +configured sequence and the steps included depending on the instrument. .. mermaid:: diff --git a/docs/developer/architecture/overview.rst b/docs/developer/architecture/overview.rst index 750ae324..64e07d2d 100644 --- a/docs/developer/architecture/overview.rst +++ b/docs/developer/architecture/overview.rst @@ -4,14 +4,9 @@ Overview High-level architecture ----------------------- -The diagram below describes the high-level architecture of WebMon, including some external resources -that WebMon interacts with. The direction of the arrows shows the direction of data flow, e.g. -Dasmon listener gets data from DASMON. The gray box labeled "DAS" are services owned by the Data -Acquisition System team that feed information to WebMon. `ONCat `_ is the -experiment data catalog, which the autoreducers catalog runs to and the frontend fetches run -metadata from. The autoreducers access instrument-specific reduction scripts and experiment data -files on the HFIR/SNS file archive. The autoreducers also write reduced data files and reduction log -files to the file archive. +The diagram below describes the high-level architecture of WebMon, including both internal resources +that are considered part of WebMon and external systems that WebMon interacts with. +The arrows represent relationships between these services and resources. .. mermaid:: @@ -52,20 +47,30 @@ files to the file archive. TranslationService-->FileArchive FileArchive<-->Autoreducers style DAS fill:#D3D3D3, stroke-dasharray: 5 5 - classDef webMonStyle fill:#FFFFE0, stroke:#B8860B - class WorkflowManager,DasmonListener,Database,Autoreducers,LiveDataServer,LiveReduce,WebMon,LiveDataDB webMonStyle + classDef externalStyle fill:#FAEFDE, stroke:#B08D55 + class DASMON,TranslationService,SMS,FileArchive,ONCat externalStyle subgraph Legend direction LR - External["External resource"] Internal["Internal resource"] - External ~~~ Internal + External["External resource"] + Internal ~~~ External end - LiveReduce ~~~ External + LiveReduce ~~~ Internal style Legend fill:#FFFFFF,stroke:#000000 - class Internal webMonStyle + class External externalStyle + +The gray box labeled "DAS" are services managed by the Data Acquisition System team that pass +information to WebMon. The autoreducers interact with the HFIR/SNS file archive to access +instrument-specific reduction scripts and experiment data files. The autoreducers also write reduced +data files and reduction log files back to the file archive. + +Another external component is the experiment data catalog, `ONCat `_, where +the autoreducers catalog experiment metadata. The WebMon frontend retrieves and displays this +metadata from ONCat. -The section :ref:`communication_flows` provides sequence diagrams to show how the services interact. +The section :ref:`communication_flows` includes sequence diagrams that show how the services +interact. Message broker --------------