Sk/doc dir sync

.env.sample

@@ -4,15 +4,27 @@ PIXL_QUERY_TIMEOUT=10
 CLI_RETRY_SECONDS=300
 
 # PIXL PostgreSQL instance
-PIXL_DB_HOST=postgres
+PIXL_DB_HOST=postgres-exposed
+PIXL_DB_PORT=5432
 PIXL_DB_NAME=pixl
 PIXL_DB_USER=pixl
 PIXL_DB_PASSWORD=
 SKIP_ALEMBIC=false
+EXTERNAL_PIXL_DB=false
+
+# PIXL DB Postgres host
+CLI_PIXL_DB_HOST=localhost
+
+# Orthanc Raw PostgreSQL instance
+ORTHANC_RAW_DB_HOST=postgres-exposed  # change to correct host if PIXL DB is external postgres instance
+ORTHANC_RAW_DB_PORT=5432
+ORTHANC_RAW_DB_NAME=pixl
+ORTHANC_RAW_DB_USER=pixl
+ORTHANC_RAW_DB_PASSWORD=
 
 # Exposed ports
 HASHER_API_PORT=
-POSTGRES_PORT=
+CLI_PIXL_DB_PORT=
 ORTHANC_ANON_DICOM_PORT=
 ORTHANC_ANON_WEB_PORT=
 ORTHANC_RAW_DICOM_PORT=

README.md

@@ -153,6 +153,13 @@ The configuration file defines:
 
 - Project name: the `<project-slug>` name of the Project
 - The DICOM dataset modalities to retain (e.g. `["DX", "CR"]` for X-Ray studies)
+- The minimum number of instances required by a series (defaults to 2). Can be set higher than 1 to filter out
+  series with a single screenshot containing patient identifiable data
+- A list of series description filters (e.g. `['loc', 'pos']`). Series with descriptions matching any of these
+  filters will be skipped
+- A list of allowed manufacturers. By default, no manufacturers are allowed. For each manufacturer:
+  - A regex to identify the allowed manufacturer (e.g. `^philips`)
+  - A list of series numbers to exclude for the given manufacturer (e.g. `[3, 4]`)
 - The [anonymisation operations](/pixl_dcmd/README.md#tag-scheme-anonymisation) to be applied to the DICOM tags,
   by providing a file path to one or multiple YAML files.
   We currently allow two types of files:

cli/README.md

@@ -20,20 +20,47 @@ uv sync
 
 ## Usage
 
-**Note** The `rabbitmq`, `export-api` and `imaging-api` services must be started prior to using the CLI
+**Note:** The `rabbitmq`, `export-api` and `imaging-api` services must be started prior to using the CLI
 This is done by spinning up the necessary Docker containers through `docker compose`.
+
+See general pixl commands and subcommands with:
+
+```bash
+pixl --help
+```
+
+### Starting PIXL
+
 For convenience, we provide the `pixl dc` command, which acts as a wrapper for `docker compose`,
 but takes care of some of the configuration for you.
 
-See the commands and subcommands with
+**1) Default Start-up**
 
 ```bash
-pixl --help
+pixl dc up
+```
+
+**2) Start-up with External PIXL DB**
+
+PIXL can be set up so that the PIXL DB uses a separate postgres instance to Orthanc Raw, e.g. for production environment configurations. 
+Edit the .env file to enable this:
+
+```bash
+EXTERNAL_PIXL_DB=true
+
+CLI_PIXL_DB_PORT=7001
+
+ORTHANC_RAW_DB_HOST=postgres
+```
+
+Start-up PIXL:
+```bash
+pixl dc up
 ```
 
 ### Configuration
 
-The `rabbitmq` and `postgres` services are configured by setting the following environment variables
+The `rabbitmq` and PIXL DB `postgres` services are configured by setting the following environment variables
 (default values shown):
 
 ```sh
@@ -42,8 +69,8 @@ RABBITMQ_PORT=7008
 RABBITMQ_USERNAME=rabbitmq_username
 RABBITMQ_PASSWORD=rabbitmq_password
 
-POSTGRES_HOST=localhost
-POSTGRES_PORT=7001
+CLI_PIXL_DB_HOST=localhost
+CLI_PIXL_DB_PORT=7001
 PIXL_DB_USER=pixl_db_username
 PIXL_DB_PASSWORD=pixl_db_password
 PIXL_DB_NAME=pixl

cli/src/pixl_cli/_config.py

@@ -29,13 +29,13 @@
         "password": config("RABBITMQ_PASSWORD"),
     },
     "postgres": {
-        "host": config("POSTGRES_HOST"),
-        "port": int(config("POSTGRES_PORT")),
+        "host": config("CLI_PIXL_DB_HOST"),
+        "port": int(config("CLI_PIXL_DB_PORT")),
         "username": config("PIXL_DB_USER"),
         "password": config("PIXL_DB_PASSWORD"),
         "database": config("PIXL_DB_NAME"),
     },
-}  # type: dict
+}
 
 
 class APIConfig:

cli/src/pixl_cli/_docker_commands.py

@@ -34,13 +34,27 @@ def dc(args: tuple[str]) -> None:
     docker_args = list(args)
 
     if "up" in args:
-        docker_args = [*args, "--wait", "--build", "--remove-orphans"]
+        docker_args = _parse_up_args(args)
     if "down" in args:
         docker_args = _check_down_args(args)
 
     run_docker_compose(docker_args, working_dir=PIXL_ROOT)
 
 
+def _parse_up_args(args: tuple[str, ...]) -> list:
+    """Check up args and set docker compose profile"""
+    args_list = list(args)
+
+    up_index = args.index("up")
+    external_pixl_db_env = config("EXTERNAL_PIXL_DB", cast=bool)
+    args_list[up_index:up_index] = (
+        ["--profile", "postgres"] if external_pixl_db_env else ["--profile", "postgres-exposed"]
+    )
+
+    args_list.extend(["--wait", "--build", "--remove-orphans"])
+    return args_list
+
+
 def _check_down_args(args: tuple[str, ...]) -> list:
     """Stop all the PIXL services"""
     if config("ENV") == "prod" and "--volumes" in args:

docker-compose.yml

@@ -46,6 +46,13 @@ x-pixl-db: &pixl-db
     PIXL_DB_PASSWORD: ${PIXL_DB_PASSWORD}
     PIXL_DB_NAME: ${PIXL_DB_NAME}
 
+x-orthanc-raw-db: &orthanc-raw-db
+    ORTHANC_RAW_DB_HOST: ${ORTHANC_RAW_DB_HOST}
+    ORTHANC_RAW_DB_PORT: ${ORTHANC_RAW_DB_PORT}
+    ORTHANC_RAW_DB_USER: ${ORTHANC_RAW_DB_USER}
+    ORTHANC_RAW_DB_PASSWORD: ${ORTHANC_RAW_DB_PASSWORD}
+    ORTHANC_RAW_DB_NAME: ${ORTHANC_RAW_DB_NAME}
+
 x-azure-keyvault: &azure-keyvault
     AZURE_CLIENT_ID: ${EXPORT_AZ_CLIENT_ID}
     AZURE_CLIENT_SECRET: ${EXPORT_AZ_CLIENT_PASSWORD}
@@ -158,8 +165,9 @@ services:
         extra_hosts:
             - "host.docker.internal:host-gateway"
         depends_on:
-            postgres:
+            postgres-exposed:
                 condition: service_healthy
+                required: false
         healthcheck:
             test:
                 [
@@ -186,7 +194,7 @@ services:
         platform: linux/amd64
         command: /run/secrets
         environment:
-            <<: [*pixl-db, *proxy-common, *pixl-common-env]
+            <<: [*orthanc-raw-db, *proxy-common, *pixl-common-env]
             ORTHANC_NAME: "PIXL: Raw"
             ORTHANC_USERNAME: ${ORTHANC_RAW_USERNAME}
             ORTHANC_PASSWORD: ${ORTHANC_RAW_PASSWORD}
@@ -216,6 +224,10 @@ services:
         depends_on:
             postgres:
                 condition: service_healthy
+                required: false
+            postgres-exposed:
+                condition: service_healthy
+                required: false
             orthanc-anon:
                 condition: service_started
         healthcheck:
@@ -283,6 +295,10 @@ services:
                 condition: service_healthy
             postgres:
                 condition: service_healthy
+                required: false
+            postgres-exposed:
+                condition: service_healthy
+                required: false
             hasher-api:
                 condition: service_healthy
         ports:
@@ -352,16 +368,17 @@ services:
 
     ################################################################################
     # Data Stores
-    postgres:
+    postgres-exposed:
+        profiles: [postgres-exposed]
         build:
             context: .
             dockerfile: ./docker/postgres/Dockerfile
             args:
                 <<: *build-args-common
         environment:
-            POSTGRES_USER: ${PIXL_DB_USER}
-            POSTGRES_PASSWORD: ${PIXL_DB_PASSWORD}
-            POSTGRES_DB: ${PIXL_DB_NAME}
+            POSTGRES_USER: ${ORTHANC_RAW_DB_USER}
+            POSTGRES_PASSWORD: ${ORTHANC_RAW_DB_PASSWORD}
+            POSTGRES_DB: ${ORTHANC_RAW_DB_NAME}
             PGTZ: ${TZ:-Europe/London}
         env_file:
             - ./docker/common.env
@@ -371,9 +388,36 @@ services:
               source: postgres-data
               target: /var/lib/postgresql/data
         ports:
-            - "${POSTGRES_PORT}:5432"
+            - "${CLI_PIXL_DB_PORT}:5432"
+        healthcheck:
+            test: ["CMD", "pg_isready", "-U", "${ORTHANC_RAW_DB_USER}", "--dbname", "${ORTHANC_RAW_DB_NAME}"]
+            interval: 10s
+            timeout: 30s
+            retries: 5
+        restart: always
+        networks:
+            - pixl-net
+    postgres:
+        profiles: [postgres]
+        build:
+            context: .
+            dockerfile: ./docker/postgres/Dockerfile
+            args:
+                <<: *build-args-common
+        environment:
+            POSTGRES_USER: ${ORTHANC_RAW_DB_USER}
+            POSTGRES_PASSWORD: ${ORTHANC_RAW_DB_PASSWORD}
+            POSTGRES_DB: ${ORTHANC_RAW_DB_NAME}
+            PGTZ: ${TZ:-Europe/London}
+        env_file:
+            - ./docker/common.env
+        command: postgres -c 'config_file=/etc/postgresql/postgresql.conf'
+        volumes:
+            - type: volume
+              source: postgres-data
+              target: /var/lib/postgresql/data
         healthcheck:
-            test: ["CMD", "pg_isready", "-U", "${PIXL_DB_USER}", "--dbname", "${PIXL_DB_NAME}"]
+            test: ["CMD", "pg_isready", "-U", "${ORTHANC_RAW_DB_USER}", "--dbname", "${ORTHANC_RAW_DB_NAME}"]
             interval: 10s
             timeout: 30s
             retries: 5

docs/README.md

@@ -0,0 +1,18 @@
+## 'PIXL/docs' Directory Contents
+
+<details>
+<summary>
+<h3> Subdirectories with links to the relevant README </h3> 
+
+</summary>
+
+[archive](./archive/README.md)
+
+[design](./design/README.md)
+
+[developer](./developer/README.md)
+
+[joss-publication](./joss-publication/README.md)
+
+</details>
+

docs/archive/PIXLv1/Considerations.md

@@ -0,0 +1,15 @@
+## Risks and Considerations
+
+### Technical Risks
+The primary technical risk is overburdening the PACS & VNA and causing an adverse impact on the operational performance of these systems.  
+To mitigate this risk, queries will be managed with a task queue. The system will enforce rate limiting of any commands sent to the PACS & VNA with an adapted [token bucket](https://en.wikipedia.org/wiki/Token_bucket) algorithm which can be adjusted at runtime in response to system load. A [circuit breaker](https://en.wikipedia.org/wiki/Circuit_breaker_design_pattern) will wrap the retrieval processes and act as fail-safe. Individual request retries will be subject to an [exponential backoff](https://en.wikipedia.org/wiki/Exponential_backoff) strategy.
+
+
+### Security Considerations
+#### Inbound access to the Cloud Environment in Azure  
+It is expected that a VPN connection (or ExpressRoute connection) between the on-prem UCLH estate and Azure will not initially be available.  
+Point-to-point firewall restrictions and Azure access tokens will manage secure communication between PIXL and the DICOM service.
+
+#### Outbound access 
+All outbound connections will be over HTTPS.
+The existing proxy service will be relied upon to manage outbound access from the GAE.