[bugfix] HTML faild validator.w3.org checks

Implements checks for mass channel activity to ensure proper functionality.

Introduces methods to verify if mass channels are active and to set errors when they are not connected.

Enhances reliability of data logging by avoiding unnecessary processing when mass channels are inactive.

.gitignore

@@ -211,3 +211,9 @@ flycheck_*.el
 /network-security.data
 
 
+/__simulate_fail.json
+/mailing_list.yaml
+
+.vscode/
+
+*.code-workspace

Dockerfile

@@ -0,0 +1,40 @@
+FROM python:3
+
+# metadata
+LABEL maintainer="Kasper D. Fischer <kasper.fischer@rub.de>"
+LABEL version="0.2-docker"
+LABEL description="Docker image for the survBot application"
+
+# install required system packages
+RUN apt update && apt install -y bind9-host iputils-ping
+
+# create user and group and home directory
+RUN groupadd -r survBot && useradd -r -g survBot survBot
+RUN mkdir -p /home/survBot && chown -R survBot:survBot /home/survBot
+
+# change working directory
+WORKDIR /usr/src/app
+
+# install required python packages
+RUN --mount=type=bind,source=requirements.txt,target=/tmp/requirements.txt \
+    pip install --no-cache-dir --requirement /tmp/requirements.txt
+
+# copy application files
+COPY survBot.py utils.py write_utils.py LICENSE README.md ./
+
+# copy configuration files
+VOLUME /usr/src/app/conf
+COPY parameters.yaml mailing_list.yam[l] simulate_fail.jso[n] conf/
+RUN ln -s conf/simulate_fail.json simulate_fail.json
+
+# copy www files
+VOLUME /usr/src/app/www
+COPY logo.pn[g] stylesheets/desktop.css stylesheets/mobile.css www/
+RUN ln -s www/survBot_out.html www/index.html
+
+# change ownership of working directory
+RUN chown -R survBot:survBot /usr/src/app
+
+# run the application as user survBot
+USER survBot
+CMD [ "python", "./survBot.py", "-html", "www", "-parfile", "conf/parameters.yaml" ]

README.md

@@ -1,9 +1,9 @@
 # survBot
 
-version: 0.1
+version: 0.2
 
 survBot is a small program used to track station quality channels of DSEBRA stations via PowBox output over SOH channels
- by analysing contents of a Seiscomp3 datapath.
+ by analyzing contents of a Seiscomp data archive.
 
 ## Requirements
 
@@ -26,20 +26,67 @@ to use the GUI:
 
 Configurations of *datapath*, *networks*, *stations* etc. can be done in the **parameters.yaml** input file.
 
-The main program is executed by entering
+The main program with html output is executed by entering
 
 ```shell script
-python survBot.py
+python survBot.py -html path_for_html_output
 ```
 
+There are example stylesheets in the folder *stylesheets* that can be copied into the path_for_html_output if desired.
+
 The GUI can be loaded via
 
 ```shell script
 python survBotGui.py
 ```
 
+### Docker
+
+To run the program in a Docker container, first build the image:
+
+```shell script
+docker build -t survbot .
+```
+
+Then run the container:
+
+```shell script
+docker run -v /path/to/conf-dir:/usr/src/app/conf -v /path/to/output:/usr/src/app/www survbot
+```
+
+The directory `/path/to/conf-dir` should contain the `parameters.yaml` file, and the directory `/path/to/output` will contain the output HTML files.
+
+### Configuration of the e-mail server settings
+
+The e-mail server settings can be configured in the `parameters.yaml` file. The following settings are available:
+
+* `mailserver`: the address of the mail server
+* `auth_type`: the authentication type for the mail server (`None`, `SSL`, `TLS`)
+* `port`: the port of the mail server
+* `user`: the username for the mail server (if required)
+* `password`: the password for the mail server (if required)
+
+The `user` and `password` fields are optional, and can be left empty if the mail server does not require authentication. The `auth_type` field can be set to `None` if no authentication is required, `SSL` if the mail server requires SSL authentication, or `TLS` if the mail server requires TLS authentication. If the `user` or `password` fields are set to `Docker` ore `ENV` the program will try to read the values from the docker secrets `mail_user` and `mail_password` or environment variables `MAIL_USER` and `MAIL_PASSWORD` respectively. Docker secrets are only available in Docker Swarm mode, i.e. if the program is run as a service.
+
+## Version Changes
+
+### 0.2
+
+* surveillance of mass, clock and gaps
+* individual mailing lists for different stations
+* html mail with recent status information
+* updated web page design
+* restructured parameter file
+* recognize if PBox is disconnected
+
+### 0.2-docker
+
+* added Dockerfile for easy deployment
+* added more settings for connection to a mail server
+
 ## Staff
 
-Original author: M.Paffrath (marcel.paffrath@rub.de)
+Original author: M.Paffrath (<marcel.paffrath@rub.de>)
+Contributions: Kasper D. Fischer (<kasper.fischer@rub.de>)
 
-November 2022
+Jan 2025

__init__.py

@@ -0,0 +1,3 @@
+# survBot is a small program used to track station quality channels of DSEBRA stations via PowBox output
+# over SOH channels by analysing contents of a Seiscomp3 datapath.
+__version__ = "0.2-docker"

mailing_list.yaml

@@ -0,0 +1,9 @@
+# specify mail addresses and station network ids for which information mails shall be sent, e.g.:
+# "mail.address@provider.com, mail.address2@provider2.com":
+#   - 1Y.GR01
+#   - 1Y.GR02
+# "mail.address3@provider.com":
+#   - 1Y.GR03
+
+#"kasper.fischer@rub.de":
+#    - 1Y.GR01

parameters.yaml

@@ -1,26 +1,34 @@
 # Parameters file for Surveillance Bot
-datapath: '/data/SDS/'      # SC3 Datapath
+datapath: "/data/SDS/"                    # path to SDS data archive
-networks: ['1Y', 'HA']
+networks: ["1Y", "HA", "MK"]              # select networks, list or str
-stations: '*'
+stations: "*"                             # select stations, list or str
-locations: '*'
+locations: "*"                            # select locations, list or str
-channels: ['EX1', 'EX2', 'EX3', 'VEI']    # Specify SOH channels, currently supported EX[1-3] and VEI
+stations_blacklist: ["ATHR", "DOMV", "EREA", "GR19", "GR27", "HAVD", "LAKA", "LFKM", "TEST"]      # exclude these stations
-stations_blacklist: ['TEST', 'EREA']
+networks_blacklist: []                    # exclude these networks
-networks_blacklist: []
+interval: 60               # Perform checks every x seconds
-interval: 20               # Perform checks every x seconds
+n_track: 360               # wait n_track * intervals before performing an action (i.e. send mail/end highlight status)
 timespan: 7                # Check data of the recent x days
-verbosity: 0
+verbosity: 0               # verbosity flag for program console output (not logging)
+logging_level: WARN        # set logging level (info, warning, debug)
 track_changes: True        # tracks all changes since GUI startup by text highlighting (GUI only)
+warn_count: False          # show number of warnings and errors in table
+min_sample: 5              # minimum samples for raising Warn/FAIL
+dt_thresh: [300, 1800]     # threshold (s) for timing delay colourisation (yellow/red)
+html_figures: True         # Create html figure directory and links
+reread_parameters: True    # reread parameters file (change parameters on runtime, not for itself/GUI refresh/datapath)
 
 POWBOX:
   pb_ok: 1  # Voltage for PowBox OK
   pb_SOH2: # PowBox channel 2 voltage translations
-    1: {"230V": 'OK', "12V": "OK"}
+    -1: {"230V": "PBox under 1V", "12V": "PBox under 1V"}
+    1: {"230V": "OK", "12V": "OK"}
     2: {"230V": "OFF", "12V": "OK"}
     3: {"230V": "OK", "12V": "overvoltage"}
     4: {"230V": "OK", "12V": "undervoltage"}
     4.5: {"230V": "OFF", "12V": "overvoltage"}
     5: {"230V": "OFF", "12V": "undervoltage"}
   pb_SOH3: # PowBox channel 3 voltage translations
+    -1: {"router": "PBox under 1V", "charger": "PBox under 1V"}
     1: {"router": "OK", "charger": "OK"}
     2: {"router": "OK", "charger": "0 < resets < 3"}
     2.5: {"router": "OK", "charger": "locked"}
@@ -28,10 +36,124 @@ POWBOX:
     4: {"router": "FAIL", "charger": "0 < resets < 3"}
     5: {"router": "FAIL", "charger": "locked"}
 
-
+# Thresholds for program warnings/voltage classifications
 THRESHOLDS:
   pb_thresh: 0.2               # Threshold for PowBox Voltage check +/- (V)
   max_temp: 50                 # max temperature for temperature warning
+  critical_temp: 65            # max temperature for critical warning (fail)
   low_volt: 12                 # min voltage for low voltage warning
   high_volt: 14.8              # max voltage for over voltage warning
   unclassified: 5              # min voltage samples not classified for warning
+  max_vm_warn: 1.5             # threshold for mass offset (warn), fail)
+  max_vm_fail: 2.5             # threshold for mass offset (warn), fail)
+  clockquality_warn: 90        # warn level - clock quality ranges from 0 % to 100 % with 100 % being the best level
+  clockquality_fail: 70        # fail level
+  min_gap: 0.1                 # minimum for gap declaration, should be > 0 [s]
+
+
+# ---------------------------------- Specification of input channels ---------------------------------------------------
+# Currently supported: EX[1-3], VEI, VM[1-3], LCQ
+#
+# For each channel a factor 'unit' for unit conversion (e.g. to SI) can be provided, as well as a 'name'
+# and 'ticks' [ymin, ymax, ystep] for plotting.
+# 'warn' and 'fail' plot horizontal lines in corresponding colors (can be str in THRESHOLDS, int/float or iterable)
+# keyword "pb_SOH2" or "pb_SOH3" can be used to extract warning values from above POWBOX parameter definition
+#
+# 'transform' can be provided for plotting to perform arithmetic operations in given order, e.g.:
+#  transform: - ["*", 20]
+#             - ["-", 20]
+#  --> PBox EX1 V to deg C: 20 * x -20
+CHANNELS:
+  EX1:
+    unit: 1e-6
+    name: "PowBox Temperature (°C)"
+    ticks: [-10, 50, 10]
+    transform:
+      - ["*", 20]
+      - ["-", 20]
+    warn: "max_temp"
+  EX2:
+    unit: 1e-6
+    name: "PowBox 230V/12V (V)"
+    ticks: [0, 5, 1]
+    warn: "pb_SOH2"
+  EX3:
+    unit: 1e-6
+    name: "PowBox Router/Charger (V)"
+    ticks: [0, 5, 1]
+    warn: "pb_SOH3"
+  VEI:
+    unit: 1e-3
+    name: "Datalogger (V)"
+    ticks: [9, 15, 1]
+    warn: ["low_volt", "high_volt"]
+    fail: 10.5
+  VM1:
+    unit: 1e-6
+    name: "Mass position W (V)"
+    ticks: [-2.5, 2.5, 1]
+    warn: [-1.5, 1.5]
+    fail: [-2.5, 2.5]
+  VM2:
+    unit: 1e-6
+    name: "Mass position V (V)"
+    ticks: [-2.5, 2.5, 1]
+    warn: [-1.5, 1.5]
+    fail: [-2.5, 2.5]
+  VM3:
+    unit: 1e-6
+    name: "Mass position U (V)"
+    ticks: [-2.5, 2.5, 1]
+    warn: [-1.5, 1.5]
+    fail: [-2.5, 2.5]
+  LCQ:
+    name: "Clock quality (%)"
+    ticks: [0, 100, 20]
+    warn: "clockquality_warn"
+    fail: "clockquality_fail"
+
+# specify data channels (can be additional to the above). From these channels only headers will be read
+data_channels: ["HHZ", "HHN", "HHE"]
+
+
+# ---------------------------------------- OPTIONAL PARAMETERS ---------------------------------------------------------
+# deactivate powbox surveillance for stations (e.g. for solar powered), key: station, value: status message (abbr.)
+no_pbox_stations: {'GR33': 'SOL', 'GR27B': 'DCN', 'RP01': 'noPBox', 'RP02': 'noPBox', 'RP03': 'noPBox', 'RP04': 'noPBox', 'RP05B': 'noPBox', 'RP06': 'noPBox', 'RP07': 'noPBox', 'RP08': 'noPBox'}
+
+# deactivate mass position surveillance for stations without connected mass channels (e.g. STS-2 instruments), key: station, value: status message (abbr.)
+no_mass_stations: {'BIA': 'DCN', 'KNEZ': 'DCN', 'KRUS': 'DCN', 'OHR': 'DCN', 'OTOV': 'DCN', 'SKO': 'DCN', 'STIP': 'DCN', 'VAY': 'DCN', 'ZUPA': 'DCN'}
+
+# add links to html table with specified key as column and value as relative link, interpretable string parameters:
+# nw (e.g. 1Y), st (e.g. GR01A), nwst_id (e.g. 1Y.GR01A)
+add_links:
+  # for example: slmon: {"URL": "path/{nw}_{st}.html", "text": "link"}
+  slmon: {"URL": "../slmon/{nw}_{st}.html", "text": "show"}
+  24h-plot: {"URL": "../scheli/{nw}/{st}.png", "text": "plot"}
+  ppsd: {"URL": "../ppsd/{nw}.{st}.html", "text": "show"}
+
+# add station-independent links below html table (list items separated with -)
+add_global_links:
+  # for example: - {"text": "our homepage", "URL": "https://www.rub.de"}
+  - {"text": "show recent events on map",
+     "URL": "https://fdsnws.geophysik.ruhr-uni-bochum.de/map/?lat=39.5&lon=21&zoom=7&baselayer=osm"}
+
+# html logo at page bottom (path relative to html directory)
+html_logo: "logo.png"
+
+# E-mail notifications
+EMAIL:
+  # specify mail server and credentials
+  # port, auth_type, user and password are only required if mailserver is not set to "localhost"
+  # user and password can be set to "ENV" or "DOCKER" to read from environment variables or docker secrets
+  mailserver: "smtp.example.org"      # mail server
+  auth_type: "SSL"                    # mail authentication type, can be "SSL", "TLS" or "None"
+  port: 465                           # mail port, default 465 for SSL, 587 for TLS
+  user: "DOCKER"                      # mail user, read from environment variable if set to "ENV" or from docker secret if set to "DOCKER"
+  password: "DOCKER"                  # mail password, read from environment variable if set to "ENV" or from docker secret if set to "DOCKER"
+  # specify mail recipients, sender and blacklists
+  addresses: ["test@example.org"]  # list of mail addresses for info mails
+  sender: "<test@example.org>"              # mail sender
+  stations_blacklist: []                                          # do not send emails for specific stations
+  networks_blacklist: []                                          # do not send emails for specific network
+  # specify recipients for single stations in a yaml: key = email-address, val = station list (e.g. [1Y.GR01, 1Y.GR02])
+  external_mail_list: "conf/mailing_list.yaml"

requirements.txt

@@ -0,0 +1,36 @@
+Brotli==1.1.0
+certifi==2025.1.31
+cffi==1.17.1
+charset-normalizer==3.4.1
+contourpy==1.3.1
+cycler==0.12.1
+decorator==5.2.1
+fonttools==4.56.0
+greenlet==3.1.1
+h2==4.2.0
+hpack==4.1.0
+hyperframe==6.1.0
+idna==3.10
+kiwisolver==1.4.8
+lxml==5.3.1
+matplotlib==3.8.4
+munkres==1.1.4
+numpy==1.26.4
+obspy==1.4.1
+packaging==24.2
+pillow==11.1.0
+pip==25.0.1
+pycparser==2.22
+pyparsing==3.2.1
+PySocks==1.7.1
+python-dateutil==2.9.0.post0
+PyYAML==6.0.2
+requests==2.32.3
+scipy==1.15.2
+setuptools==75.8.2
+six==1.17.0
+SQLAlchemy==1.4.54
+unicodedata2==16.0.0
+urllib3==2.3.0
+wheel==0.45.1
+zstandard==0.23.0

stylesheets/desktop.css

@@ -0,0 +1,57 @@
+body {
+    background-color: #ffffff;
+    text-align: center;
+    padding-bottom: 30px;
+    font-family: "Helvetica", "sans-serif";
+}
+
+table {
+    position: relative
+}
+
+td {
+    border-radius: 2px;
+    padding: 0px;
+    white-space: nowrap;
+}
+
+th {
+    background-color: #17365c;
+    color: #fff;
+    border-radius: 2px;
+    padding: 3px 1px;
+    position: sticky;
+    top: 0;
+}
+
+a:link, a:visited {
+    background-color: #e8e8e8;
+    color: #000;
+    text-decoration: none;
+    display: block;
+    border-radius: 4px;
+    border: 1px solid #ccc;
+}
+
+a:hover {
+    background-color: #aaa;
+    display: block;			  
+}
+
+.blink-bg {
+    animation: blinkingBackground 2s infinite;
+}
+@keyframes blinkingBackground{
+    0%		{ background-color: #ffcc00;}
+    50%		{ background-color: #ff3200;}
+    100%	{ background-color: #ffcc00;}
+}
+
+.footer {
+   position: fixed;
+   left: 0;
+   bottom: 0;
+   width: 100%;
+   height: 50px;
+   text-align: center;
+}

stylesheets/mobile.css

@@ -0,0 +1,61 @@
+body {
+    background-color: #ffffff;
+    text-align: center;
+    padding-bottom: 30px;
+    font-family: "Helvetica", "sans-serif";
+}
+
+table {
+    position: relative
+}
+
+td {
+    border-radius: 3px;
+    padding: 10px 2px;
+    white-space: nowrap;
+}
+
+th {
+    background-color: #17365c;
+    color: #fff;
+    border-radius: 3px;
+    padding: 10px 2px;
+    position: sticky;
+    top: 0;
+}
+
+a:link, a:visited {
+    background-color: #e8e8e8;
+    color: #000;
+    text-decoration: none;
+    display: block;
+    border-radius: 6px;
+    border: 1px solid #ccc;
+}
+
+a:hover {
+    background-color: #aaa;
+    display: block;			  
+}
+
+.hidden-mobile {
+    display: none;
+}
+
+.blink-bg {
+    animation: blinkingBackground 2s infinite;
+}
+@keyframes blinkingBackground{
+    0%		{ background-color: #ffcc00;}
+    50%		{ background-color: #ff3200;}
+    100%	{ background-color: #ffcc00;}
+}
+
+.footer {
+   position: fixed;
+   left: 0;
+   bottom: 0;
+   width: 100%;
+   height: 50px;
+   text-align: center;
+}

submit_bot.sh

@@ -1,15 +1,24 @@
 #!/bin/bash
-ulimit -s 8192
 
 #$ -l low
-#$ -l os=*stretch
+#$ -l h_vmem=2.5G
+#$ -l mem=2.5G
+#$ -l h_stack=INFINITY
 #$ -cwd
 #$ -pe smp 1
-##$ -q "*@minos15"
+#$ -binding linear:1
-
+#$ -N survBot
-export PYTHONPATH="$PYTHONPATH:/home/marcel/git/"
+#$ -o /data/www/~kasper/survBot/survBot_bg.log
+#$ -e /data/www/~kasper/survBot/survBot_bg.err
+#$ -m e
+#$ -M kasper.fischer@rub.de
 
 source /opt/anaconda3/etc/profile.d/conda.sh
-conda activate py37
+conda activate survBot
 
-python survBot.py
+# environment variables for numpy to prevent multi threading
+export MKL_NUM_THREADS=1
+export NUMEXPR_NUM_THREADS=1
+export OMP_NUM_THREADS=1
+
+python survBot.py -html '/data/www/~kasper/survBot'

survBotGUI.py

@@ -1,4 +1,5 @@
-#! /usr/bin/env python
+#!/usr/bin/env python
+# -*- coding: utf-8 -*-
 """
 GUI overlay for the main survBot to show quality control of different stations specified in parameters.yaml file.
 """
@@ -9,7 +10,8 @@ __author__ = 'Marcel Paffrath'
 import os
 import sys
 import traceback
-from datetime import timedelta
+
+import logging
 
 try:
     from PySide2 import QtGui, QtCore, QtWidgets
@@ -22,7 +24,6 @@ except ImportError:
         except ImportError:
             raise ImportError('Could import neither of PySide2, PySide6 or PyQt5')
 
-import matplotlib
 from matplotlib.figure import Figure
 
 if QtGui.__package__ in ['PySide2', 'PyQt5', 'PySide6']:
@@ -34,13 +35,15 @@ else:
 from obspy import UTCDateTime
 
 from survBot import SurveillanceBot
+from write_utils import *
+from utils import get_bg_color, modify_stream_for_plot, set_axis_yticks, set_axis_color, plot_axis_thresholds
 
 try:
     from rest_api.utils import get_station_iccid
     from rest_api.rest_api_utils import get_last_messages, send_message, get_default_params
     sms_funcs = True
 except ImportError:
-    print('Could not load rest_api utils, SMS functionality disabled.')
+    logging.warning('Could not load rest_api utils, SMS functionality disabled.')
     sms_funcs = False
 
 deg_str = '\N{DEGREE SIGN}C'
@@ -53,10 +56,9 @@ class Thread(QtCore.QThread):
     """
     update = QtCore.Signal()
 
-    def __init__(self, parent, runnable, verbosity=0):
+    def __init__(self, parent, runnable):
         super(Thread, self).__init__(parent=parent)
         self.setParent(parent)
-        self.verbosity = verbosity
         self.runnable = runnable
         self.is_active = True
 
@@ -68,33 +70,23 @@ class Thread(QtCore.QThread):
             self.update.emit()
         except Exception as e:
             self.is_active = False
-            print(e)
+            logging.error(e)
-            print(traceback.format_exc())
+            logging.debug(traceback.format_exc())
         finally:
-            if self.verbosity > 0:
+            logging.info(f'Time for Thread execution: {UTCDateTime() - t0}')
-                print(f'Time for Thread execution: {UTCDateTime() - t0}')
 
 
 class MainWindow(QtWidgets.QMainWindow):
-    def __init__(self, parameters='parameters.yaml', dt_thresh=(300, 1800)):
+    def __init__(self, parameters='parameters.yaml'):
         """
         Main window of survBot GUI.
         :param parameters: Parameters dictionary file (yaml format)
-        :param dt_thresh: threshold for timing delay colourisation (yellow/red)
         """
         super(MainWindow, self).__init__()
 
-        # some GUI default colors
-        self.colors_dict = {'FAIL': (255, 50, 0, 255),
-                            'NO DATA': (255, 255, 125, 255),
-                            'WARN': (255, 255, 125, 255),
-                            'WARNX': lambda x: (min([255, 200 + x**2]), 255, 125, 255),
-                            'OK': (125, 255, 125, 255),
-                            'undefined': (230, 230, 230, 255)}
-
         # init some attributes
-        self.dt_thresh = dt_thresh
         self.last_mouse_loc = None
+        self.status_message = ''
         self.starttime = UTCDateTime()
 
         # setup main layout of the GUI
@@ -107,6 +99,7 @@ class MainWindow(QtWidgets.QMainWindow):
         self.survBot = SurveillanceBot(parameter_path=parameters)
         self.parameters = self.survBot.parameters
         self.refresh_period = self.parameters.get('interval')
+        self.dt_thresh = [int(val) for val in self.parameters.get('dt_thresh')]
 
         # create thread that is used to update
         self.thread = Thread(parent=self, runnable=self.survBot.execute_qc)
@@ -136,10 +129,10 @@ class MainWindow(QtWidgets.QMainWindow):
         self.table.setRowCount(len(station_list))
         self.table.setHorizontalHeaderLabels(keys)
 
-        for index, st_id in enumerate(station_list):
+        for index, nwst_id in enumerate(station_list):
             item = QtWidgets.QTableWidgetItem()
-            item.setText(str(st_id.rstrip('.')))
+            item.setText(str(nwst_id.rstrip('.')))
-            item.setData(QtCore.Qt.UserRole, st_id)
+            item.setData(QtCore.Qt.UserRole, nwst_id)
             self.table.setVerticalHeaderItem(index, item)
 
         self.main_layout.addWidget(self.table)
@@ -186,38 +179,38 @@ class MainWindow(QtWidgets.QMainWindow):
         header_item = self.table.verticalHeaderItem(row_ind)
         if not header_item:
            return
-        st_id = header_item.data(QtCore.Qt.UserRole)
+        nwst_id = header_item.data(QtCore.Qt.UserRole)
 
         context_menu = QtWidgets.QMenu()
         read_sms = context_menu.addAction('Get last SMS')
         send_sms = context_menu.addAction('Send SMS')
         action = context_menu.exec_(self.mapToGlobal(self.last_mouse_loc))
         if action == read_sms:
-           self.read_sms(st_id)
+           self.read_sms(nwst_id)
         elif action == send_sms:
-           self.send_sms(st_id)
+           self.send_sms(nwst_id)
 
-    def read_sms(self, st_id):
+    def read_sms(self, nwst_id):
         """ Read recent SMS over rest_api using whereversim portal """
-        station = st_id.split('.')[1]
+        station = nwst_id.split('.')[1]
         iccid = get_station_iccid(station)
         if not iccid:
-            print('Could not find iccid for station', st_id)
+            logging.info(f'Could not find iccid for station: {nwst_id}')
             return
         sms_widget = ReadSMSWidget(parent=self, iccid=iccid)
-        sms_widget.setWindowTitle(f'Recent SMS of station: {st_id}')
+        sms_widget.setWindowTitle(f'Recent SMS of station: {nwst_id}')
         if sms_widget.data:
             sms_widget.show()
         else:
             self.notification('No recent messages found.')
 
-    def send_sms(self, st_id):
+    def send_sms(self, nwst_id):
         """ Send SMS over rest_api using whereversim portal """
-        station = st_id.split('.')[1]
+        station = nwst_id.split('.')[1]
         iccid = get_station_iccid(station)
 
         sms_widget = SendSMSWidget(parent=self, iccid=iccid)
-        sms_widget.setWindowTitle(f'Send SMS to station: {st_id}')
+        sms_widget.setWindowTitle(f'Send SMS to station: {nwst_id}')
         sms_widget.show()
 
     def set_clear_on_refresh(self):
@@ -225,37 +218,30 @@ class MainWindow(QtWidgets.QMainWindow):
 
     def fill_status_bar(self):
         """ Set status bar text """
+        self.status_message = self.survBot.status_message
         status_bar = self.statusBar()
-        timespan = timedelta(seconds=int(self.parameters.get('timespan') * 24 * 3600))
+        status_bar.showMessage(self.status_message)
-        status_bar.showMessage(f'Program starttime (UTC) {self.starttime.strftime("%Y-%m-%d %H:%M:%S")} | '
-                               f'Refresh period: {self.refresh_period}s | '
-                               f'Showing data of last {timespan}')
 
     def fill_table(self):
         """ Fills the table with most recent information. Executed after execute_qc thread is done or on refresh. """
+
+        # fill status bar first with new time
+        self.fill_status_bar()
+
         for col_ind, check_key in enumerate(self.survBot.keys):
-            for row_ind, st_id in enumerate(self.survBot.station_list):
+            for row_ind, nwst_id in enumerate(self.survBot.station_list):
-                status_dict, detailed_dict = self.survBot.analysis_results.get(st_id)
+                status_dict = self.survBot.analysis_results.get(nwst_id)
                 status = status_dict.get(check_key)
-                detailed_message = detailed_dict.get(check_key)
+                message, detailed_message = status.get_status_str()
-                if check_key == 'last active':
+
-                    bg_color = self.get_time_delay_color(status)
+                dt_thresh = [timedelta(seconds=sec) for sec in self.dt_thresh]
-                elif check_key == 'temp':
+                bg_color = get_bg_color(check_key, status, dt_thresh)
-                    bg_color = self.get_temp_color(status)
+                if check_key == 'temp':
-                    if not type(status) in [str]:
+                    if not type(message) in [str]:
-                        status = str(status) + deg_str
+                        message = str(message) + deg_str
-                else:
-                    statussplit = status.split(' ')
-                    if len(statussplit) > 1 and statussplit[0] == 'WARN':
-                        x = int(status.split(' ')[-1].lstrip('(').rstrip(')'))
-                        bg_color = self.colors_dict.get('WARNX')(x)
-                    else:
-                        bg_color = self.colors_dict.get(status)
-                if not bg_color:
-                    bg_color = self.colors_dict.get('undefined')
 
                 # Continue if nothing changed
-                text = str(status)
+                text = str(message)
                 cur_item = self.table.item(row_ind, col_ind)
                 if cur_item and text == cur_item.text():
                     if not self.parameters.get('track_changes') or self.clear_on_refresh:
@@ -266,9 +252,9 @@ class MainWindow(QtWidgets.QMainWindow):
 
                 # Create new data item
                 item = QtWidgets.QTableWidgetItem()
-                item.setText(str(status))
+                item.setText(str(message))
                 item.setTextAlignment(QtCore.Qt.AlignCenter)
-                item.setData(QtCore.Qt.UserRole, (st_id, check_key))
+                item.setData(QtCore.Qt.UserRole, (nwst_id, check_key))
 
                 # if text changed (known from above) set highlight color/font else (new init) set to default
                 cur_item = self.table.item(row_ind, col_ind)
@@ -292,24 +278,6 @@ class MainWindow(QtWidgets.QMainWindow):
         # table filling/refreshing done, set clear_on_refresh to False
         self.clear_on_refresh = False
 
-    def get_time_delay_color(self, dt):
-        """ Set color of time delay after thresholds specified in self.dt_thresh """
-        dt_thresh = [timedelta(seconds=sec) for sec in self.dt_thresh]
-        if dt < dt_thresh[0]:
-            return self.colors_dict.get('OK')
-        elif dt_thresh[0] <= dt < dt_thresh[1]:
-            return self.colors_dict.get('WARN')
-        return self.colors_dict.get('FAIL')
-
-    def get_temp_color(self, temp, vmin=-10, vmax=60, cmap='coolwarm'):
-        """ Get an rgba temperature value back from specified cmap, linearly interpolated between vmin and vmax. """
-        if type(temp) in [str]:
-            return self.colors_dict.get('undefined')
-        cmap = matplotlib.cm.get_cmap(cmap)
-        val = (temp - vmin) / (vmax - vmin)
-        rgba = [int(255 * c) for c in cmap(val)]
-        return rgba
-
     def set_font_bold(self, item):
         """ Set item font bold """
         f = item.font()
@@ -341,12 +309,17 @@ class MainWindow(QtWidgets.QMainWindow):
             vheader.setSectionResizeMode(index, QtWidgets.QHeaderView.Stretch)
 
     def plot_stream(self, item):
-        st_id, check = item.data(QtCore.Qt.UserRole)
+        nwst_id, check = item.data(QtCore.Qt.UserRole)
-        st = self.survBot.data.get(st_id)
+        st = self.survBot.data.get(nwst_id)
         if st:
             self.plot_widget = PlotWidget(self)
-            self.plot_widget.setWindowTitle(st_id)
+            self.plot_widget.setWindowTitle(nwst_id)
+            st = modify_stream_for_plot(st, parameters=self.parameters)
             st.plot(equal_scale=False, method='full', block=False, fig=self.plot_widget.canvas.fig)
+            # set_axis_ylabels(fig=self.plot_widget.canvas.fig, parameters=self.parameters)
+            set_axis_yticks(fig=self.plot_widget.canvas.fig, parameters=self.parameters)
+            set_axis_color(fig=self.plot_widget.canvas.fig)
+            plot_axis_thresholds(fig=self.plot_widget.canvas.fig, parameters=self.parameters)
             self.plot_widget.show()
 
     def notification(self, text):

survBotGUI.sh

@@ -1,14 +0,0 @@
-#!/bin/bash
-ulimit -s 8192
-
-#$ -l os=*stretch
-##$ -cwd
-#$ -pe smp 1
-##$ -q "*@minos15"
-
-export PYTHONPATH="$PYTHONPATH:/home/marcel/git/"
-
-source /opt/anaconda3/etc/profile.d/conda.sh
-conda activate py37
-
-python /home/marcel/git/survBot/survBotGUI.py

utils.py

@@ -0,0 +1,330 @@
+#!/usr/bin/env python
+# -*- coding: utf-8 -*-
+
+import logging
+
+import matplotlib
+import numpy as np
+import smtplib
+import os
+
+from obspy import Stream
+
+
+COLORS_DICT = {'FAIL': (195, 29, 14, 255),
+               'NO DATA': (255, 255, 125, 255),
+               'WARN': (250, 192, 63, 255),
+               'OK': (185, 245, 145, 255),
+               'undefined': (240, 240, 240, 255),
+               'disc': (126, 127, 131, 255), }
+
+
+def get_bg_color(check_key, status, dt_thresh=None, hex=False):
+    message = status.message
+    if check_key == 'last active':
+        bg_color = get_time_delay_color(message, dt_thresh)
+    elif check_key == 'temp':
+        bg_color = get_temp_color(message)
+    elif check_key == 'mass':
+        bg_color = get_mass_color(message)
+    else:
+        if status.is_warn:
+            bg_color = get_warn_color(status.count)
+        elif status.is_error:
+            if status.connection_error:
+                bg_color = get_color('disc')
+            else:
+                bg_color = get_color('FAIL')
+        else:
+            bg_color = get_color(message)
+    if not bg_color:
+        bg_color = get_color('undefined')
+
+    if hex:
+        bg_color = '#{:02x}{:02x}{:02x}'.format(*bg_color[:3])
+    return bg_color
+
+
+def get_color(key):
+    # some old GUI default colors
+    # colors_dict = {'FAIL': (255, 85, 50, 255),
+    #                'NO DATA': (255, 255, 125, 255),
+    #                'WARN': (255, 255, 80, 255),
+    #                'OK': (173, 255, 133, 255),
+    #                'undefined': (230, 230, 230, 255),
+    #                'disc': (255, 160, 40, 255),}
+    if not key in COLORS_DICT.keys():
+        key = 'undefined'
+    return COLORS_DICT.get(key)
+
+
+def get_color_mpl(key):
+    color_tup = get_color(key)
+    return np.array([color/255. for color in color_tup])
+
+
+def get_time_delay_color(dt, dt_thresh):
+    """ Set color of time delay after thresholds specified in self.dt_thresh """
+    if isinstance(dt, type(dt_thresh[0])):
+        if dt < dt_thresh[0]:
+            return get_color('OK')
+        elif dt_thresh[0] <= dt < dt_thresh[1]:
+            return get_color('WARN')
+    return get_color('FAIL')
+
+
+def get_warn_color(count, n_colors=20):
+    if count >= n_colors:
+        count = -1
+    gradient = np.linspace((240, 245, 110, 255), (250, 192, 63, 255), n_colors, dtype=int)
+    return tuple(gradient[count])
+
+
+def get_mass_color(message):
+    # can change this to something else if wanted. This way it always returns get_color (without warn count)
+    if isinstance(message, (float, int)):
+        return get_color('OK')
+    return get_color(message)
+
+
+def get_temp_color(temp, vmin=-10, vmax=60, cmap='coolwarm'):
+    """ Get an rgba temperature value back from specified cmap, linearly interpolated between vmin and vmax. """
+    if type(temp) in [str]:
+        if temp in COLORS_DICT.keys():
+            return get_color(temp)
+        return get_color('undefined')
+    cmap = matplotlib.cm.get_cmap(cmap)
+    val = (temp - vmin) / (vmax - vmin)
+    rgba = [int(255 * c) for c in cmap(val)]
+    return rgba
+
+
+def get_font_color(bg_color, hex=False):
+    if hex:
+        bg_color = matplotlib.colors.to_rgb(bg_color)
+    bg_color_hsv = matplotlib.colors.rgb_to_hsv(bg_color)
+    bg_color_hsl = hsv_to_hsl(bg_color_hsv)
+    font_color = (255, 255, 255, 255) if bg_color_hsl[2] < 0.6 else (0, 0, 0, 255)
+    if hex:
+        font_color = '#{:02x}{:02x}{:02x}'.format(*font_color[:3])
+    return font_color
+
+
+def hsv_to_hsl(hsv):
+    hue, saturation, value = hsv
+    lightness = value * (1 - saturation / 2)
+    saturation = 0 if lightness in (0, 1) else (value - lightness) / min(lightness, 1 - lightness)
+    return hue, saturation, lightness
+
+
+def modify_stream_for_plot(input_stream, parameters):
+    """ copy (if necessary) and modify stream for plotting """
+
+    # make a copy
+    st = Stream()
+
+    channels_dict = parameters.get('CHANNELS')
+
+    # iterate over all channels and put them to new stream in order
+    for index, ch_tup in enumerate(channels_dict.items()):
+        # unpack tuple from items
+        channel, channel_dict = ch_tup
+
+        # get correct channel from stream
+        st_sel = input_stream.select(channel=channel)
+        # in case there are != 1 there is ambiguity
+        if not len(st_sel) == 1:
+            continue
+
+        # make a copy to not modify original stream!
+        tr = st_sel[0].copy()
+
+        # multiply with conversion factor for unit
+        unit_factor = channel_dict.get('unit')
+        if unit_factor:
+            tr.data = tr.data * float(unit_factor)
+
+        # apply transformations if provided
+        transform = channel_dict.get('transform')
+        if transform:
+            tr.data = transform_trace(tr.data, transform)
+
+        # modify trace id to maintain plotting order
+        name = channel_dict.get('name')
+        tr.id = f'{index + 1}: {name} - {tr.id}'
+
+        st.append(tr)
+
+    return st
+
+
+def transform_trace(data, transf):
+    """
+    Transform trace with arithmetic operations in order, specified in transf
+    @param data: numpy array
+    @param transf: list of lists with arithmetic operations (e.g. [['*', '20'], ] -> multiply data by 20
+    """
+    # This looks a little bit hardcoded, however it is safer than using e.g. "eval"
+    for operator_str, val in transf:
+        if operator_str == '+':
+            data = data + val
+        elif operator_str == '-':
+            data = data - val
+        elif operator_str == '*':
+            data = data * val
+        elif operator_str == '/':
+            data = data / val
+        else:
+            raise IOError(f'Unknown arithmetic operator string: {operator_str}')
+
+    return data
+
+
+def set_axis_ylabels(fig, parameters):
+    """
+    Adds channel names to y-axis if defined in parameters.
+    """
+    names = [channel.get('name') for channel in parameters.get('CHANNELS').values()]
+    if not names: # or not len(st.traces):
+        return
+    if not len(names) == len(fig.axes):
+        logging.info('Mismatch in axis and label lengths. Not adding plot labels')
+        return
+    for channel_name, ax in zip(names, fig.axes):
+        if channel_name:
+            ax.set_ylabel(channel_name)
+
+
+def set_axis_color(fig, color='0.8', shade_color='0.95'):
+    """
+    Set all axes (frame) of figure to specific color. Shade every second axis.
+    """
+    for i, ax in enumerate(fig.axes):
+        for key in ['bottom', 'top', 'right', 'left']:
+            ax.spines[key].set_color(color)
+        if i % 2:
+            ax.set_facecolor(shade_color)
+
+
+def set_axis_yticks(fig, parameters):
+    """
+    Adds channel names to y-axis if defined in parameters.
+    """
+    ticks = [channel.get('ticks') for channel in parameters.get('CHANNELS').values()]
+    if not ticks:
+        return
+    if not len(ticks) == len(fig.axes):
+        logging.info('Mismatch in axis tick and label lengths. Not changing plot ticks.')
+        return
+    for ytick_tripple, ax in zip(ticks, fig.axes):
+        if not ytick_tripple:
+            continue
+        ymin, ymax, step = ytick_tripple
+
+        yticks = list(np.arange(ymin, ymax + step, step))
+        ax.set_yticks(yticks)
+        ax.set_ylim(ymin - 0.33 * step, ymax + 0.33 * step)
+
+
+def plot_axis_thresholds(fig, parameters):
+    """
+    Adds channel thresholds (warn, fail) to y-axis if defined in parameters.
+    """
+    logging.info('Plotting trace thresholds')
+
+    keys_colors = {'warn': dict(color=0.8 * get_color_mpl('WARN'), linestyle=(0, (5, 10)), alpha=0.5, linewidth=0.7),
+                   'fail': dict(color=0.8 * get_color_mpl('FAIL'), linestyle='solid', alpha=0.5, linewidth=0.7)}
+
+    for key, kwargs in keys_colors.items():
+        channel_threshold_list = [channel.get(key) for channel in parameters.get('CHANNELS').values()]
+        if not channel_threshold_list:
+            continue
+        plot_threshold_lines(fig, channel_threshold_list, parameters, **kwargs)
+
+
+def plot_threshold_lines(fig, channel_threshold_list, parameters, **kwargs):
+    for channel_thresholds, ax in zip(channel_threshold_list, fig.axes):
+        if channel_thresholds in ['pb_SOH2', 'pb_SOH3']:
+            annotate_voltage_states(ax, parameters, channel_thresholds)
+            channel_thresholds = get_warn_states_pbox(channel_thresholds, parameters)
+
+        if not channel_thresholds:
+            continue
+
+        if not isinstance(channel_thresholds, (list, tuple)):
+            channel_thresholds = [channel_thresholds]
+
+        for warn_thresh in channel_thresholds:
+            if isinstance(warn_thresh, str):
+                warn_thresh = parameters.get('THRESHOLDS').get(warn_thresh)
+            if isinstance(warn_thresh, (float, int)):
+                ax.axhline(warn_thresh, **kwargs)
+
+
+def get_warn_states_pbox(soh_key: str, parameters: dict) -> list:
+    pb_dict = parameters.get('POWBOX').get(soh_key)
+    if not pb_dict:
+        return []
+    return [key for key in pb_dict.keys() if key > 1]
+
+
+def annotate_voltage_states(ax, parameters, pb_key, color='0.75'):
+    for voltage, voltage_dict in parameters.get('POWBOX').get(pb_key).items():
+        if float(voltage) < 1:
+            continue
+        out_string = ''
+        for key, val in voltage_dict.items():
+            if val != 'OK':
+                if out_string:
+                    out_string += ' | '
+                out_string += f'{key}: {val}'
+
+        ax.annotate(out_string, (ax.get_xlim()[-1], voltage), color=color, fontsize='xx-small',
+                    horizontalalignment='right')
+
+def get_credential(source, param):
+    """
+    Retrieve a credential from a Docker secret or environment variable.
+    """
+    if source == 'DOCKER':
+        try:
+            with open('/run/secrets/'+param.lower(), 'r') as f:
+                return f.read().strip()
+        except FileNotFoundError as e:
+            logging.error(f'Could not read from Docker secret at /run/secrets/{param.lower()}')
+            logging.error(e)
+    elif source == 'ENV':
+        try:
+            return os.environ.get(param.upper())
+        except Exception as e:
+            logging.error(f'Could not read from environment variable {param.upper()}')
+            logging.error(e)
+    # return source if no credential was found
+    return source
+
+def connect_to_mail_server(mail_params):
+    """
+    Connect to mail server and return server object.
+    """
+    # get server from parameters
+    server = mail_params.get('mailserver')
+    # get auth_type from parameters
+    auth_type = mail_params.get('auth_type')
+    # set up connection to mail server
+    if auth_type == 'None':
+        s = smtplib.SMTP(server)
+    else:
+        # user and password from parameters, docker secret or environment variable
+        user = get_credential(mail_params.get('user'), 'mail_user')
+        password = get_credential(mail_params.get('password'), 'mail_password')
+        # create secure connection to server
+        if auth_type == 'SSL':
+            s = smtplib.SMTP_SSL(server, mail_params.get('port'))
+        elif auth_type == 'TLS':
+            s = smtplib.SMTP(server, mail_params.get('port'))
+            s.starttls()
+        else:
+            logging.error('Unknown authentication type. Mails can not be sent')
+            return
+        s.login(user, password)
+    return s

write_utils.py

@@ -0,0 +1,98 @@
+from base64 import b64encode
+from datetime import timedelta
+
+
+def _convert_to_textstring(lst):
+    return '\n'.join(lst)
+
+
+def get_html_table_title(parameters):
+    title = get_print_title_str(parameters)
+    return f'<h3>{title}</h3>\n'
+
+
+def get_html_text(text):
+    return f'<p>{text}</p>\n'
+
+
+def get_html_header(refresh_rate=10):
+    header = ['<!DOCTYPE html>',
+              '<html>',
+              '<head>',
+              '  <title>SurvBot status</title>',
+              '  <link rel="stylesheet" media="only screen and (max-width: 400px)" href="mobile.css">',
+              '  <link rel="stylesheet" media="only screen and (min-width: 401px)" href="desktop.css">',
+              f'  <meta http-equiv="refresh" content="{refresh_rate}">',
+              '  <meta charset="utf-8">',
+              '  <meta name="viewport" content="width=device-width, initial-scale=1">',
+              '</head>',
+              '<body>']
+    header = _convert_to_textstring(header)
+    return header
+
+
+def get_mail_html_header():
+    header = ['<html>',
+              '<head>',
+              '</head>',
+              '<body>']
+    header = _convert_to_textstring(header)
+    return header
+
+
+def init_html_table():
+    return '<table style="width:100%">\n'
+
+
+def finish_html_table():
+    return '</table>\n'
+
+
+def html_footer(footer_logo=None):
+    footer = ['</body>']
+    if footer_logo:
+        logo_items = [f'<div class="footer">',
+                      f'  <img style="float: right; padding: 10px;" src="{footer_logo}" height=30px>',
+                      f'</div>']
+        footer += logo_items
+    footer.append('</html>\n')
+
+    footer = _convert_to_textstring(footer)
+    return footer
+
+
+def add_html_image(img_data, img_format='png'):
+    return f"""<br>\n<img width="100%" src="data:image/{img_format};base64, {b64encode(img_data).decode('ascii')}">"""
+
+
+def get_html_link(text, link):
+    return f'<a href="{link}"> {text} </a>'
+
+
+def get_html_row(items, html_key='td'):
+    row_string = ''
+    default_space = '  '
+    row_string += default_space + '<tr>\n'
+    for item in items:
+        text = item.get('text')
+        if item.get('bold'):
+            text = '<b>' + text + '</b>'
+        if item.get('italic'):
+            text = '<i>' + text + '</i>'
+        tooltip = item.get('tooltip')
+        font_color = item.get('font_color')
+        hyperlink = item.get('hyperlink')
+        color = 'transparent' if hyperlink else item.get('color')
+        text_str = get_html_link(text, hyperlink) if hyperlink else text
+        html_class = item.get('html_class')
+        class_str = f' class="{html_class}"' if html_class else ''
+        row_string += 2 * default_space + f'<{html_key}{class_str} bgcolor="{color}" title="{tooltip}"' \
+                      + f'style="color:{font_color}">{text_str}</{html_key}>\n'
+    row_string += default_space + '</tr>\n'
+    return row_string
+
+
+def get_print_title_str(parameters):
+    timespan = parameters.get('timespan') * 24 * 3600
+    tdelta_str = str(timedelta(seconds=int(timespan))).replace(', 0:00:00', '')
+    return f'Analysis table of router quality within the last {tdelta_str}'