Compare commits
25 Commits
94cac54716
...
develop
| Author | SHA1 | Date | |
|---|---|---|---|
| f1d25a7809 | |||
| a905386ee8 | |||
| a64aeaa5ac | |||
| 206af05f26 | |||
| 0ee3a27733 | |||
| e2df92e6b4 | |||
| 8fa96d03d5 | |||
| 4e25190fbc | |||
| 8ac501e8dc | |||
| fcba73fcc5 | |||
| 16fbbde3d9 | |||
| b986da5fef | |||
| a6c1570539 | |||
| aaadff6306 | |||
| b46802a75e | |||
| 080e73c1db | |||
| 8a7e402ec5 | |||
| 3f07b7bcd0 | |||
| cf12500ec2 | |||
| 43912135e9 | |||
| 3e47f4275b | |||
| f4605b146b | |||
| 0ce41e5654 | |||
| 3dbba37fe9 | |||
| 9626a4c88d |
4
.gitignore
vendored
4
.gitignore
vendored
@@ -213,3 +213,7 @@ flycheck_*.el
|
|||||||
|
|
||||||
/__simulate_fail.json
|
/__simulate_fail.json
|
||||||
/mailing_list.yaml
|
/mailing_list.yaml
|
||||||
|
|
||||||
|
.vscode/
|
||||||
|
|
||||||
|
*.code-workspace
|
||||||
|
|||||||
40
Dockerfile
Normal file
40
Dockerfile
Normal file
@@ -0,0 +1,40 @@
|
|||||||
|
FROM python:3
|
||||||
|
|
||||||
|
# metadata
|
||||||
|
LABEL maintainer="Kasper D. Fischer <kasper.fischer@rub.de>"
|
||||||
|
LABEL version="0.2-docker"
|
||||||
|
LABEL description="Docker image for the survBot application"
|
||||||
|
|
||||||
|
# install required system packages
|
||||||
|
RUN apt update && apt install -y bind9-host iputils-ping
|
||||||
|
|
||||||
|
# create user and group and home directory
|
||||||
|
RUN groupadd -r survBot && useradd -r -g survBot survBot
|
||||||
|
RUN mkdir -p /home/survBot && chown -R survBot:survBot /home/survBot
|
||||||
|
|
||||||
|
# change working directory
|
||||||
|
WORKDIR /usr/src/app
|
||||||
|
|
||||||
|
# install required python packages
|
||||||
|
RUN --mount=type=bind,source=requirements.txt,target=/tmp/requirements.txt \
|
||||||
|
pip install --no-cache-dir --requirement /tmp/requirements.txt
|
||||||
|
|
||||||
|
# copy application files
|
||||||
|
COPY survBot.py utils.py write_utils.py LICENSE README.md ./
|
||||||
|
|
||||||
|
# copy configuration files
|
||||||
|
VOLUME /usr/src/app/conf
|
||||||
|
COPY parameters.yaml mailing_list.yam[l] simulate_fail.jso[n] conf/
|
||||||
|
RUN ln -s conf/simulate_fail.json simulate_fail.json
|
||||||
|
|
||||||
|
# copy www files
|
||||||
|
VOLUME /usr/src/app/www
|
||||||
|
COPY logo.pn[g] stylesheets/desktop.css stylesheets/mobile.css www/
|
||||||
|
RUN ln -s www/survBot_out.html www/index.html
|
||||||
|
|
||||||
|
# change ownership of working directory
|
||||||
|
RUN chown -R survBot:survBot /usr/src/app
|
||||||
|
|
||||||
|
# run the application as user survBot
|
||||||
|
USER survBot
|
||||||
|
CMD [ "python", "./survBot.py", "-html", "www", "-parfile", "conf/parameters.yaml" ]
|
||||||
55
README.md
55
README.md
@@ -3,7 +3,7 @@
|
|||||||
version: 0.2
|
version: 0.2
|
||||||
|
|
||||||
survBot is a small program used to track station quality channels of DSEBRA stations via PowBox output over SOH channels
|
survBot is a small program used to track station quality channels of DSEBRA stations via PowBox output over SOH channels
|
||||||
by analysing contents of a Seiscomp3 datapath.
|
by analyzing contents of a Seiscomp data archive.
|
||||||
|
|
||||||
## Requirements
|
## Requirements
|
||||||
|
|
||||||
@@ -40,16 +40,53 @@ The GUI can be loaded via
|
|||||||
python survBotGui.py
|
python survBotGui.py
|
||||||
```
|
```
|
||||||
|
|
||||||
|
### Docker
|
||||||
|
|
||||||
|
To run the program in a Docker container, first build the image:
|
||||||
|
|
||||||
|
```shell script
|
||||||
|
docker build -t survbot .
|
||||||
|
```
|
||||||
|
|
||||||
|
Then run the container:
|
||||||
|
|
||||||
|
```shell script
|
||||||
|
docker run -v /path/to/conf-dir:/usr/src/app/conf -v /path/to/output:/usr/src/app/www survbot
|
||||||
|
```
|
||||||
|
|
||||||
|
The directory `/path/to/conf-dir` should contain the `parameters.yaml` file, and the directory `/path/to/output` will contain the output HTML files.
|
||||||
|
|
||||||
|
### Configuration of the e-mail server settings
|
||||||
|
|
||||||
|
The e-mail server settings can be configured in the `parameters.yaml` file. The following settings are available:
|
||||||
|
|
||||||
|
* `mailserver`: the address of the mail server
|
||||||
|
* `auth_type`: the authentication type for the mail server (`None`, `SSL`, `TLS`)
|
||||||
|
* `port`: the port of the mail server
|
||||||
|
* `user`: the username for the mail server (if required)
|
||||||
|
* `password`: the password for the mail server (if required)
|
||||||
|
|
||||||
|
The `user` and `password` fields are optional, and can be left empty if the mail server does not require authentication. The `auth_type` field can be set to `None` if no authentication is required, `SSL` if the mail server requires SSL authentication, or `TLS` if the mail server requires TLS authentication. If the `user` or `password` fields are set to `Docker` ore `ENV` the program will try to read the values from the docker secrets `mail_user` and `mail_password` or environment variables `MAIL_USER` and `MAIL_PASSWORD` respectively. Docker secrets are only available in Docker Swarm mode, i.e. if the program is run as a service.
|
||||||
|
|
||||||
## Version Changes
|
## Version Changes
|
||||||
- surveillance of mass, clock and gaps
|
|
||||||
- individual mailing lists for different stations
|
### 0.2
|
||||||
- html mail with recent status information
|
|
||||||
- updated web page design
|
* surveillance of mass, clock and gaps
|
||||||
- restructured parameter file
|
* individual mailing lists for different stations
|
||||||
- recognize if PBox is disconnected
|
* html mail with recent status information
|
||||||
|
* updated web page design
|
||||||
|
* restructured parameter file
|
||||||
|
* recognize if PBox is disconnected
|
||||||
|
|
||||||
|
### 0.2-docker
|
||||||
|
|
||||||
|
* added Dockerfile for easy deployment
|
||||||
|
* added more settings for connection to a mail server
|
||||||
|
|
||||||
## Staff
|
## Staff
|
||||||
|
|
||||||
Original author: M.Paffrath (marcel.paffrath@rub.de)
|
Original author: M.Paffrath (<marcel.paffrath@rub.de>)
|
||||||
|
Contributions: Kasper D. Fischer (<kasper.fischer@rub.de>)
|
||||||
|
|
||||||
June 2023
|
Jan 2025
|
||||||
|
|||||||
@@ -1,3 +1,3 @@
|
|||||||
# survBot is a small program used to track station quality channels of DSEBRA stations via PowBox output
|
# survBot is a small program used to track station quality channels of DSEBRA stations via PowBox output
|
||||||
# over SOH channels by analysing contents of a Seiscomp3 datapath.
|
# over SOH channels by analysing contents of a Seiscomp3 datapath.
|
||||||
__version__ = "0.2"
|
__version__ = "0.2-docker"
|
||||||
|
|||||||
@@ -2,7 +2,7 @@
|
|||||||
# "mail.address@provider.com, mail.address2@provider2.com":
|
# "mail.address@provider.com, mail.address2@provider2.com":
|
||||||
# - 1Y.GR01
|
# - 1Y.GR01
|
||||||
# - 1Y.GR02
|
# - 1Y.GR02
|
||||||
# "mail.address3@provder.com":
|
# "mail.address3@provider.com":
|
||||||
# - 1Y.GR03
|
# - 1Y.GR03
|
||||||
|
|
||||||
#"kasper.fischer@rub.de":
|
#"kasper.fischer@rub.de":
|
||||||
|
|||||||
@@ -1,9 +1,9 @@
|
|||||||
# Parameters file for Surveillance Bot
|
# Parameters file for Surveillance Bot
|
||||||
datapath: "/data/SDS/" # SC3 Datapath
|
datapath: "/data/SDS/" # path to SDS data archive
|
||||||
networks: ["1Y", "HA", "MK"] # select networks, list or str
|
networks: ["1Y", "HA", "MK"] # select networks, list or str
|
||||||
stations: "*" # select stations, list or str
|
stations: "*" # select stations, list or str
|
||||||
locations: "*" # select locations, list or str
|
locations: "*" # select locations, list or str
|
||||||
stations_blacklist: ["TEST", "EREA", "DOMV", "LFKM", "GR19", "LAKA"] # exclude these stations
|
stations_blacklist: ["ATHR", "DOMV", "EREA", "GR19", "GR27", "HAVD", "LAKA", "LFKM", "TEST"] # exclude these stations
|
||||||
networks_blacklist: [] # exclude these networks
|
networks_blacklist: [] # exclude these networks
|
||||||
interval: 60 # Perform checks every x seconds
|
interval: 60 # Perform checks every x seconds
|
||||||
n_track: 360 # wait n_track * intervals before performing an action (i.e. send mail/end highlight status)
|
n_track: 360 # wait n_track * intervals before performing an action (i.e. send mail/end highlight status)
|
||||||
@@ -56,7 +56,7 @@ THRESHOLDS:
|
|||||||
#
|
#
|
||||||
# For each channel a factor 'unit' for unit conversion (e.g. to SI) can be provided, as well as a 'name'
|
# For each channel a factor 'unit' for unit conversion (e.g. to SI) can be provided, as well as a 'name'
|
||||||
# and 'ticks' [ymin, ymax, ystep] for plotting.
|
# and 'ticks' [ymin, ymax, ystep] for plotting.
|
||||||
# 'warn' and 'fail' plot horizontal lines in corresponding colors (can be str in TRESHOLDS, int/float or iterable)
|
# 'warn' and 'fail' plot horizontal lines in corresponding colors (can be str in THRESHOLDS, int/float or iterable)
|
||||||
# keyword "pb_SOH2" or "pb_SOH3" can be used to extract warning values from above POWBOX parameter definition
|
# keyword "pb_SOH2" or "pb_SOH3" can be used to extract warning values from above POWBOX parameter definition
|
||||||
#
|
#
|
||||||
# 'transform' can be provided for plotting to perform arithmetic operations in given order, e.g.:
|
# 'transform' can be provided for plotting to perform arithmetic operations in given order, e.g.:
|
||||||
@@ -117,6 +117,11 @@ data_channels: ["HHZ", "HHN", "HHE"]
|
|||||||
|
|
||||||
|
|
||||||
# ---------------------------------------- OPTIONAL PARAMETERS ---------------------------------------------------------
|
# ---------------------------------------- OPTIONAL PARAMETERS ---------------------------------------------------------
|
||||||
|
# deactivate powbox surveillance for stations (e.g. for solar powered), key: station, value: status message (abbr.)
|
||||||
|
no_pbox_stations: {'GR33': 'SOL', 'GR27B': 'DCN', 'RP01': 'noPBox', 'RP02': 'noPBox', 'RP03': 'noPBox', 'RP04': 'noPBox', 'RP05B': 'noPBox', 'RP06': 'noPBox', 'RP07': 'noPBox', 'RP08': 'noPBox'}
|
||||||
|
|
||||||
|
# deactivate mass position surveillance for stations without connected mass channels (e.g. STS-2 instruments), key: station, value: status message (abbr.)
|
||||||
|
no_mass_stations: {'BIA': 'DCN', 'KNEZ': 'DCN', 'KRUS': 'DCN', 'OHR': 'DCN', 'OTOV': 'DCN', 'SKO': 'DCN', 'STIP': 'DCN', 'VAY': 'DCN', 'ZUPA': 'DCN'}
|
||||||
|
|
||||||
# add links to html table with specified key as column and value as relative link, interpretable string parameters:
|
# add links to html table with specified key as column and value as relative link, interpretable string parameters:
|
||||||
# nw (e.g. 1Y), st (e.g. GR01A), nwst_id (e.g. 1Y.GR01A)
|
# nw (e.g. 1Y), st (e.g. GR01A), nwst_id (e.g. 1Y.GR01A)
|
||||||
@@ -130,17 +135,25 @@ add_links:
|
|||||||
add_global_links:
|
add_global_links:
|
||||||
# for example: - {"text": "our homepage", "URL": "https://www.rub.de"}
|
# for example: - {"text": "our homepage", "URL": "https://www.rub.de"}
|
||||||
- {"text": "show recent events on map",
|
- {"text": "show recent events on map",
|
||||||
"URL": "https://fdsnws.geophysik.ruhr-uni-bochum.de/map/?lat=39.5&lon=21&zoom=7&baselayer=mapnik"}
|
"URL": "https://fdsnws.geophysik.ruhr-uni-bochum.de/map/?lat=39.5&lon=21&zoom=7&baselayer=osm"}
|
||||||
|
|
||||||
# html logo at page bottom (path relative to html directory)
|
# html logo at page bottom (path relative to html directory)
|
||||||
html_logo: "figures/Logo_RUB_BLAU_rgb.png"
|
html_logo: "logo.png"
|
||||||
|
|
||||||
# E-mail notifications
|
# E-mail notifications
|
||||||
EMAIL:
|
EMAIL:
|
||||||
mailserver: "localhost"
|
# specify mail server and credentials
|
||||||
addresses: ["marcel.paffrath@rub.de", "kasper.fischer@rub.de"] # list of mail addresses for info mails
|
# port, auth_type, user and password are only required if mailserver is not set to "localhost"
|
||||||
sender: "webmaster@geophysik.ruhr-uni-bochum.de" # mail sender
|
# user and password can be set to "ENV" or "DOCKER" to read from environment variables or docker secrets
|
||||||
stations_blacklist: [] # do not send emails for specific stations
|
mailserver: "smtp.example.org" # mail server
|
||||||
networks_blacklist: [] # do not send emails for specific network
|
auth_type: "SSL" # mail authentication type, can be "SSL", "TLS" or "None"
|
||||||
|
port: 465 # mail port, default 465 for SSL, 587 for TLS
|
||||||
|
user: "DOCKER" # mail user, read from environment variable if set to "ENV" or from docker secret if set to "DOCKER"
|
||||||
|
password: "DOCKER" # mail password, read from environment variable if set to "ENV" or from docker secret if set to "DOCKER"
|
||||||
|
# specify mail recipients, sender and blacklists
|
||||||
|
addresses: ["test@example.org"] # list of mail addresses for info mails
|
||||||
|
sender: "<test@example.org>" # mail sender
|
||||||
|
stations_blacklist: [] # do not send emails for specific stations
|
||||||
|
networks_blacklist: [] # do not send emails for specific network
|
||||||
# specify recipients for single stations in a yaml: key = email-address, val = station list (e.g. [1Y.GR01, 1Y.GR02])
|
# specify recipients for single stations in a yaml: key = email-address, val = station list (e.g. [1Y.GR01, 1Y.GR02])
|
||||||
external_mail_list: "mailing_list.yaml"
|
external_mail_list: "conf/mailing_list.yaml"
|
||||||
|
|||||||
36
requirements.txt
Normal file
36
requirements.txt
Normal file
@@ -0,0 +1,36 @@
|
|||||||
|
Brotli==1.1.0
|
||||||
|
certifi==2025.1.31
|
||||||
|
cffi==1.17.1
|
||||||
|
charset-normalizer==3.4.1
|
||||||
|
contourpy==1.3.1
|
||||||
|
cycler==0.12.1
|
||||||
|
decorator==5.2.1
|
||||||
|
fonttools==4.56.0
|
||||||
|
greenlet==3.1.1
|
||||||
|
h2==4.2.0
|
||||||
|
hpack==4.1.0
|
||||||
|
hyperframe==6.1.0
|
||||||
|
idna==3.10
|
||||||
|
kiwisolver==1.4.8
|
||||||
|
lxml==5.3.1
|
||||||
|
matplotlib==3.8.4
|
||||||
|
munkres==1.1.4
|
||||||
|
numpy==1.26.4
|
||||||
|
obspy==1.4.1
|
||||||
|
packaging==24.2
|
||||||
|
pillow==11.1.0
|
||||||
|
pip==25.0.1
|
||||||
|
pycparser==2.22
|
||||||
|
pyparsing==3.2.1
|
||||||
|
PySocks==1.7.1
|
||||||
|
python-dateutil==2.9.0.post0
|
||||||
|
PyYAML==6.0.2
|
||||||
|
requests==2.32.3
|
||||||
|
scipy==1.15.2
|
||||||
|
setuptools==75.8.2
|
||||||
|
six==1.17.0
|
||||||
|
SQLAlchemy==1.4.54
|
||||||
|
unicodedata2==16.0.0
|
||||||
|
urllib3==2.3.0
|
||||||
|
wheel==0.45.1
|
||||||
|
zstandard==0.23.0
|
||||||
@@ -1,6 +1,5 @@
|
|||||||
body {
|
body {
|
||||||
background-color: #ffffff;
|
background-color: #ffffff;
|
||||||
place-items: center;
|
|
||||||
text-align: center;
|
text-align: center;
|
||||||
padding-bottom: 30px;
|
padding-bottom: 30px;
|
||||||
font-family: "Helvetica", "sans-serif";
|
font-family: "Helvetica", "sans-serif";
|
||||||
@@ -17,7 +16,7 @@ td {
|
|||||||
}
|
}
|
||||||
|
|
||||||
th {
|
th {
|
||||||
background-color: #999;
|
background-color: #17365c;
|
||||||
color: #fff;
|
color: #fff;
|
||||||
border-radius: 2px;
|
border-radius: 2px;
|
||||||
padding: 3px 1px;
|
padding: 3px 1px;
|
||||||
|
|||||||
@@ -1,6 +1,5 @@
|
|||||||
body {
|
body {
|
||||||
background-color: #ffffff;
|
background-color: #ffffff;
|
||||||
place-items: center;
|
|
||||||
text-align: center;
|
text-align: center;
|
||||||
padding-bottom: 30px;
|
padding-bottom: 30px;
|
||||||
font-family: "Helvetica", "sans-serif";
|
font-family: "Helvetica", "sans-serif";
|
||||||
@@ -17,10 +16,10 @@ td {
|
|||||||
}
|
}
|
||||||
|
|
||||||
th {
|
th {
|
||||||
background-color: #999;
|
background-color: #17365c;
|
||||||
color: #fff;
|
color: #fff;
|
||||||
border-radius: 3px;
|
border-radius: 3px;
|
||||||
padding: 10px, 2px;
|
padding: 10px 2px;
|
||||||
position: sticky;
|
position: sticky;
|
||||||
top: 0;
|
top: 0;
|
||||||
}
|
}
|
||||||
|
|||||||
@@ -1,19 +1,24 @@
|
|||||||
#!/bin/bash
|
#!/bin/bash
|
||||||
ulimit -s 8192
|
|
||||||
|
|
||||||
#$ -l low
|
#$ -l low
|
||||||
#$ -l h_vmem=5G
|
#$ -l h_vmem=2.5G
|
||||||
|
#$ -l mem=2.5G
|
||||||
|
#$ -l h_stack=INFINITY
|
||||||
#$ -cwd
|
#$ -cwd
|
||||||
#$ -pe smp 1
|
#$ -pe smp 1
|
||||||
#$ -N survBot_bg
|
#$ -binding linear:1
|
||||||
#$ -l os=*stretch
|
#$ -N survBot
|
||||||
|
#$ -o /data/www/~kasper/survBot/survBot_bg.log
|
||||||
|
#$ -e /data/www/~kasper/survBot/survBot_bg.err
|
||||||
|
#$ -m e
|
||||||
|
#$ -M kasper.fischer@rub.de
|
||||||
|
|
||||||
source /opt/anaconda3/etc/profile.d/conda.sh
|
source /opt/anaconda3/etc/profile.d/conda.sh
|
||||||
conda activate py37
|
conda activate survBot
|
||||||
|
|
||||||
# environment variables for numpy to prevent multi threading
|
# environment variables for numpy to prevent multi threading
|
||||||
export MKL_NUM_THREADS=1
|
export MKL_NUM_THREADS=1
|
||||||
export NUMEXPR_NUM_THREADS=1
|
export NUMEXPR_NUM_THREADS=1
|
||||||
export OMP_NUM_THREADS=1
|
export OMP_NUM_THREADS=1
|
||||||
|
|
||||||
python survBot.py -html '/data/www/~marcel/'
|
python survBot.py -html '/data/www/~kasper/survBot'
|
||||||
|
|||||||
51
survBot.py
51
survBot.py
@@ -1,8 +1,8 @@
|
|||||||
#!/usr/bin/env python
|
#!/usr/bin/env python
|
||||||
# -*- coding: utf-8 -*-
|
# -*- coding: utf-8 -*-
|
||||||
|
|
||||||
__version__ = '0.1'
|
__version__ = '0.3.1'
|
||||||
__author__ = 'Marcel Paffrath'
|
__author__ = ['Marcel Paffrath <marcel.paffrath@rub.de>', 'Kasper D. Fischer <kasper.fischer@rub.de>']
|
||||||
|
|
||||||
import os
|
import os
|
||||||
import io
|
import io
|
||||||
@@ -23,7 +23,8 @@ from obspy.clients.filesystem.sds import Client
|
|||||||
|
|
||||||
from write_utils import get_html_text, get_html_link, get_html_row, html_footer, get_html_header, get_print_title_str, \
|
from write_utils import get_html_text, get_html_link, get_html_row, html_footer, get_html_header, get_print_title_str, \
|
||||||
init_html_table, finish_html_table, get_mail_html_header, add_html_image
|
init_html_table, finish_html_table, get_mail_html_header, add_html_image
|
||||||
from utils import get_bg_color, get_font_color, modify_stream_for_plot, set_axis_yticks, set_axis_color, plot_axis_thresholds
|
from utils import get_bg_color, get_font_color, modify_stream_for_plot, set_axis_yticks, set_axis_color, plot_axis_thresholds, \
|
||||||
|
connect_to_mail_server
|
||||||
|
|
||||||
try:
|
try:
|
||||||
import smtplib
|
import smtplib
|
||||||
@@ -233,7 +234,7 @@ class SurveillanceBot(object):
|
|||||||
self.gaps = self.dataStream.get_gaps(min_gap=self.parameters['THRESHOLDS'].get('min_gap'))
|
self.gaps = self.dataStream.get_gaps(min_gap=self.parameters['THRESHOLDS'].get('min_gap'))
|
||||||
self.dataStream.merge()
|
self.dataStream.merge()
|
||||||
|
|
||||||
# organise data in dictionary with key for each station
|
# organize data in dictionary with key for each station
|
||||||
for trace in self.dataStream:
|
for trace in self.dataStream:
|
||||||
nwst_id = get_nwst_id(trace)
|
nwst_id = get_nwst_id(trace)
|
||||||
if not nwst_id in self.data.keys():
|
if not nwst_id in self.data.keys():
|
||||||
@@ -350,7 +351,7 @@ class SurveillanceBot(object):
|
|||||||
first_exec = False
|
first_exec = False
|
||||||
|
|
||||||
def console_print(self, itemlist, str_len=21, sep='|', seplen=3):
|
def console_print(self, itemlist, str_len=21, sep='|', seplen=3):
|
||||||
assert len(sep) <= seplen, f'Make sure seperator has less than {seplen} characters'
|
assert len(sep) <= seplen, f'Make sure separator has less than {seplen} characters'
|
||||||
sl = sep.ljust(seplen)
|
sl = sep.ljust(seplen)
|
||||||
sr = sep.rjust(seplen)
|
sr = sep.rjust(seplen)
|
||||||
string = sl
|
string = sl
|
||||||
@@ -536,11 +537,13 @@ class SurveillanceBot(object):
|
|||||||
|
|
||||||
# write footer with optional logo
|
# write footer with optional logo
|
||||||
logo_file = self.parameters.get('html_logo')
|
logo_file = self.parameters.get('html_logo')
|
||||||
|
logo_alt_text = self.parameters.get('logo_alt_text')
|
||||||
|
logo_height = self.parameters.get('logo_height')
|
||||||
if not os.path.isfile(pjoin(self.outpath_html, logo_file)):
|
if not os.path.isfile(pjoin(self.outpath_html, logo_file)):
|
||||||
logging.info(f'Specified file {logo_file} not found.')
|
logging.info(f'Specified file {logo_file} not found.')
|
||||||
logo_file = None
|
logo_file = None
|
||||||
|
|
||||||
outfile.write(html_footer(footer_logo=logo_file))
|
outfile.write(html_footer(footer_logo=logo_file, alt_text=logo_alt_text, height=logo_height))
|
||||||
|
|
||||||
except Exception as e:
|
except Exception as e:
|
||||||
logging.info(f'Could not write HTML table to {fnout}:')
|
logging.info(f'Could not write HTML table to {fnout}:')
|
||||||
@@ -550,7 +553,8 @@ class SurveillanceBot(object):
|
|||||||
|
|
||||||
def update_status_message(self):
|
def update_status_message(self):
|
||||||
timespan = timedelta(seconds=int(self.parameters.get('timespan') * 24 * 3600))
|
timespan = timedelta(seconds=int(self.parameters.get('timespan') * 24 * 3600))
|
||||||
self.status_message = f'Program starttime (UTC) {self.starttime.strftime("%Y-%m-%d %H:%M:%S")} | ' \
|
self.status_message = f' Version {__version__} | ' \
|
||||||
|
f'Program start time (UTC) {self.starttime.strftime("%Y-%m-%d %H:%M:%S")} | ' \
|
||||||
f'Current time (UTC) {UTCDateTime().strftime("%Y-%m-%d %H:%M:%S")} | ' \
|
f'Current time (UTC) {UTCDateTime().strftime("%Y-%m-%d %H:%M:%S")} | ' \
|
||||||
f'Refresh period: {self.refresh_period}s | ' \
|
f'Refresh period: {self.refresh_period}s | ' \
|
||||||
f'Showing data of last {timespan}'
|
f'Showing data of last {timespan}'
|
||||||
@@ -594,6 +598,7 @@ class StationQC(object):
|
|||||||
self.status_track = status_track
|
self.status_track = status_track
|
||||||
|
|
||||||
self.powbox_active = self.is_pbox_activated_check()
|
self.powbox_active = self.is_pbox_activated_check()
|
||||||
|
self.mass_active = self.is_mass_activated_check()
|
||||||
|
|
||||||
self.start()
|
self.start()
|
||||||
|
|
||||||
@@ -738,7 +743,6 @@ class StationQC(object):
|
|||||||
if add_addresses:
|
if add_addresses:
|
||||||
# create copy of addresses ( [:] ) to prevent changing original, general list with addresses
|
# create copy of addresses ( [:] ) to prevent changing original, general list with addresses
|
||||||
addresses = addresses[:] + list(add_addresses)
|
addresses = addresses[:] + list(add_addresses)
|
||||||
server = mail_params.get('mailserver')
|
|
||||||
if not sender or not addresses:
|
if not sender or not addresses:
|
||||||
logging.info('Mail sender or addresses not (correctly) defined. Return')
|
logging.info('Mail sender or addresses not (correctly) defined. Return')
|
||||||
return
|
return
|
||||||
@@ -757,8 +761,10 @@ class StationQC(object):
|
|||||||
html_str = self.add_html_mail_body(text)
|
html_str = self.add_html_mail_body(text)
|
||||||
msg.add_alternative(html_str, subtype='html')
|
msg.add_alternative(html_str, subtype='html')
|
||||||
|
|
||||||
# send message via SMTP server
|
# connect to server, send mail and close connection
|
||||||
s = smtplib.SMTP(server)
|
s = connect_to_mail_server(mail_params)
|
||||||
|
if not s: # if connection failed
|
||||||
|
return
|
||||||
s.send_message(msg)
|
s.send_message(msg)
|
||||||
s.quit()
|
s.quit()
|
||||||
|
|
||||||
@@ -1078,6 +1084,11 @@ class StationQC(object):
|
|||||||
""" Analyse datalogger mass channels. """
|
""" Analyse datalogger mass channels. """
|
||||||
key = 'mass'
|
key = 'mass'
|
||||||
|
|
||||||
|
# skip processing if mass is not active
|
||||||
|
if not self.mass_active:
|
||||||
|
self.set_mass_inactive_error(key)
|
||||||
|
return
|
||||||
|
|
||||||
# build stream with all channels
|
# build stream with all channels
|
||||||
st = Stream()
|
st = Stream()
|
||||||
for channel in channels:
|
for channel in channels:
|
||||||
@@ -1297,7 +1308,7 @@ class StationQC(object):
|
|||||||
|
|
||||||
# Warn in case of voltage under OK-level (1V)
|
# Warn in case of voltage under OK-level (1V)
|
||||||
if len(under) > 0:
|
if len(under) > 0:
|
||||||
# try calculate number of occurences from gaps between indices
|
# try calculate number of occurrences from gaps between indices
|
||||||
n_occurrences = len(np.where(np.diff(under) > 1)[0]) + 1
|
n_occurrences = len(np.where(np.diff(under) > 1)[0]) + 1
|
||||||
voltage_dict[-1] = under
|
voltage_dict[-1] = under
|
||||||
self.status_other(detailed_message=f'Trace {trace.get_id()}: '
|
self.status_other(detailed_message=f'Trace {trace.get_id()}: '
|
||||||
@@ -1339,6 +1350,12 @@ class StationQC(object):
|
|||||||
msg = self.parameters.get('no_pbox_stations')[self.station]
|
msg = self.parameters.get('no_pbox_stations')[self.station]
|
||||||
self.error(key, detailed_message=f'PowBox not connected', disc=msg)
|
self.error(key, detailed_message=f'PowBox not connected', disc=msg)
|
||||||
|
|
||||||
|
def is_mass_activated_check(self):
|
||||||
|
return self.station not in self.parameters.get('no_mass_stations', [])
|
||||||
|
|
||||||
|
def set_mass_inactive_error(self, key):
|
||||||
|
msg = self.parameters.get('no_mass_stations')[self.station]
|
||||||
|
self.error(key, detailed_message=f'Mass channels not connected', disc=msg)
|
||||||
|
|
||||||
class Status(object):
|
class Status(object):
|
||||||
""" Basic Status class. All status classes are derived from this class."""
|
""" Basic Status class. All status classes are derived from this class."""
|
||||||
@@ -1393,15 +1410,15 @@ class StatusOK(Status):
|
|||||||
|
|
||||||
|
|
||||||
class StatusWarn(Status):
|
class StatusWarn(Status):
|
||||||
def __init__(self, message='WARN', count=1, last_occurence=None, detailed_messages=None, show_count=False):
|
def __init__(self, message='WARN', count=1, last_occurrence=None, detailed_messages=None, show_count=False):
|
||||||
super(StatusWarn, self).__init__(message=message, count=count, last_occurrence=last_occurence,
|
super(StatusWarn, self).__init__(message=message, count=count, last_occurrence=last_occurrence,
|
||||||
detailed_messages=detailed_messages, show_count=show_count)
|
detailed_messages=detailed_messages, show_count=show_count)
|
||||||
self.set_warn()
|
self.set_warn()
|
||||||
|
|
||||||
|
|
||||||
class StatusError(Status):
|
class StatusError(Status):
|
||||||
def __init__(self, message='FAIL', count=1, last_occurence=None, detailed_messages=None, show_count=False):
|
def __init__(self, message='FAIL', count=1, last_occurrence=None, detailed_messages=None, show_count=False):
|
||||||
super(StatusError, self).__init__(message=message, count=count, last_occurrence=last_occurence,
|
super(StatusError, self).__init__(message=message, count=count, last_occurrence=last_occurrence,
|
||||||
detailed_messages=detailed_messages, show_count=show_count)
|
detailed_messages=detailed_messages, show_count=show_count)
|
||||||
self.set_error()
|
self.set_error()
|
||||||
self.default_message = message
|
self.default_message = message
|
||||||
@@ -1418,8 +1435,8 @@ class StatusError(Status):
|
|||||||
|
|
||||||
|
|
||||||
class StatusOther(Status):
|
class StatusOther(Status):
|
||||||
def __init__(self, messages=None, count=1, last_occurence=None, detailed_messages=None):
|
def __init__(self, messages=None, count=1, last_occurrence=None, detailed_messages=None):
|
||||||
super(StatusOther, self).__init__(count=count, last_occurrence=last_occurence,
|
super(StatusOther, self).__init__(count=count, last_occurrence=last_occurrence,
|
||||||
detailed_messages=detailed_messages)
|
detailed_messages=detailed_messages)
|
||||||
if messages is None:
|
if messages is None:
|
||||||
messages = []
|
messages = []
|
||||||
|
|||||||
51
utils.py
51
utils.py
@@ -5,6 +5,8 @@ import logging
|
|||||||
|
|
||||||
import matplotlib
|
import matplotlib
|
||||||
import numpy as np
|
import numpy as np
|
||||||
|
import smtplib
|
||||||
|
import os
|
||||||
|
|
||||||
from obspy import Stream
|
from obspy import Stream
|
||||||
|
|
||||||
@@ -173,7 +175,7 @@ def transform_trace(data, transf):
|
|||||||
elif operator_str == '/':
|
elif operator_str == '/':
|
||||||
data = data / val
|
data = data / val
|
||||||
else:
|
else:
|
||||||
raise IOError(f'Unknown arithmethic operator string: {operator_str}')
|
raise IOError(f'Unknown arithmetic operator string: {operator_str}')
|
||||||
|
|
||||||
return data
|
return data
|
||||||
|
|
||||||
@@ -279,3 +281,50 @@ def annotate_voltage_states(ax, parameters, pb_key, color='0.75'):
|
|||||||
|
|
||||||
ax.annotate(out_string, (ax.get_xlim()[-1], voltage), color=color, fontsize='xx-small',
|
ax.annotate(out_string, (ax.get_xlim()[-1], voltage), color=color, fontsize='xx-small',
|
||||||
horizontalalignment='right')
|
horizontalalignment='right')
|
||||||
|
|
||||||
|
def get_credential(source, param):
|
||||||
|
"""
|
||||||
|
Retrieve a credential from a Docker secret or environment variable.
|
||||||
|
"""
|
||||||
|
if source == 'DOCKER':
|
||||||
|
try:
|
||||||
|
with open('/run/secrets/'+param.lower(), 'r') as f:
|
||||||
|
return f.read().strip()
|
||||||
|
except FileNotFoundError as e:
|
||||||
|
logging.error(f'Could not read from Docker secret at /run/secrets/{param.lower()}')
|
||||||
|
logging.error(e)
|
||||||
|
elif source == 'ENV':
|
||||||
|
try:
|
||||||
|
return os.environ.get(param.upper())
|
||||||
|
except Exception as e:
|
||||||
|
logging.error(f'Could not read from environment variable {param.upper()}')
|
||||||
|
logging.error(e)
|
||||||
|
# return source if no credential was found
|
||||||
|
return source
|
||||||
|
|
||||||
|
def connect_to_mail_server(mail_params):
|
||||||
|
"""
|
||||||
|
Connect to mail server and return server object.
|
||||||
|
"""
|
||||||
|
# get server from parameters
|
||||||
|
server = mail_params.get('mailserver')
|
||||||
|
# get auth_type from parameters
|
||||||
|
auth_type = mail_params.get('auth_type')
|
||||||
|
# set up connection to mail server
|
||||||
|
if auth_type == 'None':
|
||||||
|
s = smtplib.SMTP(server)
|
||||||
|
else:
|
||||||
|
# user and password from parameters, docker secret or environment variable
|
||||||
|
user = get_credential(mail_params.get('user'), 'mail_user')
|
||||||
|
password = get_credential(mail_params.get('password'), 'mail_password')
|
||||||
|
# create secure connection to server
|
||||||
|
if auth_type == 'SSL':
|
||||||
|
s = smtplib.SMTP_SSL(server, mail_params.get('port'))
|
||||||
|
elif auth_type == 'TLS':
|
||||||
|
s = smtplib.SMTP(server, mail_params.get('port'))
|
||||||
|
s.starttls()
|
||||||
|
else:
|
||||||
|
logging.error('Unknown authentication type. Mails can not be sent')
|
||||||
|
return
|
||||||
|
s.login(user, password)
|
||||||
|
return s
|
||||||
|
|||||||
@@ -17,15 +17,16 @@ def get_html_text(text):
|
|||||||
|
|
||||||
def get_html_header(refresh_rate=10):
|
def get_html_header(refresh_rate=10):
|
||||||
header = ['<!DOCTYPE html>',
|
header = ['<!DOCTYPE html>',
|
||||||
'<html>',
|
'<html lang="en">',
|
||||||
'<head>',
|
'<head>',
|
||||||
' <link rel="stylesheet" media="only screen and (max-width: 400px)" href="mobile.css" />',
|
' <title>SurvBot status</title>',
|
||||||
' <link rel="stylesheet" media="only screen and (min-width: 401px)" href="desktop.css" />',
|
' <link rel="stylesheet" media="only screen and (max-width: 400px)" href="mobile.css">',
|
||||||
|
' <link rel="stylesheet" media="only screen and (min-width: 401px)" href="desktop.css">',
|
||||||
|
f' <meta http-equiv="refresh" content="{refresh_rate}">',
|
||||||
|
' <meta charset="utf-8">',
|
||||||
|
' <meta name="viewport" content="width=device-width, initial-scale=1">',
|
||||||
'</head>',
|
'</head>',
|
||||||
f'<meta http-equiv="refresh" content="{refresh_rate}" >',
|
'<body>\n']
|
||||||
'<meta charset="utf-8">',
|
|
||||||
'<meta name="viewport" content="width=device-width, initial-scale=1">',
|
|
||||||
'<body>']
|
|
||||||
header = _convert_to_textstring(header)
|
header = _convert_to_textstring(header)
|
||||||
return header
|
return header
|
||||||
|
|
||||||
@@ -47,14 +48,15 @@ def finish_html_table():
|
|||||||
return '</table>\n'
|
return '</table>\n'
|
||||||
|
|
||||||
|
|
||||||
def html_footer(footer_logo=None):
|
def html_footer(footer_logo=None, alt_text='Logo', height=30):
|
||||||
footer = ['</body>']
|
|
||||||
if footer_logo:
|
if footer_logo:
|
||||||
logo_items = [f'<div class="footer">',
|
footer = [f'<div class="footer">',
|
||||||
f' <img style="float: right; padding: 10px;" src="{footer_logo}" height=30px>',
|
f' <img style="float: right; padding: 10px;" src="{footer_logo}" height={height} alt="{alt_text}">',
|
||||||
f'</div>']
|
f'</div>']
|
||||||
footer += logo_items
|
else:
|
||||||
footer.append('</html>\n')
|
footer = []
|
||||||
|
footer.append('</body>')
|
||||||
|
footer.append('</html>')
|
||||||
|
|
||||||
footer = _convert_to_textstring(footer)
|
footer = _convert_to_textstring(footer)
|
||||||
return footer
|
return footer
|
||||||
@@ -85,8 +87,14 @@ def get_html_row(items, html_key='td'):
|
|||||||
text_str = get_html_link(text, hyperlink) if hyperlink else text
|
text_str = get_html_link(text, hyperlink) if hyperlink else text
|
||||||
html_class = item.get('html_class')
|
html_class = item.get('html_class')
|
||||||
class_str = f' class="{html_class}"' if html_class else ''
|
class_str = f' class="{html_class}"' if html_class else ''
|
||||||
row_string += 2 * default_space + f'<{html_key}{class_str} bgcolor="{color}" title="{tooltip}"' \
|
row_string += 2 * default_space + f'<{html_key}{class_str} '
|
||||||
+ f'style="color:{font_color}"> {text_str}</{html_key}>\n'
|
row_string += f'title="{tooltip}" ' if tooltip else ''
|
||||||
|
row_string += 'style="' if color or font_color else ''
|
||||||
|
row_string += f'background-color: {color};' if color else 'style="'
|
||||||
|
row_string += ' ' if font_color else ''
|
||||||
|
row_string += f'color: {font_color};' if font_color else ''
|
||||||
|
row_string += '" ' if color or font_color else ''
|
||||||
|
row_string += f'>{text_str}</{html_key}>\n'
|
||||||
row_string += default_space + '</tr>\n'
|
row_string += default_space + '</tr>\n'
|
||||||
return row_string
|
return row_string
|
||||||
|
|
||||||
|
|||||||
Reference in New Issue
Block a user