32 Commits

Author SHA1 Message Date
47c3fbabf0 Merge pull request 'release/1.0' (#3) from release/1.0 into main
Reviewed-on: #3
2022-12-20 12:05:45 +01:00
d397ce377e [update] add network/station blacklists for mail functionality 2022-12-20 10:23:25 +01:00
d764c5c256 Merge remote-tracking branch 'origin/develop' into develop 2022-12-08 13:23:45 +01:00
a56781dca3 [minor] renamed and added stylesheets to git repository 2022-12-08 13:23:21 +01:00
fc64239c88 don't analysis values of data gaps when cheching for thresholds.
use stream.merge(fill_value=np.nan)
2022-12-06 16:34:08 +01:00
f0ae7da2be [update] add min_sample parameter, which controls ne number of samples that have to be of a specific voltage state before counting them as fail-state 2022-12-06 15:31:09 +01:00
a30cd8c0d4 [update] add trace axis ticking/min-max 2022-12-02 11:15:37 +01:00
a3378874fa [minor] parameter change 2022-12-02 11:15:01 +01:00
d21fb0ca3b [minor] parameter change 2022-11-29 10:42:29 +01:00
19b8df8f7d [minor] add html class for mobile (WIP) 2022-11-29 10:42:15 +01:00
6fc1e073c0 [bugfix] compared timedelta with int 2022-11-24 10:13:21 +01:00
56351ee700 [update] send message (mail) on station timeout 2022-11-23 11:52:26 +01:00
f45c5b20c5 [bugfix] set error state to active until mail is sent 2022-11-23 11:29:05 +01:00
7a2b7add04 [minor] parameter changes 2022-11-23 11:28:36 +01:00
ae0c2ef4e9 [minor] track activity status, modify html output for stylesheet 2022-11-22 18:06:25 +01:00
d35c176aab [update] add channel naming for plots 2022-11-22 15:51:15 +01:00
9444405453 [minor] reformat yaml file 2022-11-22 13:34:58 +01:00
a6d59c8c71 [update] added possibility to modify voltage level in plots from settings in parameters.yaml 2022-11-22 12:07:16 +01:00
3fe5fc48d1 [minor] re-ordered parameters 2022-11-21 10:36:46 +01:00
7da3db260a [update] error tracking + send email functionality 2022-11-17 09:52:04 +01:00
2c1e923920 [minor] add stylesheet to html header 2022-11-16 11:24:48 +01:00
8e42ac11c7 [update] complete rework of status handling (added Warn/Error classes etc.) 2022-11-15 17:19:39 +01:00
4d4324a1e9 [refactor] st_id -> nwst_id 2022-11-15 13:48:56 +01:00
4ba9c20d0f [update] add possibility to add columns with links to other web pages (e.g. seedlink monitor) 2022-11-15 13:44:19 +01:00
cd6b40688b [update] re-read data daily, add daily overlap, add value -1 for voltage lower 1V (e.g. pbox not connected) 2022-11-14 22:31:22 +01:00
04371f92c5 [minor] update README, add check for output dir existence 2022-11-09 16:53:43 +01:00
c723a32274 [update] html writer creates hourly figures of all stations 2022-11-09 16:29:01 +01:00
18dac062ef [bugfix] too many under 1V warnings appearing, moved them to "other" 2022-11-09 09:49:27 +01:00
6791a729ed [bugfix] corrected warning number for under/overvoltage 2022-11-08 17:11:37 +01:00
c3f9ad0fd9 [update] moved html writing to survBot.py so that no GUI is needed 2022-11-08 16:45:21 +01:00
abc201c673 [update] append warnings to existing ones 2022-11-08 14:23:56 +01:00
68c6df72c9 [update] html output from GUI in background possible but maybe not optimal (needs display to work in bg) 2022-11-08 13:35:46 +01:00
10 changed files with 1020 additions and 295 deletions

View File

@@ -26,12 +26,14 @@ to use the GUI:
Configurations of *datapath*, *networks*, *stations* etc. can be done in the **parameters.yaml** input file. Configurations of *datapath*, *networks*, *stations* etc. can be done in the **parameters.yaml** input file.
The main program is executed by entering The main program with html output is executed by entering
```shell script ```shell script
python survBot.py python survBot.py -html path_for_html_output
``` ```
There are example stylesheets in the folder *stylesheets* that can be copied into the path_for_html_output if desired.
The GUI can be loaded via The GUI can be loaded via
```shell script ```shell script

View File

@@ -1,27 +1,34 @@
# Parameters file for Surveillance Bot # Parameters file for Surveillance Bot
datapath: '/data/SDS/' # SC3 Datapath datapath: "/data/SDS/" # SC3 Datapath
outpath_html: '/home/marcel/tmp/survBot_out.html' # output of HTML table networks: ["1Y", "HA"] # select networks, list or str
networks: ['1Y', 'HA'] stations: "*" # select stations, list or str
stations: '*' locations: "*" # select locations, list or str
locations: '*' channels: ["EX1", "EX2", "EX3", "VEI"] # Specify SOH channels, currently supported EX[1-3] and VEI
channels: ['EX1', 'EX2', 'EX3', 'VEI'] # Specify SOH channels, currently supported EX[1-3] and VEI stations_blacklist: ["TEST", "EREA"] # exclude these stations
stations_blacklist: ['TEST', 'EREA'] networks_blacklist: [] # exclude these networks
networks_blacklist: [] interval: 60 # Perform checks every x seconds
interval: 20 # Perform checks every x seconds n_track: 300 # wait n_track * intervals before performing an action (i.e. send mail/end highlight status)
timespan: 7 # Check data of the recent x days timespan: 7 # Check data of the recent x days
verbosity: 0 verbosity: 0 # verbosity flag
track_changes: True # tracks all changes since GUI startup by text highlighting (GUI only) track_changes: True # tracks all changes since GUI startup by text highlighting (GUI only)
warn_count: False # show number of warnings and errors in table
min_sample: 3 # minimum samples for raising Warn/FAIL
dt_thresh: [300, 1800] # threshold (s) for timing delay colourisation (yellow/red)
html_figures: True # Create html figure directory and links
reread_parameters: True # reread parameters file (change parameters on runtime, not for itself/GUI refresh/datapath)
POWBOX: POWBOX:
pb_ok: 1 # Voltage for PowBox OK pb_ok: 1 # Voltage for PowBox OK
pb_SOH2: # PowBox channel 2 voltage translations pb_SOH2: # PowBox channel 2 voltage translations
1: {"230V": 'OK', "12V": "OK"} -1: {"230V": "PBox under 1V", "12V": "PBox under 1V"}
1: {"230V": "OK", "12V": "OK"}
2: {"230V": "OFF", "12V": "OK"} 2: {"230V": "OFF", "12V": "OK"}
3: {"230V": "OK", "12V": "overvoltage"} 3: {"230V": "OK", "12V": "overvoltage"}
4: {"230V": "OK", "12V": "undervoltage"} 4: {"230V": "OK", "12V": "undervoltage"}
4.5: {"230V": "OFF", "12V": "overvoltage"} 4.5: {"230V": "OFF", "12V": "overvoltage"}
5: {"230V": "OFF", "12V": "undervoltage"} 5: {"230V": "OFF", "12V": "undervoltage"}
pb_SOH3: # PowBox channel 3 voltage translations pb_SOH3: # PowBox channel 3 voltage translations
-1: {"router": "PBox under 1V", "charger": "PBox under 1V"}
1: {"router": "OK", "charger": "OK"} 1: {"router": "OK", "charger": "OK"}
2: {"router": "OK", "charger": "0 < resets < 3"} 2: {"router": "OK", "charger": "0 < resets < 3"}
2.5: {"router": "OK", "charger": "locked"} 2.5: {"router": "OK", "charger": "locked"}
@@ -29,10 +36,49 @@ POWBOX:
4: {"router": "FAIL", "charger": "0 < resets < 3"} 4: {"router": "FAIL", "charger": "0 < resets < 3"}
5: {"router": "FAIL", "charger": "locked"} 5: {"router": "FAIL", "charger": "locked"}
# Thresholds for program warnings/voltage classifications
THRESHOLDS: THRESHOLDS:
pb_thresh: 0.2 # Threshold for PowBox Voltage check +/- (V) pb_thresh: 0.2 # Threshold for PowBox Voltage check +/- (V)
max_temp: 50 # max temperature for temperature warning max_temp: 50 # max temperature for temperature warning
low_volt: 12 # min voltage for low voltage warning low_volt: 12 # min voltage for low voltage warning
high_volt: 14.8 # max voltage for over voltage warning high_volt: 14.8 # max voltage for over voltage warning
unclassified: 5 # min voltage samples not classified for warning unclassified: 5 # min voltage samples not classified for warning
# ---------------------------------------- OPTIONAL PARAMETERS ---------------------------------------------------------
# add links to html table with specified key as column and value as relative link, interpretable string parameters:
# nw (e.g. 1Y), st (e.g. GR01A), nwst_id (e.g. 1Y.GR01A)
add_links:
# for example: slmon: {"URL": "path/{nw}_{st}.html", "text": "link"}
slmon: {"URL": "../slmon/{nw}_{st}.html", "text": "show"}
24h-plot: {"URL": "../scheli/{nw}/{st}.png", "text": "plot"}
# E-mail notifications
EMAIL:
mailserver: "localhost"
addresses: ["marcel.paffrath@rub.de", "kasper.fischer@rub.de"] # list of mail addresses for info mails
sender: "webmaster@geophysik.ruhr-uni-bochum.de" # mail sender
stations_blacklist: ['GR33'] # do not send emails for specific stations
networks_blacklist: [] # do not send emails for specific network
# names for plotting of the above defined parameter "channels" in the same order
channel_names: ["Temperature (°C)", "230V/12V Status (V)", "Router/Charger State (V)", "Logger Voltage (V)"]
# specify y-ticks (and ylims) giving, (ymin, ymax, step) for each of the above channels (0: default)
CHANNEL_TICKS:
- [-10, 50, 10]
- [1, 5, 1]
- [1, 5, 1]
- [9, 15, 1]
# Factor for channel to SI-units (for plotting)
CHANNEL_UNITS:
EX1: 1e-6
EX2: 1e-6
EX3: 1e-6
VEI: 1e-3
# Transform channel for plotting, perform arithmetic operations in given order, e.g.: PBox EX1 V to deg C: 20 * x -20
CHANNEL_TRANSFORM:
EX1:
- ["*", 20]
- ["-", 20]

39
stylesheets/desktop.css Normal file
View File

@@ -0,0 +1,39 @@
body {
background-color: #ffffff;
place-items: center;
text-align: center;
}
td {
border-radius: 4px;
padding: 0px;
}
th {
background-color: #999;
border-radius: 4px;
padding: 3px 1px;
}
a:link, a:visited {
background-color: #ccc;
color: #000;
text-decoration: none;
display: block;
border-radius: 4px;
border: 1px solid #bbb;
}
a:hover {
background-color: #aaa;
display: block;
}
.blink-bg {
animation: blinkingBackground 2s infinite;
}
@keyframes blinkingBackground{
0% { background-color: #ffcc00;}
50% { background-color: #ff3200;}
100% { background-color: #ffcc00;}
}

43
stylesheets/mobile.css Normal file
View File

@@ -0,0 +1,43 @@
body {
background-color: #ffffff;
place-items: center;
text-align: center;
}
td {
border-radius: 4px;
padding: 10px 2px;
}
th {
background-color: #999;
border-radius: 4px;
padding: 10px, 2px;
}
a:link {
background-color: #ccc;
color: #000;
text-decoration: none;
display: block;
border-radius: 4px;
border: 1px solid #bbb;
}
a:hover {
background-color: #aaa;
display: block;
}
.hidden-mobile {
display: none;
}
.blink-bg {
animation: blinkingBackground 2s infinite;
}
@keyframes blinkingBackground{
0% { background-color: #ffee00;}
50% { background-color: #ff3200;}
100% { background-color: #ffee00;}
}

15
submit_bot.sh Normal file → Executable file
View File

@@ -2,15 +2,18 @@
ulimit -s 8192 ulimit -s 8192
#$ -l low #$ -l low
#$ -l os=*stretch #$ -l h_vmem=5G
#$ -cwd #$ -cwd
#$ -pe smp 1 #$ -pe smp 1
##$ -q "*@minos15" #$ -N survBot_bg
#$ -l os=*stretch
export PYTHONPATH="$PYTHONPATH:/home/marcel/git/"
export PYTHONPATH="$PYTHONPATH:/home/marcel/git/code_base/"
source /opt/anaconda3/etc/profile.d/conda.sh source /opt/anaconda3/etc/profile.d/conda.sh
conda activate py37 conda activate py37
python survBot.py # environment variables for numpy to prevent multi threading
export MKL_NUM_THREADS=1
export NUMEXPR_NUM_THREADS=1
export OMP_NUM_THREADS=1
python survBot.py -html '/data/www/~marcel/'

File diff suppressed because it is too large Load Diff

View File

@@ -1,4 +1,5 @@
#!/usr/bin/env python #!/usr/bin/env python
# -*- coding: utf-8 -*-
""" """
GUI overlay for the main survBot to show quality control of different stations specified in parameters.yaml file. GUI overlay for the main survBot to show quality control of different stations specified in parameters.yaml file.
""" """
@@ -9,7 +10,6 @@ __author__ = 'Marcel Paffrath'
import os import os
import sys import sys
import traceback import traceback
import argparse
try: try:
from PySide2 import QtGui, QtCore, QtWidgets from PySide2 import QtGui, QtCore, QtWidgets
@@ -22,7 +22,6 @@ except ImportError:
except ImportError: except ImportError:
raise ImportError('Could import neither of PySide2, PySide6 or PyQt5') raise ImportError('Could import neither of PySide2, PySide6 or PyQt5')
import matplotlib
from matplotlib.figure import Figure from matplotlib.figure import Figure
if QtGui.__package__ in ['PySide2', 'PyQt5', 'PySide6']: if QtGui.__package__ in ['PySide2', 'PyQt5', 'PySide6']:
@@ -35,6 +34,7 @@ from obspy import UTCDateTime
from survBot import SurveillanceBot from survBot import SurveillanceBot
from write_utils import * from write_utils import *
from utils import get_bg_color, modify_stream_for_plot, trace_ylabels, trace_yticks
try: try:
from rest_api.utils import get_station_iccid from rest_api.utils import get_station_iccid
@@ -77,24 +77,14 @@ class Thread(QtCore.QThread):
class MainWindow(QtWidgets.QMainWindow): class MainWindow(QtWidgets.QMainWindow):
def __init__(self, parameters='parameters.yaml', dt_thresh=(300, 1800)): def __init__(self, parameters='parameters.yaml'):
""" """
Main window of survBot GUI. Main window of survBot GUI.
:param parameters: Parameters dictionary file (yaml format) :param parameters: Parameters dictionary file (yaml format)
:param dt_thresh: threshold for timing delay colourisation (yellow/red)
""" """
super(MainWindow, self).__init__() super(MainWindow, self).__init__()
# some GUI default colors
self.colors_dict = {'FAIL': (255, 50, 0, 255),
'NO DATA': (255, 255, 125, 255),
'WARN': (255, 255, 125, 255),
'WARNX': lambda x: (min([255, 200 + x**2]), 255, 125, 255),
'OK': (125, 255, 125, 255),
'undefined': (230, 230, 230, 255)}
# init some attributes # init some attributes
self.dt_thresh = dt_thresh
self.last_mouse_loc = None self.last_mouse_loc = None
self.status_message = '' self.status_message = ''
self.starttime = UTCDateTime() self.starttime = UTCDateTime()
@@ -109,6 +99,7 @@ class MainWindow(QtWidgets.QMainWindow):
self.survBot = SurveillanceBot(parameter_path=parameters) self.survBot = SurveillanceBot(parameter_path=parameters)
self.parameters = self.survBot.parameters self.parameters = self.survBot.parameters
self.refresh_period = self.parameters.get('interval') self.refresh_period = self.parameters.get('interval')
self.dt_thresh = [int(val) for val in self.parameters.get('dt_thresh')]
# create thread that is used to update # create thread that is used to update
self.thread = Thread(parent=self, runnable=self.survBot.execute_qc) self.thread = Thread(parent=self, runnable=self.survBot.execute_qc)
@@ -138,10 +129,10 @@ class MainWindow(QtWidgets.QMainWindow):
self.table.setRowCount(len(station_list)) self.table.setRowCount(len(station_list))
self.table.setHorizontalHeaderLabels(keys) self.table.setHorizontalHeaderLabels(keys)
for index, st_id in enumerate(station_list): for index, nwst_id in enumerate(station_list):
item = QtWidgets.QTableWidgetItem() item = QtWidgets.QTableWidgetItem()
item.setText(str(st_id.rstrip('.'))) item.setText(str(nwst_id.rstrip('.')))
item.setData(QtCore.Qt.UserRole, st_id) item.setData(QtCore.Qt.UserRole, nwst_id)
self.table.setVerticalHeaderItem(index, item) self.table.setVerticalHeaderItem(index, item)
self.main_layout.addWidget(self.table) self.main_layout.addWidget(self.table)
@@ -183,78 +174,43 @@ class MainWindow(QtWidgets.QMainWindow):
self.last_mouse_loc = event.pos() self.last_mouse_loc = event.pos()
return super(QtWidgets.QMainWindow, self).eventFilter(object, event) return super(QtWidgets.QMainWindow, self).eventFilter(object, event)
def write_html_table(self):
fnout = self.parameters.get('outpath_html')
if not fnout:
return
try:
with open(fnout, 'w') as outfile:
write_html_header(outfile)
#write_html_table_title(outfile, self.parameters)
init_html_table(outfile)
nrows = self.table.rowCount()
ncolumns = self.table.columnCount()
# add header item 0 fix default black bg color for headers
station_header = QtWidgets.QTableWidgetItem(text='Station')
station_header.setText('Station')
header_items = [station_header]
for column in range(ncolumns):
hheader = self.table.horizontalHeaderItem(column)
header_items.append(hheader)
write_html_row(outfile, header_items, html_key='th')
for row in range(nrows):
vheader = self.table.verticalHeaderItem(row)
col_items = [vheader]
for column in range(ncolumns):
col_items.append(self.table.item(row, column))
write_html_row(outfile, col_items)
finish_html_table(outfile)
write_html_text(outfile, self.status_message)
write_html_footer(outfile)
except Exception as e:
print(f'Could not write HTML table to {fnout}:')
print(e)
def sms_context_menu(self, row_ind): def sms_context_menu(self, row_ind):
""" Open a context menu when left-clicking vertical header item """ """ Open a context menu when left-clicking vertical header item """
header_item = self.table.verticalHeaderItem(row_ind) header_item = self.table.verticalHeaderItem(row_ind)
if not header_item: if not header_item:
return return
st_id = header_item.data(QtCore.Qt.UserRole) nwst_id = header_item.data(QtCore.Qt.UserRole)
context_menu = QtWidgets.QMenu() context_menu = QtWidgets.QMenu()
read_sms = context_menu.addAction('Get last SMS') read_sms = context_menu.addAction('Get last SMS')
send_sms = context_menu.addAction('Send SMS') send_sms = context_menu.addAction('Send SMS')
action = context_menu.exec_(self.mapToGlobal(self.last_mouse_loc)) action = context_menu.exec_(self.mapToGlobal(self.last_mouse_loc))
if action == read_sms: if action == read_sms:
self.read_sms(st_id) self.read_sms(nwst_id)
elif action == send_sms: elif action == send_sms:
self.send_sms(st_id) self.send_sms(nwst_id)
def read_sms(self, st_id): def read_sms(self, nwst_id):
""" Read recent SMS over rest_api using whereversim portal """ """ Read recent SMS over rest_api using whereversim portal """
station = st_id.split('.')[1] station = nwst_id.split('.')[1]
iccid = get_station_iccid(station) iccid = get_station_iccid(station)
if not iccid: if not iccid:
print('Could not find iccid for station', st_id) print('Could not find iccid for station', nwst_id)
return return
sms_widget = ReadSMSWidget(parent=self, iccid=iccid) sms_widget = ReadSMSWidget(parent=self, iccid=iccid)
sms_widget.setWindowTitle(f'Recent SMS of station: {st_id}') sms_widget.setWindowTitle(f'Recent SMS of station: {nwst_id}')
if sms_widget.data: if sms_widget.data:
sms_widget.show() sms_widget.show()
else: else:
self.notification('No recent messages found.') self.notification('No recent messages found.')
def send_sms(self, st_id): def send_sms(self, nwst_id):
""" Send SMS over rest_api using whereversim portal """ """ Send SMS over rest_api using whereversim portal """
station = st_id.split('.')[1] station = nwst_id.split('.')[1]
iccid = get_station_iccid(station) iccid = get_station_iccid(station)
sms_widget = SendSMSWidget(parent=self, iccid=iccid) sms_widget = SendSMSWidget(parent=self, iccid=iccid)
sms_widget.setWindowTitle(f'Send SMS to station: {st_id}') sms_widget.setWindowTitle(f'Send SMS to station: {nwst_id}')
sms_widget.show() sms_widget.show()
def set_clear_on_refresh(self): def set_clear_on_refresh(self):
@@ -262,39 +218,30 @@ class MainWindow(QtWidgets.QMainWindow):
def fill_status_bar(self): def fill_status_bar(self):
""" Set status bar text """ """ Set status bar text """
timespan = timedelta(seconds=int(self.parameters.get('timespan') * 24 * 3600)) self.status_message = self.survBot.status_message
self.status_message = f'Program starttime (UTC) {self.starttime.strftime("%Y-%m-%d %H:%M:%S")} | ' \
f'Current time (UTC) {UTCDateTime().strftime("%Y-%m-%d %H:%M:%S")} | ' \
f'Refresh period: {self.refresh_period}s | '\
f'Showing data of last {timespan}'
status_bar = self.statusBar() status_bar = self.statusBar()
status_bar.showMessage(self.status_message) status_bar.showMessage(self.status_message)
def fill_table(self): def fill_table(self):
""" Fills the table with most recent information. Executed after execute_qc thread is done or on refresh. """ """ Fills the table with most recent information. Executed after execute_qc thread is done or on refresh. """
# fill status bar first with new time
self.fill_status_bar()
for col_ind, check_key in enumerate(self.survBot.keys): for col_ind, check_key in enumerate(self.survBot.keys):
for row_ind, st_id in enumerate(self.survBot.station_list): for row_ind, nwst_id in enumerate(self.survBot.station_list):
status_dict, detailed_dict = self.survBot.analysis_results.get(st_id) status_dict = self.survBot.analysis_results.get(nwst_id)
status = status_dict.get(check_key) status = status_dict.get(check_key)
detailed_message = detailed_dict.get(check_key) message, detailed_message = status.get_status_str()
if check_key == 'last active':
bg_color = self.get_time_delay_color(status) dt_thresh = [timedelta(seconds=sec) for sec in self.dt_thresh]
elif check_key == 'temp': bg_color = get_bg_color(check_key, status, dt_thresh)
bg_color = self.get_temp_color(status) if check_key == 'temp':
if not type(status) in [str]: if not type(message) in [str]:
status = str(status) + deg_str message = str(message) + deg_str
else:
statussplit = status.split(' ')
if len(statussplit) > 1 and statussplit[0] == 'WARN':
x = int(status.split(' ')[-1].lstrip('(').rstrip(')'))
bg_color = self.colors_dict.get('WARNX')(x)
else:
bg_color = self.colors_dict.get(status)
if not bg_color:
bg_color = self.colors_dict.get('undefined')
# Continue if nothing changed # Continue if nothing changed
text = str(status) text = str(message)
cur_item = self.table.item(row_ind, col_ind) cur_item = self.table.item(row_ind, col_ind)
if cur_item and text == cur_item.text(): if cur_item and text == cur_item.text():
if not self.parameters.get('track_changes') or self.clear_on_refresh: if not self.parameters.get('track_changes') or self.clear_on_refresh:
@@ -305,9 +252,9 @@ class MainWindow(QtWidgets.QMainWindow):
# Create new data item # Create new data item
item = QtWidgets.QTableWidgetItem() item = QtWidgets.QTableWidgetItem()
item.setText(str(status)) item.setText(str(message))
item.setTextAlignment(QtCore.Qt.AlignCenter) item.setTextAlignment(QtCore.Qt.AlignCenter)
item.setData(QtCore.Qt.UserRole, (st_id, check_key)) item.setData(QtCore.Qt.UserRole, (nwst_id, check_key))
# if text changed (known from above) set highlight color/font else (new init) set to default # if text changed (known from above) set highlight color/font else (new init) set to default
cur_item = self.table.item(row_ind, col_ind) cur_item = self.table.item(row_ind, col_ind)
@@ -330,26 +277,6 @@ class MainWindow(QtWidgets.QMainWindow):
# table filling/refreshing done, set clear_on_refresh to False # table filling/refreshing done, set clear_on_refresh to False
self.clear_on_refresh = False self.clear_on_refresh = False
# write html output if parameter is set
self.write_html_table()
def get_time_delay_color(self, dt):
""" Set color of time delay after thresholds specified in self.dt_thresh """
dt_thresh = [timedelta(seconds=sec) for sec in self.dt_thresh]
if dt < dt_thresh[0]:
return self.colors_dict.get('OK')
elif dt_thresh[0] <= dt < dt_thresh[1]:
return self.colors_dict.get('WARN')
return self.colors_dict.get('FAIL')
def get_temp_color(self, temp, vmin=-10, vmax=60, cmap='coolwarm'):
""" Get an rgba temperature value back from specified cmap, linearly interpolated between vmin and vmax. """
if type(temp) in [str]:
return self.colors_dict.get('undefined')
cmap = matplotlib.cm.get_cmap(cmap)
val = (temp - vmin) / (vmax - vmin)
rgba = [int(255 * c) for c in cmap(val)]
return rgba
def set_font_bold(self, item): def set_font_bold(self, item):
""" Set item font bold """ """ Set item font bold """
@@ -382,12 +309,15 @@ class MainWindow(QtWidgets.QMainWindow):
vheader.setSectionResizeMode(index, QtWidgets.QHeaderView.Stretch) vheader.setSectionResizeMode(index, QtWidgets.QHeaderView.Stretch)
def plot_stream(self, item): def plot_stream(self, item):
st_id, check = item.data(QtCore.Qt.UserRole) nwst_id, check = item.data(QtCore.Qt.UserRole)
st = self.survBot.data.get(st_id) st = self.survBot.data.get(nwst_id)
if st: if st:
self.plot_widget = PlotWidget(self) self.plot_widget = PlotWidget(self)
self.plot_widget.setWindowTitle(st_id) self.plot_widget.setWindowTitle(nwst_id)
st = modify_stream_for_plot(st, parameters=self.parameters)
st.plot(equal_scale=False, method='full', block=False, fig=self.plot_widget.canvas.fig) st.plot(equal_scale=False, method='full', block=False, fig=self.plot_widget.canvas.fig)
trace_ylabels(fig=self.plot_widget.canvas.fig, parameters=self.parameters)
trace_yticks(fig=self.plot_widget.canvas.fig, parameters=self.parameters)
self.plot_widget.show() self.plot_widget.show()
def notification(self, text): def notification(self, text):

View File

@@ -1,14 +0,0 @@
#!/bin/bash
ulimit -s 8192
#$ -l os=*stretch
##$ -cwd
#$ -pe smp 1
##$ -q "*@minos15"
export PYTHONPATH="$PYTHONPATH:/home/marcel/git/"
source /opt/anaconda3/etc/profile.d/conda.sh
conda activate py37
python /home/marcel/git/survBot/survBotGUI.py

145
utils.py Normal file
View File

@@ -0,0 +1,145 @@
#!/usr/bin/env python
# -*- coding: utf-8 -*-
import matplotlib
def get_bg_color(check_key, status, dt_thresh=None, hex=False):
message = status.message
if check_key == 'last active':
bg_color = get_time_delay_color(message, dt_thresh)
elif check_key == 'temp':
bg_color = get_temp_color(message)
else:
if status.is_warn:
bg_color = get_color('WARNX')(status.count)
elif status.is_error:
bg_color = get_color('FAIL')
else:
bg_color = get_color(message)
if not bg_color:
bg_color = get_color('undefined')
if hex:
bg_color = '#{:02x}{:02x}{:02x}'.format(*bg_color[:3])
return bg_color
def get_color(key):
# some GUI default colors
colors_dict = {'FAIL': (255, 50, 0, 255),
'NO DATA': (255, 255, 125, 255),
'WARN': (255, 255, 80, 255),
'WARNX': lambda x: (min([255, 200 + x ** 2]), 255, 80, 255),
'OK': (125, 255, 125, 255),
'undefined': (230, 230, 230, 255)}
return colors_dict.get(key)
def get_time_delay_color(dt, dt_thresh):
""" Set color of time delay after thresholds specified in self.dt_thresh """
if dt < dt_thresh[0]:
return get_color('OK')
elif dt_thresh[0] <= dt < dt_thresh[1]:
return get_color('WARN')
return get_color('FAIL')
def get_temp_color(temp, vmin=-10, vmax=60, cmap='coolwarm'):
""" Get an rgba temperature value back from specified cmap, linearly interpolated between vmin and vmax. """
if type(temp) in [str]:
return get_color('undefined')
cmap = matplotlib.cm.get_cmap(cmap)
val = (temp - vmin) / (vmax - vmin)
rgba = [int(255 * c) for c in cmap(val)]
return rgba
def modify_stream_for_plot(st, parameters):
""" copy (if necessary) and modify stream for plotting """
ch_units = parameters.get('CHANNEL_UNITS')
ch_transf = parameters.get('CHANNEL_TRANSFORM')
# if either of both are defined make copy
if ch_units or ch_transf:
st = st.copy()
# modify trace for plotting by multiplying unit factor (e.g. 1e-3 mV to V)
if ch_units:
for tr in st:
channel = tr.stats.channel
unit_factor = ch_units.get(channel)
if unit_factor:
tr.data = tr.data * float(unit_factor)
# modify trace for plotting by other arithmetic expressions
if ch_transf:
for tr in st:
channel = tr.stats.channel
transf = ch_transf.get(channel)
if transf:
tr.data = transform_trace(tr.data, transf)
return st
def transform_trace(data, transf):
"""
Transform trace with arithmetic operations in order, specified in transf
@param data: numpy array
@param transf: list of lists with arithmetic operations (e.g. [['*', '20'], ] -> multiply data by 20
"""
# This looks a little bit hardcoded, however it is safer than using e.g. "eval"
for operator_str, val in transf:
if operator_str == '+':
data = data + val
elif operator_str == '-':
data = data - val
elif operator_str == '*':
data = data * val
elif operator_str == '/':
data = data / val
else:
raise IOError(f'Unknown arithmethic operator string: {operator_str}')
return data
def trace_ylabels(fig, parameters, verbosity=0):
"""
Adds channel names to y-axis if defined in parameters.
Can get mixed up if channel order in stream and channel names defined in parameters.yaml differ, but it is
difficult to assess the correct order from Obspy plotting routing.
"""
names = parameters.get('channel_names')
if not names: # or not len(st.traces):
return
if not len(names) == len(fig.axes):
if verbosity:
print('Mismatch in axis and label lengths. Not adding plot labels')
return
for channel_name, ax in zip(names, fig.axes):
if channel_name:
ax.set_ylabel(channel_name)
def trace_yticks(fig, parameters, verbosity=0):
"""
Adds channel names to y-axis if defined in parameters.
Can get mixed up if channel order in stream and channel names defined in parameters.yaml differ, but it is
difficult to assess the correct order from Obspy plotting routing.
"""
ticks = parameters.get('CHANNEL_TICKS')
if not ticks:
return
if not len(ticks) == len(fig.axes):
if verbosity:
print('Mismatch in axis tick and label lengths. Not changing plot ticks.')
return
for ytick_tripple, ax in zip(ticks, fig.axes):
if not ytick_tripple:
continue
ymin, ymax, step = ytick_tripple
yticks = list(range(ymin, ymax + step, step))
ax.set_yticks(yticks)
ax.set_ylim(ymin - step, ymax + step)

View File

@@ -1,48 +1,68 @@
from datetime import timedelta from datetime import timedelta
def write_html_table_title(fobj, parameters): def write_html_table_title(fobj, parameters):
title = get_print_title_str(parameters) title = get_print_title_str(parameters)
fobj.write(f'<h3>{title}</h3>\n') fobj.write(f'<h3>{title}</h3>\n')
def write_html_text(fobj, text): def write_html_text(fobj, text):
fobj.write(f'<p>{text}</p>\n') fobj.write(f'<p>{text}</p>\n')
def write_html_header(fobj):
def write_html_header(fobj, refresh_rate=10):
header = ['<!DOCTYPE html>', header = ['<!DOCTYPE html>',
'<html>', '<html>',
'<style>', '<head>',
'table, th, td {', ' <link rel="stylesheet" media="only screen and (max-width: 400px)" href="mobile.css" />',
'border:1px solid black;', ' <link rel="stylesheet" media="only screen and (min-width: 401px)" href="desktop.css" />',
'}', '</head>',
'</style>', f'<meta http-equiv="refresh" content="{refresh_rate}" >',
'<meta charset="utf-8">',
'<meta name="viewport" content="width=device-width, initial-scale=1">',
'<body>'] '<body>']
for item in header: for item in header:
fobj.write(item + '\n') fobj.write(item + '\n')
def init_html_table(fobj): def init_html_table(fobj):
fobj.write('<table style="width:100%">\n') fobj.write('<table style="width:100%">\n')
def finish_html_table(fobj): def finish_html_table(fobj):
fobj.write('</table>\n') fobj.write('</table>\n')
def write_html_footer(fobj): def write_html_footer(fobj):
footer = ['</body>', footer = ['</body>',
'</html>'] '</html>']
for item in footer: for item in footer:
fobj.write(item + '\n') fobj.write(item + '\n')
def write_html_row(fobj, items, html_key='td'): def write_html_row(fobj, items, html_key='td'):
fobj.write('<tr>\n') default_space = ' '
fobj.write(default_space + '<tr>\n')
for item in items: for item in items:
text = item.text() text = item.get('text')
color = item.backgroundColor().name() if item.get('bold'):
# fix for black background of headers text = '<b>' + text + '</b>'
if item.get('italic'):
text = '<i>' + text + '</i>'
tooltip = item.get('tooltip')
color = item.get('color')
# check for black background of headers (shouldnt happen anymore)
color = '#e6e6e6' if color == '#000000' else color color = '#e6e6e6' if color == '#000000' else color
fobj.write(f'<{html_key} bgcolor="{color}">' + text + f'</{html_key}>\n') hyperlink = item.get('hyperlink')
fobj.write('</tr>\n') image_str = f'<a href="{hyperlink}">' if hyperlink else ''
html_class = item.get('html_class')
class_str = f' class="{html_class}"' if html_class else ''
fobj.write(2 * default_space + f'<{html_key}{class_str} bgcolor="{color}" title="{tooltip}"> {image_str}'
+ text + f'</{html_key}>\n')
fobj.write(default_space + '</tr>\n')
def get_print_title_str(parameters): def get_print_title_str(parameters):
timespan = parameters.get('timespan') * 24 * 3600 timespan = parameters.get('timespan') * 24 * 3600
tdelta_str = str(timedelta(seconds=int(timespan))) tdelta_str = str(timedelta(seconds=int(timespan))).replace(', 0:00:00', '')
return f'Analysis table of router quality within the last {tdelta_str}' return f'Analysis table of router quality within the last {tdelta_str}'