marcel/pylot
3
0
Fork 0
Code Issues 7 Pull Requests 1 Releases Wiki Activity

Compare commits

merge into: marcel:feature/residual_plot
Branches Tags
marcel:develop
marcel:feature/residual_plot
marcel:master
marcel:correlation_picker
marcel:38-simplify-data-structure
marcel:improve-util-utils
marcel:0.2
marcel:0.1a
...
pull from: marcel:develop
Branches Tags
marcel:develop
marcel:feature/residual_plot
marcel:master
marcel:correlation_picker
marcel:38-simplify-data-structure
marcel:improve-util-utils
marcel:0.2
marcel:0.1a

6 Commits
feature/re ... develop

Author SHA1 Message Date
Marcel
71d4269a4f [bugfix] reverting code from commit 3069e7d5. Checking for coordinates in dataless Parser IS necessary to make sure correct Metadata were found. Fixes #37. 
[minor] Commented out search for network name in metadata filename considered being unsafe
 2024-10-09 17:07:22 +02:00
Marcel
81e34875b9 [update] small changes increasing code robustness 2024-10-09 16:59:12 +02:00
Marcel
d7ee820de3 [minor] adding missing image to doc 2024-09-30 16:40:41 +02:00
Marcel Office Desktop
621cbbfbda [minor] modify README 2024-09-18 16:59:34 +02:00
Marcel Office Desktop
050b9fb0c4 Merge remote-tracking branch 'origin/develop' into develop 2024-09-18 16:57:42 +02:00
Marcel Office Desktop
eb3cd713c6 [update] add description for pick correlation algorithm 2024-09-18 16:56:54 +02:00
8 changed files with 205 additions and 110 deletions
Show Stats Expand all files Collapse all files

6
PyLoT.py Normal file → Executable file
View File

@ -3590,7 +3590,7 @@ class MainWindow(QMainWindow):
def calc_magnitude(self):
self.init_metadata()
if not self.metadata:
return None
return []
wf_copy = self.get_data().getWFData().copy()
@ -3599,6 +3599,10 @@ class MainWindow(QMainWindow):
for station in np.unique(list(self.getPicks('manual').keys()) + list(self.getPicks('auto').keys())):
wf_select += wf_copy.select(station=station)
if not wf_select:
logging.warning('Empty Stream in calc_magnitude. Return.')
return []
corr_wf = restitute_data(wf_select, self.metadata)
# calculate moment magnitude
moment_mag = MomentMagnitude(corr_wf, self.get_data().get_evt_data(), self.inputs.get('vp'),

3
README.md
View File

@ -63,6 +63,9 @@ You may need to do some modifications to these files. Especially folder names sh
PyLoT has been tested on Mac OSX (10.11), Debian Linux 8 and on Windows 10/11.
## Example Dataset
An example dataset with waveform data, metadata and automatic picks in the obspy-dmt dataset format for testing the teleseismic picking can be found at https://zenodo.org/doi/10.5281/zenodo.13759803
## Release notes
#### Features:

77
docs/correlation.md Normal file
View File

@ -0,0 +1,77 @@
# Pick-Correlation Correction
## Introduction
Currently, the pick-correlation correction algorithm is not accessible from they PyLoT GUI. The main file *pick_correlation_correction.py* is located in the directory *pylot\correlation*.
The program only works for an obspy dmt database structure.
The basic workflow of the algorithm is shown in the following diagram. The first step **(1)** is the normal (automatic) picking procedure in PyLoT. Everything from step **(2)** to **(5)** is part of the correlation correction algorithm.
*Note: The first step is not required in case theoretical onsets are used instead of external picks when the parameter use_taupy_onsets is set to True. However, an existing event quakeML (.xml) file generated by PyLoT might be required for each event in case not external picks are used.*
![images/workflow_stacking.png](images/workflow_stacking.png)
A detailed description of the algorithm can be found in the corresponding publication:
*Paffrath, M., Friederich, W., and the AlpArray and AlpArray-SWATH D Working Groups: Teleseismic P waves at the AlpArray seismic network: wave fronts, absolute travel times and travel-time residuals, Solid Earth, 12, 1635–1660, https://doi.org/10.5194/se-12-1635-2021, 2021.*
## How to use
To use the program you have to call the main program providing two mandatory arguments: a path to the obspy dmt database folder *dmt_database_path* and the path to the PyLoT infile *pylot.in* for picking of the beam trace:
```python pick_correlation_correction.py dmt_database_path pylot.in```
By default, the parameter file *parameters.yaml* is used. You can use the command line option *--params* to specify a different parameter file and other optional arguments such as *-pd* for plotting detailed information or *-n 4* to use 4 cores for parallel processing:
```python pick_correlation_correction.py dmt_database_path pylot.in --params parameters_adriaarray.yaml -pd -n 4```
## Cross-Correlation Parameters
The program uses the parameters in the file *parameters.yaml* by default. You can use the command line option *--params* to specify a different parameter file. An example of the parameter file is provided in the *correlation\parameters.yaml* file.
In the top level of the parameter file the logging level *logging* can be set, as well as a list of pick phases *pick_phases* (e.g. ['P', 'S']).
For each pick phase the different parameters can be set in the first sub-level of the parameter file, e.g.:
```yaml
logging: info
pick_phases: ['P', 'S']
P:
min_corr_stacking: 0.8
min_corr_export: 0.6
[...]
S:
min_corr_stacking: 0.7
[...]
```
The following parameters are available:
| Parameter Name | Description | Parameter Type |
|--------------------------------|----------------------------------------------------------------------------------------------------|----------------|
| min_corr_stacking | Minimum correlation coefficient for building beam trace | float |
| min_corr_export | Minimum correlation coefficient for pick export | float |
| min_stack | Minimum number of stations for building beam trace | int |
| t_before | Correlation window before reference pick | float |
| t_after | Correlation window after reference pick | float |
| cc_maxlag | Maximum shift for initial correlation | float |
| cc_maxlag2 | Maximum shift for second (final) correlation (also for calculating pick uncertainty) | float |
| initial_pick_outlier_threshold | Threshold for excluding large outliers of initial (AIC) picks | float |
| export_threshold | Automatically exclude all onsets which deviate more than this threshold from corrected taup onsets | float |
| min_picks_export | Minimum number of correlated picks for export | int |
| min_picks_autopylot | Minimum number of reference auto picks to continue with event | int |
| check_RMS | Do RMS check to search for restitution errors (very experimental) | bool |
| use_taupy_onsets | Use taupy onsets as reference picks instead of external picks | bool |
| station_list | Use the following stations as reference for stacking | list[str] |
| use_stacked_trace | Use existing stacked trace if found (spare re-computation) | bool |
| data_dir | obspyDMT data subdirectory (e.g. 'raw', 'processed') | str |
| pickfile_extension | Use quakeML files (PyLoT output) with the following extension | str |
| dt_stacking | Time difference for stacking window (in seconds) | list[float] |
| filter_options | Filter for first correlation (rough) | dict |
| filter_options_final | Filter for second correlation (fine) | dict |
| filter_type | Filter type (e.g. bandpass) | str |
| sampfreq | Sampling frequency (in Hz) | float |
## Example Dataset
An example dataset with waveform data, metadata and automatic picks in the obspy-dmt dataset format for testing can be found at https://zenodo.org/doi/10.5281/zenodo.13759803

BIN
docs/images/workflow_stacking.png Normal file
View File

Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 64 KiB

199
docs/tuning.md
View File

@ -6,122 +6,123 @@ A description of the parameters used for determining automatic picks.
Parameters applied to the traces before picking algorithm starts.
| Name | Description |
|---------------------|-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
| *P Start*, *P
Stop* | Define time interval relative to trace start time for CF calculation on vertical trace. Value is relative to theoretical onset time if 'Use TauPy' option is enabled in main settings of 'Tune Autopicker' dialogue. |
| *S Start*, *S
Stop* | Define time interval relative to trace start time for CF calculation on horizontal traces. Value is relative to theoretical onset time if 'Use TauPy' option is enabled in main settings of 'Tune Autopicker' dialogue. |
| *Bandpass
Z1* | Filter settings for Butterworth bandpass applied to vertical trace for calculation of initial P pick. |
| *Bandpass
Z2* | Filter settings for Butterworth bandpass applied to vertical trace for calculation of precise P pick. |
| *Bandpass
H1* | Filter settings for Butterworth bandpass applied to horizontal traces for calculation of initial S pick. |
| *Bandpass
H2* | Filter settings for Butterworth bandpass applied to horizontal traces for calculation of precise S pick. |
| Name | Description |
|---------------|-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
| *P Start*, *P | |
| Stop* | Define time interval relative to trace start time for CF calculation on vertical trace. Value is relative to theoretical onset time if 'Use TauPy' option is enabled in main settings of 'Tune Autopicker' dialogue. |
| *S Start*, *S | |
| Stop* | Define time interval relative to trace start time for CF calculation on horizontal traces. Value is relative to theoretical onset time if 'Use TauPy' option is enabled in main settings of 'Tune Autopicker' dialogue. |
| *Bandpass | |
| Z1* | Filter settings for Butterworth bandpass applied to vertical trace for calculation of initial P pick. |
| *Bandpass | |
| Z2* | Filter settings for Butterworth bandpass applied to vertical trace for calculation of precise P pick. |
| *Bandpass | |
| H1* | Filter settings for Butterworth bandpass applied to horizontal traces for calculation of initial S pick. |
| *Bandpass | |
| H2* | Filter settings for Butterworth bandpass applied to horizontal traces for calculation of precise S pick. |
## Inital P pick
Parameters used for determination of initial P pick.
| Name | Description |
|--------------|------------------------------------------------------------------------------------------------------------------------------|
| *
tLTA* | Size of gliding LTA window in seconds used for calculation of HOS-CF. |
| *pickwin
P* | Size of time window in seconds in which the minimum of the AIC-CF in front of the maximum of the HOS-CF is determined. |
| *
AICtsmooth* | Average of samples in this time window will be used for smoothing of the AIC-CF. |
| *
checkwinP* | Time in front of the global maximum of the HOS-CF in which to search for a second local extrema. |
| *minfactorP* | Used with *
checkwinP*. If a second local maximum is found, it has to be at least as big as the first maximum * *minfactorP*. |
| *
tsignal* | Time window in seconds after the initial P pick used for determining signal amplitude. |
| *
tnoise* | Time window in seconds in front of initial P pick used for determining noise amplitude. |
| *tsafetey* | Time in seconds between *tsignal* and *
tnoise*. |
| *
tslope* | Time window in seconds after initial P pick in which the slope of the onset is calculated. |
| Name | Description |
|-------------------------------------------------------------------------------------------------------------------|------------------------------------------------------------------------------------------------------------------------|
| * | |
| tLTA* | Size of gliding LTA window in seconds used for calculation of HOS-CF. |
| *pickwin | |
| P* | Size of time window in seconds in which the minimum of the AIC-CF in front of the maximum of the HOS-CF is determined. |
| * | |
| AICtsmooth* | Average of samples in this time window will be used for smoothing of the AIC-CF. |
| * | |
| checkwinP* | Time in front of the global maximum of the HOS-CF in which to search for a second local extrema. |
| *minfactorP* | Used with * |
| checkwinP*. If a second local maximum is found, it has to be at least as big as the first maximum * *minfactorP*. | |
| * | |
| tsignal* | Time window in seconds after the initial P pick used for determining signal amplitude. |
| * | |
| tnoise* | Time window in seconds in front of initial P pick used for determining noise amplitude. |
| *tsafetey* | Time in seconds between *tsignal* and * |
| tnoise*. | |
| * | |
| tslope* | Time window in seconds after initial P pick in which the slope of the onset is calculated. |
## Inital S pick
Parameters used for determination of initial S pick
| Name | Description |
|---------------|------------------------------------------------------------------------------------------------------------------------------|
| *
tdet1h* | Length of time window in seconds in which AR params of the waveform are determined. |
| *
tpred1h* | Length of time window in seconds in which the waveform is predicted using the AR model. |
| *
AICtsmoothS* | Average of samples in this time window is used for smoothing the AIC-CF. |
| *
pickwinS* | Time window in which the minimum in the AIC-CF in front of the maximum in the ARH-CF is determined. |
| *
checkwinS* | Time in front of the global maximum of the ARH-CF in which to search for a second local extrema. |
| *minfactorP* | Used with *
checkwinS*. If a second local maximum is found, it has to be at least as big as the first maximum * *minfactorS*. |
| *
tsignal* | Time window in seconds after the initial P pick used for determining signal amplitude. |
| *
tnoise* | Time window in seconds in front of initial P pick used for determining noise amplitude. |
| *tsafetey* | Time in seconds between *tsignal* and *
tnoise*. |
| *
tslope* | Time window in seconds after initial P pick in which the slope of the onset is calculated. |
| Name | Description |
|-------------------------------------------------------------------------------------------------------------------|-----------------------------------------------------------------------------------------------------|
| * | |
| tdet1h* | Length of time window in seconds in which AR params of the waveform are determined. |
| * | |
| tpred1h* | Length of time window in seconds in which the waveform is predicted using the AR model. |
| * | |
| AICtsmoothS* | Average of samples in this time window is used for smoothing the AIC-CF. |
| * | |
| pickwinS* | Time window in which the minimum in the AIC-CF in front of the maximum in the ARH-CF is determined. |
| * | |
| checkwinS* | Time in front of the global maximum of the ARH-CF in which to search for a second local extrema. |
| *minfactorP* | Used with * |
| checkwinS*. If a second local maximum is found, it has to be at least as big as the first maximum * *minfactorS*. | |
| * | |
| tsignal* | Time window in seconds after the initial P pick used for determining signal amplitude. |
| * | |
| tnoise* | Time window in seconds in front of initial P pick used for determining noise amplitude. |
| *tsafetey* | Time in seconds between *tsignal* and * |
| tnoise*. | |
| * | |
| tslope* | Time window in seconds after initial P pick in which the slope of the onset is calculated. |
## Precise P pick
Parameters used for determination of precise P pick.
| Name | Description |
|--------------|-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
| *Precalcwin* | Time window in seconds for recalculation of the HOS-CF. The new CF will be two times the size of *
Precalcwin*, since it will be calculated from the initial pick to +/- *Precalcwin*. |
| *
tsmoothP* | Average of samples in this time window will be used for smoothing the second HOS-CF. |
| *
ausP* | Controls artificial uplift of samples during precise picking. A common local minimum of the smoothed and unsmoothed HOS-CF is found when the previous sample is larger or equal to the current sample times (1+*
ausP*). |
| Name | Description |
|-------------------------------------------------------------------------------------|------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
| *Precalcwin* | Time window in seconds for recalculation of the HOS-CF. The new CF will be two times the size of * |
| Precalcwin*, since it will be calculated from the initial pick to +/- *Precalcwin*. | |
| * | |
| tsmoothP* | Average of samples in this time window will be used for smoothing the second HOS-CF. |
| * | |
| ausP* | Controls artificial uplift of samples during precise picking. A common local minimum of the smoothed and unsmoothed HOS-CF is found when the previous sample is larger or equal to the current sample times (1+* |
| ausP*). | |
## Precise S pick
Parameters used for determination of precise S pick.
| Name | Description |
|--------------|-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
| *
tdet2h* | Time window for determination of AR coefficients. |
| *
tpred2h* | Time window in which the waveform is predicted using the determined AR parameters. |
| *Srecalcwin* | Time window for recalculation of ARH-CF. New CF will be calculated from initial pick +/- *
Srecalcwin*. |
| *
tsmoothS* | Average of samples in this time window will be used for smoothing the second ARH-CF. |
| *
ausS* | Controls artificial uplift of samples during precise picking. A common local minimum of the smoothed and unsmoothed ARH-CF is found when the previous sample is larger or equal to the current sample times (1+*
ausS*). |
| *
pickwinS* | Time window around initial pick in which to look for a precise pick. |
| Name | Description |
|--------------|------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
| * | |
| tdet2h* | Time window for determination of AR coefficients. |
| * | |
| tpred2h* | Time window in which the waveform is predicted using the determined AR parameters. |
| *Srecalcwin* | Time window for recalculation of ARH-CF. New CF will be calculated from initial pick +/- * |
| Srecalcwin*. | |
| * | |
| tsmoothS* | Average of samples in this time window will be used for smoothing the second ARH-CF. |
| * | |
| ausS* | Controls artificial uplift of samples during precise picking. A common local minimum of the smoothed and unsmoothed ARH-CF is found when the previous sample is larger or equal to the current sample times (1+* |
| ausS*). | |
| * | |
| pickwinS* | Time window around initial pick in which to look for a precise pick. |
## Pick quality control
Parameters used for checking quality and integrity of automatic picks.
| Name | Description |
|--------------------------------------------|--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
| *
minAICPslope* | Initial P picks with a slope lower than this value will be discared. |
| *
minAICPSNR* | Initial P picks with a SNR below this value will be discarded. |
| *
minAICSslope* | Initial S picks with a slope lower than this value will be discarded. |
| *
minAICSSNR* | Initial S picks with a SNR below this value will be discarded. |
| *minsiglength*, *noisefacor*. *minpercent* | Parameters for checking signal length. In the time window of size *
| Name | Description |
|--------------------------------------------|-----------------------------------------------------------------------|
| * | |
| minAICPslope* | Initial P picks with a slope lower than this value will be discared. |
| * | |
| minAICPSNR* | Initial P picks with a SNR below this value will be discarded. |
| * | |
| minAICSslope* | Initial S picks with a slope lower than this value will be discarded. |
| * | |
| minAICSSNR* | Initial S picks with a SNR below this value will be discarded. |
| *minsiglength*, *noisefacor*. *minpercent* | Parameters for checking signal length. In the time window of size * |
minsiglength* after the initial P pick *
minpercent* of samples have to be larger than the RMS value. |
| *
@ -139,12 +140,12 @@ wdttolerance* | Maximum allowed deviation of S onset
Parameters for discrete quality classes.
| Name | Description |
|------------------|-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
| *
timeerrorsP* | Width of the time windows in seconds between earliest and latest possible pick which represent the quality classes 0, 1, 2, 3 for P onsets. |
| *
timeerrorsS* | Width of the time windows in seconds between earliest and latest possible pick which represent the quality classes 0, 1, 2, 3 for S onsets. |
| *nfacP*, *nfacS* | For determination of latest possible onset time. The time when the signal reaches an amplitude of *
nfac* * mean value of the RMS amplitude in the time window *tnoise* corresponds to the latest possible onset time. |
| Name | Description |
|--------------------------------------------------------------------------------------------------------------------|---------------------------------------------------------------------------------------------------------------------------------------------|
| * | |
| timeerrorsP* | Width of the time windows in seconds between earliest and latest possible pick which represent the quality classes 0, 1, 2, 3 for P onsets. |
| * | |
| timeerrorsS* | Width of the time windows in seconds between earliest and latest possible pick which represent the quality classes 0, 1, 2, 3 for S onsets. |
| *nfacP*, *nfacS* | For determination of latest possible onset time. The time when the signal reaches an amplitude of * |
| nfac* * mean value of the RMS amplitude in the time window *tnoise* corresponds to the latest possible onset time. | |

4
pylot/core/util/array_map.py
View File

@ -134,8 +134,8 @@ class Array_map(QtWidgets.QWidget):
self.cmaps_box = QtWidgets.QComboBox()
self.cmaps_box.setMaxVisibleItems(20)
[self.cmaps_box.addItem(map_name) for map_name in sorted(plt.colormaps())]
# try to set to viridis as default
self.cmaps_box.setCurrentIndex(self.cmaps_box.findText('viridis'))
# try to set to plasma as default
self.cmaps_box.setCurrentIndex(self.cmaps_box.findText('plasma'))
self.top_row.addWidget(QtWidgets.QLabel('Select a phase: '))
self.top_row.addWidget(self.comboBox_phase)

19
pylot/core/util/dataprocessing.py
View File

@ -268,9 +268,6 @@ class Metadata(object):
if not fnames:
# search for station name in filename
fnames = glob.glob(os.path.join(path_to_inventory, '*' + station + '*'))
if not fnames:
# search for network name in filename
fnames = glob.glob(os.path.join(path_to_inventory, '*' + network + '*'))
if not fnames:
if self.verbosity:
print('Could not find filenames matching station name, network name or seed id')
@ -282,7 +279,7 @@ class Metadata(object):
continue
invtype, robj = self._read_metadata_file(os.path.join(path_to_inventory, fname))
try:
# robj.get_coordinates(station_seed_id) # TODO: Commented out, failed with Parser, is this needed?
robj.get_coordinates(station_seed_id)
self.inventory_files[fname] = {'invtype': invtype,
'data': robj}
if station_seed_id in self.seed_ids.keys():
@ -290,6 +287,7 @@ class Metadata(object):
self.seed_ids[station_seed_id] = fname
return True
except Exception as e:
logging.warning(e)
continue
print('Could not find metadata for station_seed_id {} in path {}'.format(station_seed_id, path_to_inventory))
@ -654,6 +652,8 @@ def restitute_data(data, metadata, unit='VEL', force=False, ncores=0):
"""
# data = remove_underscores(data)
if not data:
return
# loop over traces
input_tuples = []
@ -661,9 +661,14 @@ def restitute_data(data, metadata, unit='VEL', force=False, ncores=0):
input_tuples.append((tr, metadata, unit, force))
data.remove(tr)
pool = gen_Pool(ncores)
result = pool.imap_unordered(restitute_trace, input_tuples)
pool.close()
if ncores == 0:
result = []
for input_tuple in input_tuples:
result.append(restitute_trace(input_tuple))
else:
pool = gen_Pool(ncores)
result = pool.imap_unordered(restitute_trace, input_tuples)
pool.close()
for tr, remove_trace in result:
if not remove_trace:

7
pylot/correlation/parameters_adriaarray.yaml
View File

@ -9,7 +9,7 @@
# initial_pick_outlier_threshold: (hopefully) threshold for excluding large outliers of initial (AIC) picks
# export_threshold: automatically exclude all onsets which deviate more than this threshold from corrected taup onsets
# min_picks_export: minimum number of correlated picks for export
# min_picks_autopylot: minimum number of reference autopicks picks to continue with event
# min_picks_autopylot: minimum number of reference auto picks to continue with event
# check_RMS: do RMS check to search for restitution errors (very experimental)
# use_taupy_onsets: use taupy onsets as reference picks instead of external picks
# station_list: use the following stations as reference for stacking
@ -17,6 +17,11 @@
# data_dir: obspyDMT data subdirectory (e.g. 'raw', 'processed')
# pickfile_extension: use quakeML files (PyLoT output) with the following extension, e.g. '_autopylot' for pickfiles
# such as 'PyLoT_20170501_141822_autopylot.xml'
# dt_stacking: time shift for stacking (e.g. [0, 250] for 0 and 250 seconds shift)
# filter_options: filter for first correlation (rough)
# filter_options_final: filter for second correlation (fine)
# filter_type: e.g. 'bandpass'
# sampfreq: sampling frequency of the data
logging: info
pick_phases: ['P', 'S']