Actions

Difference between revisions of "Metadata of Model results in Coastal Engineering"

From BAWiki

m (first draft)
(Replaced the text)
Line 1: Line 1:
 
[[de:Metadaten_von_Modellergebnissen_im_Küstenwasserbau]]
 
[[de:Metadaten_von_Modellergebnissen_im_Küstenwasserbau]]
 
'''This page is under construction!'''
 
'''This page is under construction!'''
The metadata set is a thoroughly documented package that describes the simulations and hydrodynamic model results in coastal engineering. The details included in metadata utterly summaries different aspect of the data in information packages as identification, contact, quality, lineage, hierarchy, distribution, constraint, extent and reference. Completeness of metadata ensures proper utilization of data or their reproducibility. Furthermore, the metadata facilitate administration and search as well as sort and classification of the data. This page explains how to collect and provide institutional (BAW internal) metadata for coastal simulation data considering the existing [http://wiki-intern.baw.de/images/b/b4/Metadatenprofil_GDI-BAW-V_1.3_Technische_Doku_1.0.pdf |BAW metadata profile]. 
 
  
'''1. Collecting metadata of a simulation'''
+
The metadata set is a thoroughly documented package that describes the simulations and hydrodynamic model results in coastal engineering. The details included in metadata utterly summaries different aspect of the data in information packages as identification, contact, quality, lineage, hierarchy, distribution, constraint, extent and reference. Completeness of metadata ensures proper utilization of data or their reproducibility. Furthermore, the metadata facilitate administration and search as well as sort and classification of the data. This page explains how to collect and provide institutional (BAW internal) metadata for coastal simulation data considering the existing [http://wiki-intern.baw.de/images/b/b4/Metadatenprofil_GDI-BAW-V_1.3_Technische_Doku_1.0.pdf BAW metadata profile]. 
  
Collecting the metadata has been implemented in the modelling system UnTRIM, consisting of the subsequent partial models:
+
'''[1] Collecting Metadata'''
  
* Hydrodynamics in the package Utromp2007
+
Meta-information of a simulation is collected in the processing chain for a simulation data. The UnTRIM and UnTRIM2 modeling approach through the use of the NetCDF data format processes the metadata in either of the subsequent sub-models:
 +
 
 +
* Hydrodynamics in utromp2007 and  SubGrid in utromp2009
 
* Morphodynamics in SediMorph
 
* Morphodynamics in SediMorph
* Spectral Waves in the k-Model
+
* Waves in the k-Model
 +
 
 +
Setting environmental parameters is a requirement and an initial step to conduct a simulation. These environmental variables are set through run and project scripts (in German: Auftragsskripte), which also comprise metadata of respective simulation. The user navigates adequately these variables on the script and provides the required attributes as metadata (see figure 01). Recording meta-information is a single time customization task for a project with multiple simulation conditions. Each of the subsequent methods e.g. hydrodynamics, morphodynamics and sea motion is providing and securing their relevant metadata in the processing chain from their run and project scripts.
 +
 
 +
[[File:Run and project script.png|Run and project script ]]
 +
Figure 01. Run and project script (Auftragsskript) to insert the meta-information for a simulation.
  
Missing meta information has to be set by environment variables, which are read by the partial models. It is recommended to bundle those variables in a project wide shell script. For detailed information have a look at the [http://wiki.baw.de/de/index.php/Erfassung_der_Metadaten_einer_Simulation| guidelines in German].
+
Most of the metadata elements are acquired from the steering data of the processing environment automatically. However, there are approximately 15 customized elements relevant to the project management and contact methods the user is providing. A quality assurance aspect in this process is the validity check through comparisons of the inserted values with the existing code list as configuration data. The user provides the following elements in the project script:  
  
'''2. Collecting meta data of an analysis'''
+
* DMQS_PSP: Project PSP number from EWisA
 +
* DMQS_TITLE: Project title from EWisA
 +
* DMQS_SUMMARY: summary of the project in a max of 240 characters
 +
* DMQS_CREATOR_NAME: "<surname>, <name>, <optional_title>" of project manager
 +
* DMQS_CREATOR_EMAIL: email address of project manager
 +
* DMQS_CREATOR_ROLE:  role of project manager
 +
* DMQS_GEO_ID: a list for valid geographical identification located in \\themis\system\akprog\cfg\dmqs\geo\valid_geo_ids.dat ($CFGDIR).
 +
* Further elements, in case the project is funded through mFUND:
 +
** MFUND_PROJECT: name of the funded project
 +
** MFUND_ID: project funding number
 +
For a more accurate description of a simulation file, its lineage and hierarchal information is also required to be stored. The elements relevant to the simulation description and hierarchy to set are:
 +
* DMQS_VARIANT: describes the geometrical property of a model
 +
* DMQS_SCENARIO: describes the time or boundary conditions and other features of a simulation e.g. wind or storm event
 +
* RUN: distinguishes further a simulation through its specific feature/characteristic 
 +
* DMQS_RUN_COMMENT: free text for an exact description of each individual simulation
 +
Note: for ease and access the template files are store as:
 +
* The template file for the project script is located under $PROGHOME/bin/dmqs/set_dmqs_template.sh
 +
* Template files for the run script are located under $PROGHOME/examples/slurm
 +
* List of the valid names are located under:
 +
** $PROGHOME/cfg/dmqs/variant/01_template.dat
 +
** $PROGHOME/cfg/dmqs/scenario/01_template.dat
 +
** $PROGHOME/cfg/dmqs/geo/valid_geo_ids.dat
 +
Post-processing and analyzing algorithms append their process relevant meta-information to their output NetCDF files. Furthermore, the lineage information and steering information is complemented in this process as well. Figure 02 further illustrates the post-processing chain, which automatically writes its meta-information in the NetCDF files.
  
The analysis tools at the BAW compute characteristic values out of the model results and write these values again in NetCDF files. They also read the meta data of the modelling results, complement them and write it to its output file. [http://wiki.baw.de/de/index.php/Erfassung_von_Analyse-Metadaten| Full story.].
+
[[File:Post-processing chain.png| Figure 02: post-processing chain.]]
[[Datei:DMQS Shellskripte.png|820px|thumb|center|Abb. 1: Shellskripte zum Erfassen der DMQS-Metadaten eines Simulationslaufes]]
+
Figure 02: post-processing chain.
  
'''3. Creation of sets of meta data'''
+
'''[2] Extracting Metadata '''
  
The visualisation tool [Davit] has been enabled to generate sets of meta data in the machine readable XML-Format and as PDF documents. The user can activate the DMQS metadata generation in Davit's screenshot menu where the output of figures is defined. A reduced set of meta data describing the picture and a second one containing information about the underlying NetCDF file are generated. [http://wiki.baw.de/de/index.php/Erzeugung_der_Metadatensätze| Full story.].
+
An extra metadata generation tool is integrated into the visualization software [https://wiki.baw.de/en/index.php/DAVIT Davit]. This tool is developed and customized based on the BAW requirements, to convert the NetCDF CF metadata elements to ISO compliant XML metadata. Extensible Markup Language (XML) is both human and machine-readable and the standard data format for storing and exchanging of geographical metadata and attributes via the web. The exported XML metadata set is further mapped based on ISO schema in adherence to the relevant standards listed in BAW metadata profile [http://wiki-intern.baw.de/images/b/b4/Metadatenprofil_GDI-BAW-V_1.3_Technische_Doku_1.0.pdf (GDI-BAW V1.3)], which makes it compatible with a wide range of information systems. Furthermore, to ensure the validity and completeness of the exported metadata *.XML file, a new popup window appears if any of the core elements is missing in the NetCDF file.
  
'''4. Editing and copying of sets of meta data'''
+
'''[3] Editing Metadata'''
  
After simulation or analysis result files are often moved to other hard disks. In order to store data permanently it is e.g. moved into an archive. Since the meta data contains links to steering and result files, the paths in the links have to be adapted. Copying and editing is done with automatically generated scripts. [http://wiki.baw.de/de/index.php/Editieren_und_Kopieren_der_Metadatensätze| Full story.].
+
In this process, it is considered that the metadata is complete and valid. However, if this is not the case, further manual editing possibilities are:
 +
# NetCDF attribute via NCO-Tool ‘ncatted’ or MATLAB ‘ncwriteatt’ command,
 +
# XML file in a notepad e.g. notepad++,
 +
# InGrid editor (IGE) interface of MIS-BAW.
  
'''5. Referencing sets of meta data in a document'''
 
  
The reader of a dokument can get information about a figure displaying model results by clicking on it. This is technically done by a Word macro referencing the set of meta data in PDF format. Figure and PDF have similar names:
+
'''[4] Administrating Metadata'''
* <path>/<picture>.png
 
* <path>/<picture>.png.pdf: set of meta data for the figure <br>
 
[http://wiki.baw.de/de/index.php/Einbinden_von_Metadatensätzen_in_ein_Dokument| Full story].
 
  
 +
The produced XML dataset is then imported in the metadata information system MIS-BAW. The user can import a dataset via either (a) InGrid editor (IGE) or (b) CSW-T interface via a curl command. The imported elements could be also further edited in IGE. In addition, IGE provides a further possibility of manual entry of a new metadata set.
 +
 +
For detailed information have a look at the [https://wiki.baw.de/de/index.php/Erfassung_der_Metadaten_einer_Simulation guidelines in German].
 +
 +
'''Acknowledgment'''
 +
 +
This project took advantage of NetCDF software developed by [http://www.unidata.ucar.edu/software/netcdf/ UCAR/Unidata].
  
Acknowledgement: ''This project took advantage of netCDF software developed by UCAR/Unidata ([http://www.unidata.ucar.edu/software/netcdf/ www.unidata.ucar.edu/software/netcdf/]).''<br>
 
<big>'''Under Construction'''</big> <br>
 
 
===Documentation/Literature ===
 
===Documentation/Literature ===
Meta data of model results is a part of the project data management und quality assurance in waterways engineering (DMQS). See further information on the [http://intranet.baw.de/cocoon/portal/res?t=1&ci=1&si=0&i=344&d=Aufgabe&v=DETAIL&x=DETAIL&hra=12&hi=344&hd=Aufgabe&hc=8766&sichtc=cms&r=&cid=8766&rt=&nd=Aufgabe&ni=344&uki=344&ukd=Aufgabe&uksi=&ukg=1&ucid=8766&bereich=&ber=&lim=&ord=&os=&asc=&pm=&arch=false&l=DE&s=de&su=&p=&bn=&bc=&refModule=Hauptnavigation intranet pages of the BAW]. <br>  
+
 
 +
Metadata of model results is a part of the project data management and quality assurance (DMQS) in waterways engineering. Further information is available on the [http://intranet.baw.de/cocoon/portal/res?t=1&ci=1&si=0&i=344&d=Aufgabe&v=DETAIL&x=DETAIL&hra=12&hi=344&hd=Aufgabe&hc=8766&sichtc=cms&r=&cid=8766&rt=&nd=Aufgabe&ni=344&uki=344&ukd=Aufgabe&uksi=&ukg=1&ucid=8766&bereich=&ber=&lim=&ord=&os=&asc=&pm=&arch=false&l=DE&s=de&su=&p=&bn=&bc=&refModule=Hauptnavigation intranet pages of the BAW]. <br>  
 
----
 
----
 
back to [[Data Management and Information Systems]]
 
back to [[Data Management and Information Systems]]
 
----
 
----
 
[[Overview]]
 
[[Overview]]
 +
<br> <big>'''Under Construction'''</big> <br>

Revision as of 16:47, 10 March 2020

This page is under construction!

The metadata set is a thoroughly documented package that describes the simulations and hydrodynamic model results in coastal engineering. The details included in metadata utterly summaries different aspect of the data in information packages as identification, contact, quality, lineage, hierarchy, distribution, constraint, extent and reference. Completeness of metadata ensures proper utilization of data or their reproducibility. Furthermore, the metadata facilitate administration and search as well as sort and classification of the data. This page explains how to collect and provide institutional (BAW internal) metadata for coastal simulation data considering the existing BAW metadata profile.

[1] Collecting Metadata

Meta-information of a simulation is collected in the processing chain for a simulation data. The UnTRIM and UnTRIM2 modeling approach through the use of the NetCDF data format processes the metadata in either of the subsequent sub-models:

  • Hydrodynamics in utromp2007 and SubGrid in utromp2009
  • Morphodynamics in SediMorph
  • Waves in the k-Model

Setting environmental parameters is a requirement and an initial step to conduct a simulation. These environmental variables are set through run and project scripts (in German: Auftragsskripte), which also comprise metadata of respective simulation. The user navigates adequately these variables on the script and provides the required attributes as metadata (see figure 01). Recording meta-information is a single time customization task for a project with multiple simulation conditions. Each of the subsequent methods e.g. hydrodynamics, morphodynamics and sea motion is providing and securing their relevant metadata in the processing chain from their run and project scripts.

Run and project script Figure 01. Run and project script (Auftragsskript) to insert the meta-information for a simulation.

Most of the metadata elements are acquired from the steering data of the processing environment automatically. However, there are approximately 15 customized elements relevant to the project management and contact methods the user is providing. A quality assurance aspect in this process is the validity check through comparisons of the inserted values with the existing code list as configuration data. The user provides the following elements in the project script:

  • DMQS_PSP: Project PSP number from EWisA
  • DMQS_TITLE: Project title from EWisA
  • DMQS_SUMMARY: summary of the project in a max of 240 characters
  • DMQS_CREATOR_NAME: "<surname>, <name>, <optional_title>" of project manager
  • DMQS_CREATOR_EMAIL: email address of project manager
  • DMQS_CREATOR_ROLE: role of project manager
  • DMQS_GEO_ID: a list for valid geographical identification located in \\themis\system\akprog\cfg\dmqs\geo\valid_geo_ids.dat ($CFGDIR).
  • Further elements, in case the project is funded through mFUND:
    • MFUND_PROJECT: name of the funded project
    • MFUND_ID: project funding number

For a more accurate description of a simulation file, its lineage and hierarchal information is also required to be stored. The elements relevant to the simulation description and hierarchy to set are:

  • DMQS_VARIANT: describes the geometrical property of a model
  • DMQS_SCENARIO: describes the time or boundary conditions and other features of a simulation e.g. wind or storm event
  • RUN: distinguishes further a simulation through its specific feature/characteristic
  • DMQS_RUN_COMMENT: free text for an exact description of each individual simulation

Note: for ease and access the template files are store as:

  • The template file for the project script is located under $PROGHOME/bin/dmqs/set_dmqs_template.sh
  • Template files for the run script are located under $PROGHOME/examples/slurm
  • List of the valid names are located under:
    • $PROGHOME/cfg/dmqs/variant/01_template.dat
    • $PROGHOME/cfg/dmqs/scenario/01_template.dat
    • $PROGHOME/cfg/dmqs/geo/valid_geo_ids.dat

Post-processing and analyzing algorithms append their process relevant meta-information to their output NetCDF files. Furthermore, the lineage information and steering information is complemented in this process as well. Figure 02 further illustrates the post-processing chain, which automatically writes its meta-information in the NetCDF files.

Figure 02: post-processing chain. Figure 02: post-processing chain.

[2] Extracting Metadata

An extra metadata generation tool is integrated into the visualization software Davit. This tool is developed and customized based on the BAW requirements, to convert the NetCDF CF metadata elements to ISO compliant XML metadata. Extensible Markup Language (XML) is both human and machine-readable and the standard data format for storing and exchanging of geographical metadata and attributes via the web. The exported XML metadata set is further mapped based on ISO schema in adherence to the relevant standards listed in BAW metadata profile (GDI-BAW V1.3), which makes it compatible with a wide range of information systems. Furthermore, to ensure the validity and completeness of the exported metadata *.XML file, a new popup window appears if any of the core elements is missing in the NetCDF file.

[3] Editing Metadata

In this process, it is considered that the metadata is complete and valid. However, if this is not the case, further manual editing possibilities are:

  1. NetCDF attribute via NCO-Tool ‘ncatted’ or MATLAB ‘ncwriteatt’ command,
  2. XML file in a notepad e.g. notepad++,
  3. InGrid editor (IGE) interface of MIS-BAW.


[4] Administrating Metadata

The produced XML dataset is then imported in the metadata information system MIS-BAW. The user can import a dataset via either (a) InGrid editor (IGE) or (b) CSW-T interface via a curl command. The imported elements could be also further edited in IGE. In addition, IGE provides a further possibility of manual entry of a new metadata set.

For detailed information have a look at the guidelines in German.

Acknowledgment

This project took advantage of NetCDF software developed by UCAR/Unidata.

Documentation/Literature

Metadata of model results is a part of the project data management and quality assurance (DMQS) in waterways engineering. Further information is available on the intranet pages of the BAW.


back to Data Management and Information Systems


Overview
Under Construction