NCCUTOUT: Difference between revisions

Latest revision as of 10:36, 6 September 2022

Basic Information

Name of Program

NCCUTOUT

Version-Date

July 2022

Description-Date

September 2022

Catchwords

data analysis
postprocessor
cut out data
synoptic data sets
tidal characteristic numbers
tide-independent characteristic numbers
differences of data sets
CF NetCDF format for 2D/3D data
support of computational results based on sub-grid bathymetry
support of DMQS metadata and variables
Storage of the content of the ASCII input control files in netcdf.nc (as a variable)
Storage of MD5 hash values of input files in netcdf.nc (as a variable)
optional use of Message Passing Interface (MPI, MPI Forum)

Acknowledgment: This project took advantage of netCDF software developed by UCAR/Unidata (www.unidata.ucar.edu/software/netcdf/).

Short Description of Functionality

Fig. 1: Results of the German Bight cut out from the modell area of the project EasyGSH-DB (BAW, 2018)

The postprocessor NCCUTOUT may help users to reduce the size of result files in order to save disk space and computation time in the subsequent processing steps. It cuts out data of UGRID compliant NetCDF files of the area type. In UGRID metadata language the files have to have at least one mesh topology variable with the attribute topology_dimension = 2. The cutting is typically defined by a polygon. The elements inside of the polygon get extracted to the result file.

Fig. 2: Results at separate faces (see north western and south eastern part) cut out of a model of the Jade Weser Estuary (BAW, 2018)

But the cut out area can also be defined by two or more polygons which need not be direct neighbours, s. Fig. 2. Though the results may look like seperate spots of a location grid they share a common mesh topology.

Furthermore, the user can reduce disk space by extracting only a subset of the original geophysical variables. He can add them to a white list or alternatively to a black list.

Input-Files

general input data (file type nccutout.dat);
input UGRID CF NetCDFs (file type cf-netcdf.nc);
cutout polygons (file type ipds.dat);
for automatic quality assurance (file type bounds_verify.dat).

Output-Files

output UGRID CF NetCDFs (file type cf-netcdf.nc);
(optional) informative printer file of program execution (file type nccutout.sdr)
(optional) trace of program execution (file type nccutout.trc)

Methodology

Fig. 3: Extracting faces with all their subfaces in order to generate consistent meshes.(BAW,2018)

NCCUTOUT can process NetCDF files supporting the UGRID idea of parent and child meshes, e.g. a computational grid with a coarse and a sub grid with higher resolution. Whether data gets cut out or not depends from the position of the geometric elements in the coarsest mesh. This is displayed in Fig. 3. Certain faces of the coarser computational grid are cut out and subsequently all subfaces inside these coarser faces are extracted. This method leads to consistent meshes and parent child connections.

Regarding the postprocessing workflow it is recommended to use NCCUTOUT directly after the numerical engines, e.g. UNTRIM2007 and UnTRIM2, so that the complete postprocessing benefits from the reduction.

An introduction how to collect DMQS metadata can be found here.

Performance: Most of the program's CPU time is used by basic IO routines. Since output is in the tested examples considerably slower than input a reduction of the output data, area and geophysical variables, makes NCCUTOUT run faster.

Program(s) to run before this Program

NCANALYSE, NCCHUNKIE, NCDELTA, NCDVAR, NCMERGE, UNTRIM2007, UNTRIM2

Program(s) to run after this Program

DAVIT, NCAGGREGATE, NCANALYSE, NCAUTO, NCCHUNKIE, NCDELTA, NCDVAR, NCMERGE, NCPLOT, NC2TABLE and MATLAB

Additional Information

Language

Fortran95

Additional software

---

Original Version

P. Schade, G. Lang

Maintenance

Working group POS

Documentation/Literature

Template Files:
- general template files available in $PROGHOME/examples/nccutout
- in test directory for hydro-dynamic UNTRIM2007 results: $PROGHOME/examples/nccutout/run/utr2007/elbe/nccut
- in test directory for hydro-dynamic UNTRIM2 results: $PROGHOME/examples/nccutout/run/utr2009/easy/nccut
- in test directory for NCANALYSE results: $PROGHOME/examples/nccutout/run/ncana/easy/nccut

back to Program Descriptions

Overview

@@ Line 2: / Line 2: @@
 |name_de=NCCUTOUT
 |name=NCCUTOUT
-|version=November 2018
+|version=July 2022
-|version_descr=November 2018
+|version_descr=September 2022
 |catchwords=
 data analysis<br />
@@ Line 15: / Line 15: @@
 support of computational results based on sub-grid bathymetry<br />
 support of DMQS metadata and variables<br />
-[[NetCDF#Quality assurance using NetCDF attributes|automatic quality assurance (value range)]]<br />
+Storage of the content of the ASCII input control files in [[CF-NETCDF.NC|netcdf.nc]] (as a variable)<br />
-xxx automatic computation of chunk sizes for result variables to support orthogonal data access
+Storage of [https://en.wikipedia.org/wiki/MD5 MD5 hash values] of input files in [[CF-NETCDF.NC|netcdf.nc]] (as a variable)<br />
+optional use of ''Message Passing Interface'' (MPI, [https://www.mpi-forum.org/ MPI Forum])
   Acknowledgment: ''This project took advantage of netCDF software developed by UCAR/Unidata ([http://www.unidata.ucar.edu/software/netcdf/ www.unidata.ucar.edu/software/netcdf/]).''
 |shortdescription=
-The postprocessor NCCUTOUT may help users to reduce the size of result files in order to save disk space and time in the subsequent processing steps. It cuts out data of UGRID compliant NetCDF files of the area type. In UGRID metadata language the files have to have at least one mesh topology variable with the attribute topology_dimension = 2.
+<P>[[File:Nccutout_one_polygon.png|250px|thumb|right|Fig. 1: Results of the German Bight cut out from the modell area of the project EasyGSH-DB (BAW, 2018)]]
+The postprocessor NCCUTOUT may help users to reduce the size of result files in order to save disk space and computation time in the subsequent processing steps. It cuts out data of UGRID compliant NetCDF files of the area type. In UGRID metadata language the files have to have at least one mesh topology variable with the attribute topology_dimension = 2.  The cutting is typically defined by a polygon. The elements inside of the polygon get extracted to the result file. </P>
-The cutting is typically defined by one polygon. The elements inside of the polygon get extracted to the result file. The user can select with a positive or negative list, which variables should be extracted.
+<P>[[File:Nccutout_more_polygons.png|250px|thumb|right|Fig. 2: Results at separate faces (see north western and south eastern part) cut out of a model of the Jade Weser Estuary (BAW, 2018) ]]
+But the cut out area can also be defined by two or more polygons which need not be direct neighbours, s. Fig. 2. Though the results may look like seperate spots of a location grid they share a common mesh topology.</P>
-But the cut out area can also be defined by two or more polygons which need not be direct neighbours. Though the results may look like seperated areas they share a common mesh topology.
-[[File:NCCUTOUT_patch.png|thumb|'''Picture ''control volumes and exchanges'''''.]]
+Furthermore, the user can reduce disk space by extracting only a subset of the original geophysical variables. He can add them to a white list or alternatively to a black list. <BR>
 |inputfiles=
 # '''general input data''' (file type [[NCCUTOUT.DAT|nccutout.dat]]);
 # '''input UGRID CF NetCDFs''' (file type [[CF-NETCDF.NC|cf-netcdf.nc]]);
 # '''cutout polygons''' (file type [[IPDS.DAT|ipds.dat]]);
-# xxx for [[NetCDF#Automatic quality assurance using NetCDF attributes|automatic quality assurance]] (file type [[BOUNDS.CFG.DAT|bounds_verify.dat]]).
+# for [[NetCDF#Automatic quality assurance using NetCDF attributes|automatic quality assurance]] (file type [[BOUNDS.CFG.DAT|bounds_verify.dat]]).
 |outputfiles=
@@ Line 41: / Line 41: @@
 |methodology=
-NCCUTOUT can process NetCDF files supporting the UGRID idea of parent and child meshes, e.g. a computational grid with a coarse and a sub grid with higher resolution. Whether data gets cut out or not depends from the position of the geometric elements in the coarsest mesh. This is displayed in Fig. 1. Certain faces of the coarser computational grid are cut out and subsequently all subfaces inside these coarser faces are extracted. This methods leads to consistent meshes and parent child connections.
+<P>[[File:Nccutout_faces_subfaces.png|250px|thumb|right|Fig. 3: Extracting faces with all their subfaces in order to generate consistent meshes.(BAW,2018)]] NCCUTOUT can process NetCDF files supporting the UGRID idea of parent and child meshes, e.g. a computational grid with a coarse and a sub grid with higher resolution. Whether data gets cut out or not depends from the position of the geometric elements in the coarsest mesh. This is displayed in Fig. 3. Certain faces of the coarser computational grid are cut out and subsequently all subfaces inside these coarser faces are extracted. This method leads to consistent meshes and parent child connections.</P>
+<P>
+Regarding the postprocessing workflow it is recommended to use NCCUTOUT directly after the numerical engines, e.g. UNTRIM2007 and UnTRIM2, so that the complete postprocessing benefits from the reduction.</P>
 <P>
-It is recommended to use NCCUTOUT for reducing the size of the output of the numerical engines, e.g. UNTRIM*, so that the complete postprocessing benefits from the reduction.</P>
+An introduction how to collect DMQS metadata can be found [[Metadata_of_Model_results_in_Coastal_Engineering|here]].</P>
 <P>
-An introduction how to collect DMQS metadata can be found [[Metadata_of_Model_results_in_Coastal_Engineering|here]] (only in German).
+Performance: Most of the program's CPU time is used by basic IO routines. Since output is in the tested examples considerably slower than input a reduction of the output data, area and geophysical variables, makes NCCUTOUT run faster.
-|preprocessor=[[NCANALYSE]], [[NCCHUNKIE]], [[NCDELTA]], [[INSEL2IPDS]], [[UNTRIM2007]], [[UNTRIM2]]
+|preprocessor=[[NCANALYSE]], [[NCCHUNKIE]], [[NCDELTA]], [[NCDVAR]], [[NCMERGE]], [[UNTRIM2007]], [[UNTRIM2]]
-|postprocessor=[[DAVIT]], [[NCANALYSE]], [[NCAUTO]], [[NCCHUNKIE]], [[NCDELTA]], [[NCPLOT]], [[NC2TABLE]], [[PLOTPROFILZEIT]] und MATLAB
+|postprocessor=[[DAVIT]], [[NCAGGREGATE]], [[NCANALYSE]], [[NCAUTO]], [[NCCHUNKIE]], [[NCDELTA]], [[NCDVAR]], [[NCMERGE]], [[NCPLOT]], [[NC2TABLE]] and  [[MATLAB]]
 |language=Fortran95
 |add_software=---
-|contact_original=[mailto:peter.schade@baw.de P. Schade], [mailto:guenther.lang@baw.de G. Lang]
+|contact_original=P. Schade, G. Lang
-|contact_maintenance=[mailto:peter.schade@baw.de P. Schade], [mailto:guenther.lang@baw.de G. Lang]
+|contact_maintenance=[mailto:pos.proghome@baw.de Working group POS]
 |documentation=
 * Template Files:
-** template files available in '''$PROGHOME/examples/nccutout'''
+** general template files available in '''$PROGHOME/examples/nccutout'''
+** in test directory for hydro-dynamic [[UNTRIM2007]] results: '''$PROGHOME/examples/nccutout/run/utr2007/elbe/nccut'''
+** in test directory for hydro-dynamic [[UNTRIM2]] results: '''$PROGHOME/examples/nccutout/run/utr2009/easy/nccut'''
+** in test directory for [[NCANALYSE]] results: '''$PROGHOME/examples/nccutout/run/ncana/easy/nccut'''
 }}