NetCDF - Revision history

Ak3glang: /* Important NetCDF Utilities */ link to utility fi

2020-06-19T14:48:40Z

Important NetCDF Utilities: link to utility fi

← Older revision		Revision as of 14:48, 19 June 2020
Line 10:		Line 10:

	Important (helpful) NetCDF ''Utilities'' are:		Important (helpful) NetCDF ''Utilities'' are:
	* [~~http~~://www.unidata.ucar.edu/software/netcdf/~~docs~~/~~netcdf_utilities_guide~~.html~~#guide_ncdump~~ NCDUMP] create (selective) text representation of the contents of a NetCDF file;		* [https://www.unidata.ucar.edu/software/netcdf/workshops/2011/utilities/Ncdump.html NCDUMP] create (selective) text representation of the contents of a NetCDF file;
	* [~~http~~://www.unidata.ucar.edu/software/netcdf/~~docs~~/~~netcdf_utilities_guide~~.html~~#guide_nccopy~~ NCCOPY] (selective) copy an existing NetCDF file to another, change level of compression, change internal file structure (''File Chunking''); and		* [https://www.unidata.ucar.edu/software/netcdf/workshops/2011/utilities/Nccopy.html NCCOPY] (selective) copy an existing NetCDF file to another, change level of compression, change internal file structure (''File Chunking''); and
	* [~~http~~://www.unidata.ucar.edu/software/netcdf/~~docs~~/~~netcdf_utilities_guide~~.html~~#guide_ncgen~~ NCGEN] create NetCDF file from a CDL text file; optionally also C or FORTRAN code can be automatically generated.		* [https://www.unidata.ucar.edu/software/netcdf/workshops/2011/utilities/Ncgen.html NCGEN] create NetCDF file from a CDL text file; optionally also C or FORTRAN code can be automatically generated.
	A good overall view on netCDF is given in [http://www.unidata.ucar.edu/software/netcdf/docs/index.html NetCDF documentation].		A good overall view on netCDF is given in [http://www.unidata.ucar.edu/software/netcdf/docs/index.html NetCDF documentation].

imported>Lang Guenther: /* Literature */ Lang, G. (2018) added

2018-06-05T09:25:16Z

Literature: Lang, G. (2018) added

← Older revision		Revision as of 09:25, 5 June 2018
Line 30:		Line 30:

	[https://www.researchgate.net/publication/303822983_Uniform_post-processing_of_computational_results_based_on_UGRID_CF_NetCDF_files Lang, G. (2016) "Uniform post-processing of computational results based on UGRID CF netCDF files"], 13th International [[UNTRIM\|UnTRIM]] Users Workshop 2016, Villa Madruzzo, Italy, May 30th - June 1st ([[doi:10.13140/RG.2.1.5059.8000]]).		[https://www.researchgate.net/publication/303822983_Uniform_post-processing_of_computational_results_based_on_UGRID_CF_NetCDF_files Lang, G. (2016) "Uniform post-processing of computational results based on UGRID CF netCDF files"], 13th International [[UNTRIM\|UnTRIM]] Users Workshop 2016, Villa Madruzzo, Italy, May 30th - June 1st ([[doi:10.13140/RG.2.1.5059.8000]]).

			[https://www.researchgate.net/publication/325545134_A_few_remarks_on_chunked_IO_using_netCDF-4HDF5_Fortran_API Lang, G. (2018) "A few remarks on chunked I/O using netCDF-4/HDF5"], 15th International [[UNTRIM\|UnTRIM]] Users Workshop 2018, Villa Madruzzo, Italy, May 28th - May 30th ([[doi:10.13140/RG.2.2.31262.23368]]).

	[http://www.mdpi.com/2077-1312/2/1/194/htm Signell, R. P. und Snowden, D. P. (2014) ''Advances in a Distributed Approach for Ocean Model Data Interoperability''. J. Mar. Sci. Eng. 2014, 2, 194-208]. Describes also the benefits using UGRID CF metadata standard to store data in netCDF files.		[http://www.mdpi.com/2077-1312/2/1/194/htm Signell, R. P. und Snowden, D. P. (2014) ''Advances in a Distributed Approach for Ocean Model Data Interoperability''. J. Mar. Sci. Eng. 2014, 2, 194-208]. Describes also the benefits using UGRID CF metadata standard to store data in netCDF files.

imported>Lang Guenther: /* File Chunking */ NCCHUNKIE added

2018-04-23T07:39:39Z

File Chunking: NCCHUNKIE added

← Older revision		Revision as of 07:39, 23 April 2018
Line 17:		Line 17:
	==File Chunking==		==File Chunking==

	The chunk size of variables stored in a CF NetCDF file may have significant influence on read performance in case data have to be read along different dimensions, e.g. spatial versus time-series access. Chunk size can be individually tuned using the NetCDF API. As a simple alternative, already helpful in many situations, you can also make use of the [http://www.unidata.ucar.edu/software/netcdf/docs/netcdf_utilities_guide.html#guide_nccopy NCCOPY] program. For further informations about chunking please read the following informations:		The chunk size of variables stored in a CF NetCDF file may have significant influence on read performance in case data have to be read along different dimensions, e.g. spatial versus time-series access. Chunk size can be individually tuned using the NetCDF API. As a simple alternative, already helpful in many situations, you can also make use of the [http://www.unidata.ucar.edu/software/netcdf/docs/netcdf_utilities_guide.html#guide_nccopy NCCOPY] or the [[NCCHUNKIE]] program. For further informations about chunking please read the following informations:
	* [http://www.unidata.ucar.edu/blogs/developer/en/entry/chunking_data_why_it_matters ''Chunking Data - Why it matters'']; und		* [http://www.unidata.ucar.edu/blogs/developer/en/entry/chunking_data_why_it_matters ''Chunking Data - Why it matters'']; und
	* [http://www.unidata.ucar.edu/blogs/developer/en/entry/chunking_data_choosing_shapes ''Chunking Data - Choosing shapes''].		* [http://www.unidata.ucar.edu/blogs/developer/en/entry/chunking_data_choosing_shapes ''Chunking Data - Choosing shapes''].

imported>Lang Guenther: /* How to acknowledge Unidata */ DOI added

2018-02-09T13:58:11Z

How to acknowledge Unidata: DOI added

← Older revision		Revision as of 13:58, 9 February 2018
Line 46:		Line 46:

	* Unidata, (year): Package name version number [software]. Boulder, CO: UCAR/Unidata Program Center. Available from URL-to-software-page.		* Unidata, (year): Package name version number [software]. Boulder, CO: UCAR/Unidata Program Center. Available from URL-to-software-page.
			===DOI===

			* The registered Digital Object Identifier for all versions of netCDF software is [http://doi.org/10.5065/D6H70CW6 http://doi.org/10.5065/D6H70CW6].

	==Where is NetCDF used?==		==Where is NetCDF used?==

imported>Lang Guenther: /* Automatic verification of value range */ Link to machine epsilon

2017-11-28T21:26:45Z

Automatic verification of value range: Link to machine epsilon

← Older revision		Revision as of 21:26, 28 November 2017
Line 67:		Line 67:
	exist. $PROGHOME/cfg/dmqs/bounds/bounds_verify.dat contains typical valid value range data for all existing classes of variables.		exist. $PROGHOME/cfg/dmqs/bounds/bounds_verify.dat contains typical valid value range data for all existing classes of variables.

	The result of all comparisons done for actual value range vs. allowed value range is stored in a (printer) SDR file. These informations indicate, whether variables ly inside or outside the accepted valid value range. A fast overview is obtained by means of '''grep Pruefergebnis''' applied to the SDR file.		The result of all comparisons done for actual value range vs. allowed value range is stored in a (printer) SDR file. These informations indicate, whether variables ly inside or outside the accepted valid value range. A fast [[overview]] is obtained by means of '''grep Pruefergebnis''' applied to the SDR file.

	For real numbers machine epsilon can be obtained e. g. from (Fortran) EPSILON:		For real numbers [https://en.wikipedia.org/wiki/Machine_epsilon machine epsilon] can be obtained e. g. from (Fortran) EPSILON:
	* single precision data: approx. 1.2E-07;		* single precision data: approx. 1.2E-07;
	* double precision data: approx. 2.2E-16.		* double precision data: approx. 2.2E-16.

imported>Lang Guenther: /* Automatic verification of value range */ section on machine epsilon clarified

2017-11-27T16:06:03Z

Automatic verification of value range: section on machine epsilon clarified

← Older revision		Revision as of 16:06, 27 November 2017
Line 57:		Line 57:
	===Attribute ''actual_range''===		===Attribute ''actual_range''===

	This attribute stores the actual value range for (geophysical) variables. Execution of '''ncdump -h''' delivers all metadata stored in a NetCDF file. This output can be searched using '''grep''' to retrieve '''actual_range'''. In doing so a fast and simple overview is obtained, whether the actual range of a variable is outside or inside a meaningful value range.		This attribute stores the actual value range for (geophysical) variables. Execution of '''ncdump -h''' delivers all metadata stored in a NetCDF file. This output can be searched using '''grep''' to retrieve '''actual_range'''. In doing so a fast and simple [[overview]] is obtained, whether the actual range of a variable is outside or inside a meaningful value range.

	===Automatic verification of value range===		===Automatic verification of value range===
Line 69:		Line 69:
	The result of all comparisons done for actual value range vs. allowed value range is stored in a (printer) SDR file. These informations indicate, whether variables ly inside or outside the accepted valid value range. A fast overview is obtained by means of '''grep Pruefergebnis''' applied to the SDR file.		The result of all comparisons done for actual value range vs. allowed value range is stored in a (printer) SDR file. These informations indicate, whether variables ly inside or outside the accepted valid value range. A fast overview is obtained by means of '''grep Pruefergebnis''' applied to the SDR file.

	For real numbers ~~the following tolerances ar accepted~~:		For real numbers machine epsilon can be obtained e. g. from (Fortran) EPSILON:
	* single precision data: approx. 1.2E-06;		* single precision data: approx. 1.2E-07;
	* double precision data: approx. 2.2E-~~015~~.		* double precision data: approx. 2.2E-16.
	~~These tolerances are identical to ten times~~ the ~~value provided~~ by ~~Fortran's~~ EPSILON ~~intrinsic function~~.		For real data the tolerance used is given by 2 * EPSILON * ABS(data).

	=Global Attributes=		=Global Attributes=

imported>Lang Guenther: /* Where is NetCDF used? */ informations about quality assurance

2017-11-23T11:06:03Z

Where is NetCDF used?: informations about quality assurance

@@ Line 29: / Line 29: @@
 [http://visa.lab.asu.edu/web/wp-content/uploads/2015/12/S08210-3.pdf Biookaghazadeh, Saman, et al. (2015) ''Enabling scientific data storage and processing on big-data systems''. Big Data (Big Data), 2015 IEEE International Conference on. IEEE, 2015] Use of data stored in netCDF files in big data analysis system [http://hadoop.apache.org Hadoop].
-[https://www.researchgate.net/publication/303822983_Uniform_post-processing_of_computational_results_based_on_UGRID_CF_NetCDF_files Lang, G. (2016) "Uniform post-processing of computational results based on UGRID CF netCDF files"], 13th International UnTRIM Users Workshop 2016, Villa Madruzzo, Italy, May 30th - June 1st ([[doi:10.13140/RG.2.1.5059.8000]]).
+[https://www.researchgate.net/publication/303822983_Uniform_post-processing_of_computational_results_based_on_UGRID_CF_NetCDF_files Lang, G. (2016) "Uniform post-processing of computational results based on UGRID CF netCDF files"], 13th International [[UNTRIM|UnTRIM]] Users Workshop 2016, Villa Madruzzo, Italy, May 30th - June 1st ([[doi:10.13140/RG.2.1.5059.8000]]).
 [http://www.mdpi.com/2077-1312/2/1/194/htm Signell, R. P. und Snowden, D. P. (2014) ''Advances in a Distributed Approach for Ocean Model Data Interoperability''. J. Mar. Sci. Eng. 2014, 2, 194-208]. Describes also the benefits using UGRID CF metadata standard to store data in netCDF files.
@@ Line 49: / Line 49: @@
 ==Where is NetCDF used?==
-For an overview please visit [http://www.unidata.ucar.edu/software/netcdf/usage.html ''Where is NetCDF used?''].
+For an [[overview]] please visit [http://www.unidata.ucar.edu/software/netcdf/usage.html ''Where is NetCDF used?''].
 =Global Attributes=

imported>Lang Guenther: /* Grids */ title of section modified

2016-10-11T07:34:39Z

Grids: title of section modified

← Older revision		Revision as of 07:34, 11 October 2016
Line 53:		Line 53:
	=Global Attributes=		=Global Attributes=
	* [[NetCDF global attributes]]		* [[NetCDF global attributes]]
	=Grids=		=Locations, Profiles and Grids=
	* [[NetCDF multiple locations]]: several (point) locations, e. g. equivalent to contents of file [[LOCATION_GRID.DAT\|location_grid.dat]];		* [[NetCDF multiple locations]]: several (point) locations, e. g. equivalent to contents of file [[LOCATION_GRID.DAT\|location_grid.dat]];
	* [[NetCDF multiple profiles]]: several longitudinal and cross-sectional profiles, e. g. equivalent to contents of file [[PROFIL05.BIN\|profil05.bin]];		* [[NetCDF multiple profiles]]: several longitudinal and cross-sectional profiles, e. g. equivalent to contents of file [[PROFIL05.BIN\|profil05.bin]];

imported>Lang Guenther: /* Literature */ literature added

2016-06-07T18:57:39Z

Literature: literature added

← Older revision		Revision as of 18:57, 7 June 2016
Line 28:		Line 28:

	[http://visa.lab.asu.edu/web/wp-content/uploads/2015/12/S08210-3.pdf Biookaghazadeh, Saman, et al. (2015) ''Enabling scientific data storage and processing on big-data systems''. Big Data (Big Data), 2015 IEEE International Conference on. IEEE, 2015] Use of data stored in netCDF files in big data analysis system [http://hadoop.apache.org Hadoop].		[http://visa.lab.asu.edu/web/wp-content/uploads/2015/12/S08210-3.pdf Biookaghazadeh, Saman, et al. (2015) ''Enabling scientific data storage and processing on big-data systems''. Big Data (Big Data), 2015 IEEE International Conference on. IEEE, 2015] Use of data stored in netCDF files in big data analysis system [http://hadoop.apache.org Hadoop].

			[https://www.researchgate.net/publication/303822983_Uniform_post-processing_of_computational_results_based_on_UGRID_CF_NetCDF_files Lang, G. (2016) "Uniform post-processing of computational results based on UGRID CF netCDF files"], 13th International UnTRIM Users Workshop 2016, Villa Madruzzo, Italy, May 30th - June 1st ([[doi:10.13140/RG.2.1.5059.8000]]).

	[http://www.mdpi.com/2077-1312/2/1/194/htm Signell, R. P. und Snowden, D. P. (2014) ''Advances in a Distributed Approach for Ocean Model Data Interoperability''. J. Mar. Sci. Eng. 2014, 2, 194-208]. Describes also the benefits using UGRID CF metadata standard to store data in netCDF files.		[http://www.mdpi.com/2077-1312/2/1/194/htm Signell, R. P. und Snowden, D. P. (2014) ''Advances in a Distributed Approach for Ocean Model Data Interoperability''. J. Mar. Sci. Eng. 2014, 2, 194-208]. Describes also the benefits using UGRID CF metadata standard to store data in netCDF files.

imported>Lang Guenther: /* Purpose of these BAWiki Pages */ text modified

2016-04-16T12:48:12Z

Purpose of these BAWiki Pages: text modified

← Older revision		Revision as of 12:48, 16 April 2016
Line 3:		Line 3:
	==Purpose of these BAWiki Pages==		==Purpose of these BAWiki Pages==

	These BAWiki pages do describe all NetCDF conventions required to store baw-specific data in NetCDF data files (see [http://www.unidata.ucar.edu/software/netcdf/ ''network common data form'']). I. e. all ''local'' conventions are listed, which go beyond the international agreed-upon [http://cf-convention.github.io/ CF-metadata convention].		These BAWiki pages do describe all NetCDF conventions required to store baw-specific data in NetCDF data files (see [http://www.unidata.ucar.edu/software/netcdf/ ''network common data form'']). I. e. all ''local'' conventions are listed, which go beyond the international agreed-upon [http://cf-convention.github.io/ CF-metadata convention]. In many situations where the international agreed-upon CF conventions are insufficient, essentially the ''Unstructured Grid Metadata Conventions for Scientific Datasets'' (UGRID Conventions) published on [http://ugrid-conventions.github.io/ugrid-conventions/#ugrid-conventions-v10 GITHUB] are used. Some further widely spread templates known are the [http://www.nodc.noaa.gov/data/formats/netcdf/ NODC NetCDF Templates]. The NODC data center has been recently merged with other data centers and is now part of [http://www.ncei.noaa.gov/ ''National Centers for Environmental Information'' (NCEI)].

	In cases where the international agreed-upon CF conventions are insufficient, essentially the ''Unstructured Grid Metadata Conventions for Scientific Datasets'' (UGRID Conventions) published on [http://ugrid-conventions.github.io/ugrid-conventions/#ugrid-conventions-v10 GITHUB] are used.		The BAW instance of a NetCDF file developed since 2010 is a file of type [[CF-NETCDF.NC]]. Since version NetCDF-4.0 HDF (''Hierarchical Data File'', see [http://www.hdfgroup.org/HDF5/ HDF5 Group]) is used as the underlying file format. Due to the use of HDF concepts like online compression of data stored in NetCDF files is supported as well as chunking of variables to balance read performance in case of different access to data, e.g. time-series vs. synoptic data set access.

	~~Further activities related with OpenDAP to extract a selection of data defined on unstructured grids can be found on e. g. [http://docs.opendap.org/index.php/OPULS:_UGrid_Subsetting OPULS].~~

	Some further widely spread templates known are the [http://www.nodc.noaa.gov/data/formats/netcdf/ NODC NetCDF Templates]. The NODC data center has been recently merged with other data centers and is now part of [http://www.ncei.noaa.gov/ ''National Centers for Environmental Information'' (NCEI)].

	The BAW instance of a NetCDF file developed since 2010 is a file of type [[CF-NETCDF.NC]].

	Since version NetCDF-4.0 HDF (''Hierarchical Data File'', see [http://www.hdfgroup.org/HDF5/ HDF5 Group]) is used as the underlying file format. Due to the use of HDF concepts like online compression of data stored in NetCDF files is supported as well as chunking of variables to balance read performance in case of different access to data, e.g. time-series vs. synoptic data set access.

	==Important NetCDF Utilities==		==Important NetCDF Utilities==