Actions

NCCHUNKIE: Difference between revisions

From BAWiki

imported>Schade Peter
mNo edit summary
(somelinks to IZW-Campus were updated)
 
(10 intermediate revisions by the same user not shown)
Line 2: Line 2:
|name_de=NCCHUNKIE
|name_de=NCCHUNKIE
|name=NCCHUNKIE
|name=NCCHUNKIE
|version=April 2018
|version=April 2022
|version_descr=May 2018
|version_descr=September 2022
|catchwords=
|catchwords=
data conversion<br />
data conversion<br />
postprocessor<br />
postprocessor<br />
automatic adjustment of number of data READ to chunk size of input data <br />
automatic adjustment of number of data READ to chunk size of input data <br />
automatic computation of chunk sizes for result variables to support orthogonal data access
automatic computation of chunk sizes for result variables to support orthogonal data access <br />
parrallelization (''collective'' IO) using [http://de.wikipedia.org/wiki/Message_Passing_Interface MPI]<br />


  Acknowledgment: ''This project took advantage of netCDF software developed by UCAR/Unidata ([http://www.unidata.ucar.edu/software/netcdf/ www.unidata.ucar.edu/software/netcdf/]).''
  Acknowledgment: ''This project took advantage of netCDF software developed by UCAR/Unidata ([http://www.unidata.ucar.edu/software/netcdf/ www.unidata.ucar.edu/software/netcdf/]).''
Line 17: Line 18:
# Resulting chunks sizes lie somewhere between ''Disc Block Size'' and ''Chunk Buffer Size'';  
# Resulting chunks sizes lie somewhere between ''Disc Block Size'' and ''Chunk Buffer Size'';  
# Online compression is used during storage of data (low level of compression used, level 1);
# Online compression is used during storage of data (low level of compression used, level 1);
# A netCDF-4 file in ''classic model format'' is created (which is a netCDF compatible HDF5 format);
# A netCDF-4 file is created (serial version creates NetCDF4 ''classic model format'');
# Parameters ''cache size'' and ''cache nelems'' used within netCDF-4 API are automatically determined.
# Parameters ''cache size'' and ''cache nelems'' used within netCDF-4 API are automatically determined.


Line 23: Line 24:


|inputfiles=
|inputfiles=
# No input steering data file required (interactive specification of file name);
# No input steering data file required (parameters in command line or interactive specification);
# '''UGRID CF NetCDF data set''' (file type [[CF-NETCDF.NC]]).
# '''UGRID CF NetCDF data set''' (file type [[CF-NETCDF.NC]]).


Line 34: Line 35:
Some concepts published in [https://support.hdfgroup.org/pubs/papers/2008-06_netcdf4_perf_report.pdf https://support.hdfgroup.org/pubs/papers/2008-06_netcdf4_perf_report.pdf] were used.
Some concepts published in [https://support.hdfgroup.org/pubs/papers/2008-06_netcdf4_perf_report.pdf https://support.hdfgroup.org/pubs/papers/2008-06_netcdf4_perf_report.pdf] were used.


|preprocessor=[[DATACONVERT]], [[GRIDCONVERT]], [[NCANALYSE]], [[NCAGGREGATE]], [[NCCUTOUT]], [[NCDELTA]], [[NetCDF Operators]], [[UNTRIM2007]], [[UNTRIM2]]
|preprocessor=[[DATACONVERT]], [[GRIDCONVERT]], [[NCANALYSE]], [[NCAGGREGATE]], [[NCCUTOUT]], [[NCDELTA]], [[NCDVAR]], [[NCMERGE]], [[NetCDF Operators]], [[UNTRIM2007]], [[UNTRIM2]]
|postprocessor=[[NCANALYSE]], [[NCAGGREGATE]], [[NCDELTA]], [[NCCHUNKIE]], [[NCCUTOUT]], [[NC2TABLE]], [[NetCDF Operators]]
|postprocessor=[[NCANALYSE]], [[NCAGGREGATE]], [[NCDELTA]], [[NCDVAR]], [[NCMERGE]], [[NCCHUNKIE]], [[NCCUTOUT]], [[NC2TABLE]], [[NetCDF Operators]]
|language=Fortran95
|language=Fortran95
|add_software=---
|add_software=---
|contact_original=[mailto:guenther.lang@baw.de G. Lang]
|contact_original=G. Lang
|contact_maintenance=[mailto:guenther.lang@baw.de G. Lang], [mailto:susanne.spohr@baw.de S. Spohr]
|contact_maintenance=[mailto:pos.proghome@baw.de Working group POS]
|documentation=
|documentation=
No template files available.
* [https://izw-campus.baw.de/ ''IZW-Campus''] (Podcast, available in German only)
** 2020-11-02: [https://izw-campus.baw.de/goto.php?target=cat_1832&client_id=iliasclient ''Chunking und NCCHUNKIE''];
** 2019-08-02: [https://izw-campus.baw.de/goto.php?target=cat_1831&client_id=iliasclient ''Anmerkungen zur Chunked I/O mit NetCDF-4 / HDF5''].
* Template file(s):
** No template files available.
}}
}}

Latest revision as of 08:04, 7 November 2022

Basic Information

Name of Program

NCCHUNKIE

Version-Date

April 2022

Description-Date

September 2022

Catchwords

data conversion
postprocessor
automatic adjustment of number of data READ to chunk size of input data
automatic computation of chunk sizes for result variables to support orthogonal data access
parrallelization (collective IO) using MPI

Acknowledgment: This project took advantage of netCDF software developed by UCAR/Unidata (www.unidata.ucar.edu/software/netcdf/).

Short Description of Functionality

Program NCCHUNKIE can be used to chunk data stored in cf-netcdf.nc files:

  1. Chunk sizes are computed automatically, and all dimensions are chunked to support orthogonal data access;
  2. Resulting chunks sizes lie somewhere between Disc Block Size and Chunk Buffer Size;
  3. Online compression is used during storage of data (low level of compression used, level 1);
  4. A netCDF-4 file is created (serial version creates NetCDF4 classic model format);
  5. Parameters cache size and cache nelems used within netCDF-4 API are automatically determined.

Just in case HDF error is detected during read of a data record, the program tries to reconstruct the wanted data set from adjacent (in time) records for the same variable. This type of repair works for time dependent variables only.

Input-Files

  1. No input steering data file required (parameters in command line or interactive specification);
  2. UGRID CF NetCDF data set (file type CF-NETCDF.NC).

Output-Files

  1. UGRID CF NetCDF data set (file type CF-NETCDF.NC);
  2. informative printer file of program execution (file type NCCHUNKIE.sdr) with informations related to program execution, required time for READ and WRITE of data, effective data transfer rates.
  3. trace of program execution (file type NCCHUNKIE.trc)

Methodology

Some concepts published in https://support.hdfgroup.org/pubs/papers/2008-06_netcdf4_perf_report.pdf were used.

Program(s) to run before this Program

DATACONVERT, GRIDCONVERT, NCANALYSE, NCAGGREGATE, NCCUTOUT, NCDELTA, NCDVAR, NCMERGE, NetCDF Operators, UNTRIM2007, UNTRIM2

Program(s) to run after this Program

NCANALYSE, NCAGGREGATE, NCDELTA, NCDVAR, NCMERGE, NCCHUNKIE, NCCUTOUT, NC2TABLE, NetCDF Operators

Additional Information

Language

Fortran95

Additional software

---

Original Version

G. Lang

Maintenance

Working group POS

Documentation/Literature

back to Program Descriptions

Overview

Retrieved from "http://wiki.baw.de/en/index.php?title=NCCHUNKIE&oldid=15321"