NetCDF Operators (NCO)
http://nco.sourceforge.net/
Introduction and History
• Suite of Command Line Operators
• Designed to operate on netCDF/HDF files
• Each is a stand alone executable
• Very efficient for specific tasks
• Available for various computer architectures
NCO philosophy
•
•
•
•
Command line operators which perform useful,
scriptable operations on netCDF files.
Suite of operations useful in manipulating raw data
into a processed state.
Many command line switches enable frequently used
sequences of these operators tobe executed from a
shell script or batch file.
Finally, NCO was written to consume the absolute
minimum amount of system memory required to
perform a given job. The arithmetic operators are
extremely efficient!
NCO: a set of command-line tools
NCO is a command-line language. You use it by
typing an operator after the prompt (e.g., ‘$’)
following the syntax:
$ operator [options] input [output]
The operators (1)
• ncap2
• ncatted
• ncbo
• nces
• ncecat
• ncflint
netCDF Arithmetic Processor
netCDF Attribute Editor
netCDF Binary Operator
netCDF Ensemble Statistics
netCDF Ensemble Concatenator
netCDF File Interpolator
The operators (2)
• ncks
• ncpdq
• ncra
• ncrcat
• ncrename
• ncwa
netCDF Kitchen Sink
netCDF Permute Dimensions Quickly
netCDF Record Averager
netCDF Record Concatenator
netCDF Renamer
netCDF Weighted Averager
Getting help
To learn more about an operator read its
"man" page. E.g.:
$ man ncbo
Too many to present all
There are too many operators, and too much
functionality, to present a summary.
So let’s focus on some of the great stuff you might
want to do with NCO.
And in the meantime, there is plenty of detailed
documentation and examples at:
http://nco.sourceforge.net/#RTFM
View the data (ncks)
ncks will also give you a view of the data:
$ ncks example_data/ggas2014121200_00-18.nc
Summary of example_data/ggas2014121200_00-18.nc:
filetype = NC_FORMAT_CLASSIC, 0 groups (max. depth =
0), 4 dimensions (3 fixed, 1 record), 11 variables
(11 atomic-type, 0 non-atomic), 85 attributes (4
global, 0 group, 81 variable)
Root record dimension 0: name = time, size = 4
...
time[0]=0 surface[0]=0 latitude[0]=89.4628 longitude[0]=0 CI[0]=1
(0 - 1)
time[0]=0 surface[0]=0 latitude[0]=89.4628 longitude[1]=0.703125
CI[1]=1 (0 - 1)
Do quick extractions (ncks)
ncks will let you extract variables/subsets:
$ ncks -d longitude,5,15 -v CI
example_data/ggas2014121200_00-18.nc -O a.nc
The above command will extract the variable "CI"
over the subset longitude = [5:15] and write those
elements to the output file "a.nc".
Editing netCDF attributes - ncatted
ncatted edits attributes in a netCDF file. It can
append, create, delete, modify, and overwrite
attributes at both the “global” and “variable” level.
E.g.: Append the string Data version 2.0.\n to the
global attribute history:
$ ncatted -a history,global,a,c,'Data
version 2.0\n' in.nc
Editing netCDF attributes - ncatted
Change the value of the long_name attribute for
variable T from whatever it currently is to
“temperature”:
$ ncatted -a long_name,T,o,c,temperature in.nc
Delete all existing units attributes:
$ ncatted -a units,,d,, in.nc
ncbo – binary operations
The ncbo operator performs the binary operations
of file addition, subtraction, multiplication, division,
and broadcasting. E.g.:
ncbo –y add 1.nc 2.nc 3.nc
In the above, "-y" specifies the type of operation.
NOTE: "ncdiff" is available as a shortcut to typing
"ncbo -y sub" or "ncbo --op_typ=sub"
Concatenating files (ncrcat)
You can use ncrcat (NetCDF Record
Concatenater) to concatenate a set of files
together that share a common "record" dimension.
E.g. A set of four time steps that each have only a
single time:
$ ncrcat ggas20141201.nc ggas20141202.nc
ggas20141203.nc ggas20141204.nc -O a.nc
Concatenating files (ncecat)
Consider five realizations, 85a.nc, 85b.nc, 85c.nc,
85d.nc, 85e.nc of 1985 predictions from the same climate
model. You can use ncecat (NetCDF Ensemble
Concatenator) as follows:
$ ncecat 85?.nc 85_ens.nc
This glues the individual realizations together into the single
file, 85_ens.nc. If an input variable was dimensioned [lat,
lon], it will have dimensions [record, lat, lon] in the output
file. A restriction of ncecat is that the hyperslabs of the
processed variables must be the same from file to file.
Averaging records (ncra)
ncra (NetCDF Record Averager) averages record
variables across an arbitrary number of input-files.
The record dimension is, by default, retained as a
degenerate (size 1) dimension in the output
variables.
ncra weights each record (e.g., time slice) in the
input-files equally. (See "ncwa" for weighted
averaging).
Averaging records (ncra)
Average files 85.nc, 86.nc, … 89.nc along the
record dimension, and store the results in out.nc:
$ ncra 85.nc 86.nc 87.nc 88.nc 89.nc out.nc
Scripting multiple operators
One of the strengths of NCO is that you can write
complex processing chains by employing a series
of operators to produce your desired outputs.
Here is an example…
Chaining operators – example (1)
Say files 85_0112.nc and 86_0112.nc each contain 12
months of data. Compute the change in the monthly
averages from 1985 to 1986:
$ ncbo –y sub 86_0112.nc 85_0112.nc 86m85_0112.nc
Now we want to compute the monthly anomalies of T from
the yearly average of T for the year 1985. First we create
the 1985 average from the monthly data, which is stored
with the record dimension time:
$ ncwa -a time 85_0112.nc 85.nc
The "-a" option removes the time dimension from the
output file.
Chaining operators – example (2)
We are now ready to use ncbo to compute the anomalies
for 1985:
$ ncbo -y sub -v T 85_0112.nc 85.nc t_anm_85_0112.nc
Each of the 12 records in t_anm_85_0112.nc now
contains the monthly deviation of T from the annual mean
of T for each grid point.
Chaining operators – example (3)
Say we wish to compute the monthly gridpoint anomalies
from the zonal annual mean. A zonal mean is a quantity that
has been averaged over the longitudinal (or x) direction.
First we use ncwa to average over longitudinal direction
lon, creating 85_x.nc, the zonal mean of 85.nc. Then we
use ncbo to subtract the zonal annual means from the
monthly gridpoint data: :
$ ncwa -a lon 85.nc 85_x.nc
$ ncbo –y sub 85_0112.nc 85_x.nc tx_anm_85_0112.nc
Further information
NCO:
http://nco.sourceforge.net/nco.html
NCO User's Guide in many formats:
http://nco.sourceforge.net/#RTFM