IDL Tips for IMAGE Mission Investigators
Rich Baldwin -- Raytheon ITSS
ABSTRACT
The IMAGE Mission Data Systems will be using IDL as a major tool to access
derive and display data products. An overview of IDL's I/O and general
capabilities will be reviewed. I/O issues for cdf files will be discussed
and presented. Examples of the IDL CDAW library will be presented along
with a brief overview of IDL's INSIGHT tool. The general focus of the
discussion will be to assist the IMAGE investigator in using tools to access
IMAGE data as quickly and as easily as possible.
Introduction
IDL, the Interactive Data Language, has proven to be a powerful research tool
in many technical areas. It is ideal for data analysis, visualization, and
cross-platform application development. IDL combines interactive analysis and
display tools with many common scientific database formats. CDF, the
Common Data Format can be used through IDL with a suite of procedures to access
variables, global and variable attributes, and other ancillary cdf information.
Some of these functions will be demonstrated below.
The SPDF, Space Physic Data Facility, has combined the power of IDL and the WWW
to produce CDAWeb. This interactive web engine displays space physics data
"on-the-fly". The base of CDAWeb is a library of IDL routines, CDAWlib.
We also will be looking at some examples which use these procedures.
UDF is another data format which will prominently be used for the IMAGE Mission.
Development of tools which will read UDF files through IDL are still being
developed.
IDL has another product called INSIGHT which is a GUI product that can help
investigators quickly display their data once the data is in a simple IDL form.
Setup
An IDL License needs to be obtained. Once the IDL source has been installed.
Unix users will want to run
source /usr/local/rsi/idl_5.2/bin/idl_setup
at the UNIX prompt or in their .cshrc file. This will define the environment
variables and aliases required by IDL.
Windows users should check the path under the "properties" dialog to make sure
that it references the location of the IDL libraries.
Getting Started with IDL and CDF
There are many courses offered through RSI for IDL training which involve the
basics as well as advanced topics such as Widgets and OO programming, but for
our purposes a simple online tutorial basics_1 or basics_2 is sufficient
for starters. Since CDF is built into IDL, there is no additional
setup for the CDF required. Help on the setup and use of CDF tools is also available.
Our first example idl procedure opens a cdf file and reads some global cdf
information.
pro example1a
cdfid = cdf_open('po_k0_uvi_19980317_v01.cdf') ; Open cdf
global_struct = cdf_inquire(cdfid) ; Grab cdf info.
help, /struct, global_struct ; Print info.
end
OR
pro example1a
cdfid = cdf_open('im_k0_euv_19951018_v02.cdf') ; Open cdf
global_struct = cdf_inquire(cdfid) ; Grab cdf info.
help, /struct, global_struct ; Print info.
end
Example Run/Output
Now, we will display some polar imager data.
pro example1b
cdfid = cdf_open('po_k0_uvi_19980317_v01.cdf') ; Open cdf
cdf_varget, cdfid, 'IMAGE_DATA', data ; Read variable
help, data ; Print variable information
print, size(data) ; Print size of variable
img1=data(0,*,*) ; Extract a slice of the array
loadct, 39 ; Load a color table
tv, img1 ; Display image
end
Example Run/Output
OR with IMAGE EUV
pro example1b
cdfid = cdf_open('/ncf/rumba1/istp/image/euv/1995/im_k0_euv_19951018_v02.cdf')
cdf_varget, cdfid, 'IMAGE', data
help, data
print, size(data)
img1=data
loadct, 39
tv, img1
end
Example Run/Output
Let's take data from another cdf and run it through CDAWlib's auroral_image
procedure, and then write a gif file.
pro example1c
set_plot, 'Z'
cdfid = cdf_open('de_uv_sai_19811029_v01.cdf') ; Open CDF
cdf_varget, cdfid, 'Image_Counts', data, rec_start=12 ; Read variable
cdf_varget, cdfid, 'GeoLat_Dec', lat , rec_start=12
cdf_varget, cdfid, 'GeoLng_RAs', lon , rec_start=12
cdf_close, cdfid
print, size(data)
print, size(lat)
print, size(lon)
; Isolate longitude points between -180 and +180
ncd= where(lon gt 180.0,ncdn)
if(ncdn ne 0) then lon(ncd)=lon(ncd)-360.0
loadct, 39
CENTERLATLON=[80,270]
; Call auroral_image procedure
auroral_image, data, lon, lat, /continent,/label, method="PL",/grid, $
centerLonLat=CENTERLATLON,/CENTERPOLE, /nocolorbar,proj=6,$
fillValue=-1.0e+31,rangeLonLat=[40,-180.,90.,180.], $
status=status,charsize=2.0
print, status
write_gif, 'test.gif', tvrd()
end
Example Run/Output
GIF image
This same example can be run with a LENA cdf (im_k0_len_19951019_v01.cdf).
Read the IMAGE and Geo_POSITION variables, convert the geographic position
vectors to latitudes and longitudes.
Let's try our luck at making movies. The example below reads in a cdf, grabs
several frames of data by slicing the data variable. It is important to
note that MPEG files have 3 color channels each mapping the pixels from the
data array to construct the image.
pro example1d
set_plot, 'Z' ; Set Z output buffer
cdfid = cdf_open('de_uv_sai_19811029_v01.cdf') ; Open cdf
; Setup variable arrays
bdata=fltarr(21,121,150);
blon=fltarr(21,121,150);
blat=fltarr(21,121,150);
; Loop through reading and storing variables
for i=0, 20 do begin
cdf_varget, cdfid, 'Image_Counts', data, rec_start=i
cdf_varget, cdfid, 'GeoLat_Dec', lat , rec_start=i
cdf_varget, cdfid, 'GeoLng_RAs', lon , rec_start=i
; Convert longitudes to -180 to 180
ncd= where(lon gt 180.0,ncdn)
if(ncdn ne 0) then lon(ncd)=lon(ncd)-360.0
; Set pixel value to black for any point below 40 degrees latitude
; Set latitude to a fill value for any point below 40 degrees latitude
wlat=where(lat lt 40,wlatc)
if(wlatc gt 0) then data(wlat)=0
if(wlatc gt 0) then lat(wlat)=-1.0e+31
; Store data
bdata(i,*,*)=data
blon(i,*,*)=lon
blat(i,*,*)=lat
endfor
cdf_close, cdfid
print, size(bdata)
print, size(blat)
print, size(blon)
loadct, 39 ; Load color table
CENTERLATLON=[80,270]
mpegID=mpeg_open([640,480]) ; Open an MPEG file
; Pull data back out of the arrays and plot data to Z buffer w/ auroal_image
for i=0,20 do begin
data=bdata(i,*,*)
lat=blat(i,*,*)
lon=blon(i,*,*)
auroral_image, data, lon, lat, /continent,/label, method="PL",/grid, $
centerLonLat=CENTERLATLON,/CENTERPOLE, /nocolorbar,proj=6,$
fillValue=-1.0e+31,rangeLonLat=[40,-180.,90.,180.], $
status=status,charsize=2.0
; Grab each image out of the Z buffer
image = tvrd()
; Load the display color translation tables. r,g and b range from 0 to 255.
tvlct, r,g,b, /get
; Set the colors for each channel
ii=bytarr(3,640,480)
ii(0,*,*)=r[image]
ii(1,*,*)=g[image]
ii(2,*,*)=b[image]
; Store the frame and index
mpeg_put, mpegID, IMAGE=ii, FRAME=i, ORDER=1
erase ; erase Z-buffer
endfor
mpeg_save, mpegID, FILENAME='example1d.mpg'
mpeg_close, mpegID
end
Example Run/Output
MPEG Movie
Gettings started with CDAWlib
Here is a simple example of an polar image plot. Notice that the polar UVI
master CDF is used here to insure global and variable attributes comply with
ISTP guidelines. READ_MYCDF and PLOTMASTER are two cornerstone pieces of
the CDAWlib library.
pro example2a
; Note a restore or compile of CDAWlib needs to be executed prior
start='1998/03/14 07:36:00'
stop='1998/03/14 08:40:00'
a=strarr(2)
a(0)='/ncf/rumba1/istp/0MASTERS/po_k0_uvi_00000000_v01.cdf'
a(1)='/ncf/rumba1/istp/polar/uvi/1998/po_k0_uvi_19980314_v01.cdf'
buf1 = read_myCDF(['GEOD_IMAGE'], a, /NODATASTRUCT, TSTART=start, TSTOP=stop, /DEBUG)
s = plotmaster(buf1, /AUTO, /CDAWEB, PID=28241, TSTART=start, TSTOP=stop, $
/GIF, OUTDIR='~/public_html/',/SMOOTH, /SLOW, /DEBUG, /NONOISE)
end
Example Run/Output
Gif Image
Data can also be listed in tabular form using CDAWlib. Below is such an
example.
pro example2b
start='1995/01/01 00:00:00'
stop='1995/01/02 23:00:00'
a=strarr(3)
a(0)='/ncf/rumba1/istp/0MASTERS/l0_k0_mpa_00000000_v01.cdf'
a(1)='/ncf/rumba1/istp/lanl/90_mpa/1995/l0_k0_mpa_19950101_v01.cdf'
a(2)='/ncf/rumba1/istp/lanl/90_mpa/1995/l0_k0_mpa_19950102_v01.cdf'
buf = read_myCDF(['dens_lop','sc_pos_geo'], a, /NODATASTRUCT, TSTART=start, TSTOP=stop)
s = LIST_mystruct(buf, TSTART=start, TSTOP=stop, /NOVATT, /NORV)
print, s
end
Example Run/Output
listing
OR
pro example2b
start='1995/10/18 00:00:00'
stop='1995/10/19 00:00:00'
a=strarr(2)
a(0)='/ncf/rumba1/istp/0MASTERS/im_k0_euv_00000000_v01.cdf'
a(1)='/ncf/rumba1/istp/image/euv/1995/im_k0_euv_19951018_v02.cdf'
buf = read_myCDF(['Roll_Angle'], a, /NODATASTRUCT, TSTART=start, TSTOP=stop)
s = LIST_mystruct(buf, TSTART=start, TSTOP=stop, /NOVATT, /NORV)
print, s
end
Example Run/Output
listing
READ_MYCDF takes as its arguments an array of variable names, an array of
cdfs, and a start and stop time. READ_MYCDF returns an IDL structure which
is simply a container of other data types including other structures.
WRITE_MYCDF used below will take an idl sturcture and write out a CDF.
For example:
IDL>restore, '/home/rumba/cdaweb/lib/spdflib5.dat'
IDL>start='1996/01/13 00:00:00'
IDL>stop='1996/01/14 00:00:00'
IDL>a=strarr(3)
IDL>a(0)='/ncf/rumba1/istp/0MASTERS/cn_k0_mari_00000000_v01.cdf'
IDL>a(1)='/ncf/rumba1/istp/canopus/mari_mag/1996/cn_k0_mari_19960113_v01.cdf'
IDL>a(2)='/ncf/rumba1/istp/canopus/mari_mag/1996/cn_k0_mari_19960114_v01.cdf'
IDL>buf1 = read_myCDF(['CU','CL'], a, TSTART=start, TSTOP=stop)
IDL> help, /struct, buf1
** Structure <401cde88>, 3 tags, length=2128, refs=1:
CU STRUCT -> Array[1]
CL STRUCT -> Array[1]
EPOCH STRUCT -> Array[1]
IDL> help, /struct buf1.cu
** Structure <401d4048>, 43 tags, length=704, refs=2:
VARNAME STRING 'CU'
PROJECT STRING 'ISTP>International Solar-Terrestrial Physics'
DISCIPLINE STRING 'Space Physics>Ionospheric Science'
SOURCE_NAME STRING 'CANOPUS>Canadian Auroral Network Open Program'...
DATA_TYPE STRING 'K0>Key Parameter'
DESCRIPTOR STRING 'MARI>Magnetometer and Riometer Array'
DATA_VERSION STRING '1'
TITLE STRING 'MARI Magnetometer Key Parameter Data 18-Aug-1'...
TEXT STRING Array[6]
MODS STRING 'Created 19-AUG-1994'
ADID_REF STRING 'NSSD0119'
LOGICAL_FILE_ID STRING 'CN_K0_MARI_19940818_V01'
LOGICAL_SOURCE STRING 'CN_K0_MARI'
LOGICAL_SOURCE_DESCRIPTION
STRING = 'CANOPUS MARI Magnetometer Key Parameters'
PI_NAME STRING 'G. Rostoker'
PI_AFFILIATION STRING 'U. Alberta'
MISSION_GROUP STRING 'Ground-Based Investigations>CANOPUS'
INSTRUMENT_TYPE STRING 'Ground-Based Magnetometers, Riometers, Sounde'...
TEXT_SUPPLEMENT_1
STRING = ''
FIELDNAM STRING 'CU index'
CATDESC STRING 'Local Auroral Electrojet index, Upper bound, '...
VALIDMIN FLOAT 0.00000
VALIDMAX FLOAT 5000.00
SCALEMIN FLOAT 0.00000
SCALEMAX FLOAT 5000.00
UNITS STRING 'nT'
UNIT_PTR STRING ''
DEPEND_0 STRING 'Epoch'
DEPEND_1 STRING ''
LABLAXIS STRING 'Ejet Up, Cu'
LABL_PTR_1 STRING ''
MONOTON STRING ''
DICT_KEY STRING 'index'
FORMAT STRING 'G13.6'
FORM_PTR STRING ''
FILLVAL FLOAT -1.00000e+31
VAR_TYPE STRING 'data'
AVG_TYPE STRING ''
DISPLAY_TYPE STRING 'time_series'
VAR_NOTES STRING 'Local equivalent to AU index, but computed fr'...
CDFTYPE STRING 'CDF_REAL4'
IDLSIZE LONG Array[4]
DAT FLOAT Array[1440]
IDL> print, buf1.cu.dat
IDL> .run write_mycdf.pro
IDL> dat3 = buf1.cu.dat*3.0
IDL> buf1.cu.dat=dat3
IDL> s = write_mycdf(buf1,'new.cdf')
IDL> nu_cu = buf1.cu
IDL> nu_cu.varname='NU_CU'
IDL> nu_cu.dat= dat3
IDL> new_buf1 = create_struct('EPOCH',buf1.epoch,'CU',buf1.cu,'CL',buf1.cl,'NU_CU',nu_cu)
IDL> help, /struct, new_buf1
** Structure <4009dd48>, 4 tags, length=31536, refs=1:
EPOCH STRUCT -> Array[1]
CU STRUCT -> Array[1]
CL STRUCT -> Array[1]
NU_CU STRUCT -> Array[1]
IDL> s= write_mycdf(new_buf1,'nu_cu.cdf')
Multiple data sets represented by IDL structures can be read and feed into
PLOTMASTER.
pro example2c
start='1996/01/13 00:00:00'
stop='1996/01/14 00:00:00'
a=strarr(3)
a(0)='/ncf/rumba1/istp/0MASTERS/cn_k0_mari_00000000_v01.cdf'
a(1)='/ncf/rumba1/istp/canopus/mari_mag/1996/cn_k0_mari_19960113_v01.cdf'
a(2)='/ncf/rumba1/istp/canopus/mari_mag/1996/cn_k0_mari_19960114_v01.cdf'
buf1 = read_myCDF(['CU','CL'], a, /NODATASTRUCT, TSTART=start, TSTOP=stop)
a=strarr(3)
a(0)='/ncf/rumba1/istp/0MASTERS/cn_k1_mari_00000000_v01.cdf'
a(1)='/ncf/rumba1/istp/canopus/mari_rio/1996/cn_k1_mari_19960113_v01.cdf'
a(2)='/ncf/rumba1/istp/canopus/mari_rio/1996/cn_k1_mari_19960114_v01.cdf'
buf2 = read_myCDF(['Rio'], a, /NODATASTRUCT, TSTART=start, TSTOP=stop)
s = plotmaster(buf1,buf2, /AUTO, /CDAWEB, PID=3737, TSTART=start, TSTOP=stop, $
/GIF, OUTDIR='~/public_html/', /SMOOTH, /SLOW, /DEBUG)
end
Example Run/Output
plot 1
plot 2
Images can be plotted nicely through CDAWlib PLOTMASTER.
pro example2d
start='1981/10/29 00:00:00'
stop='1981/10/30 00:00:00'
a=strarr(2)
a(0)='/ncf/rumba1/istp/0MASTERS/de_uv_sai_00000000_v01.cdf'
a(1)='/ncf/rumba1/istp/de/sai/1981/de_uv_sai_19811029_v01.cdf'
buf1 = read_myCDF(['Image_Counts','GeoLat_Dec','GeoLng_RAs'], a, /NODATASTRUCT, TSTART=start, TSTOP=stop, /DEBUG)
s = plotmaster(buf1, /AUTO, /CDAWEB, PID=28241, TSTART=start, TSTOP=stop, $
/GIF, OUTDIR='~/public_html/',/SMOOTH, /SLOW, /DEBUG, /NONOISE)
end
Example Run/Output
image
CDAWeb has "virtual variables" which allow data to be presented in new
representations (by applying a pre-determined function to an existing
cdf variable).
pro example2e
start='1981/10/29 00:00:00'
stop='1981/10/30 00:00:00'
a=strarr(2)
a(0)='/ncf/rumba1/istp/0MASTERS/de_uv_sai_00000000_v01.cdf'
a(1)='/ncf/rumba1/istp/de/sai/1981/de_uv_sai_19811029_v01.cdf'
buf1 = read_myCDF(['Image_CountsM'], a, /NODATASTRUCT, TSTART=start, TSTOP=stop, /DEBUG)
s = plotmaster(buf1, /AUTO, /CDAWEB, PID=28241, TSTART=start, TSTOP=stop, $
/GIF, OUTDIR='~/public_html/',/SMOOTH, /SLOW, /DEBUG, /NONOISE)
end
Example Run/Output
image
INSIGHT
Example Run/Output
Conclusion
Other Links:
CDAWLIB
Master CDFs
CDF Installation
CDAWEB IMAGE DB
CDAWEB DB