Requirements for Telemetry Distribution Format
- The values of all instrument products (including science
and engineering data) delivered to the IMAGE spacecraft
CIDP will be stored as generated by the
instruments.
- The organization of instrument data products will adhere
to a space physics recognized structure that shall include
a formalism for holding and defining telemetry data
products.
- Stand-alone use of telemetry data files shall be readily
supported on common desktop and departmental computer
systems.
- The detailed structure and definition of instrument data
product elements, as stored in the telemetry distribution
format, must be permanently defined early in the
development of the SMOC.
- Access to and use of the telemetry distribution format
shall be based on Computer-Off-The-Shelf (COTS) products
to that degree which is possible and practical. Use of
custom or uncommon software shall be
minimized.
- The cost of implementation must be minimized and shall not
exceed the funding available through the IMAGE Mission for
this purpose.
Level-0 Data: Each instrument delivers binary
information to the spacecraft data system. This stream of data
is the Level-0 data for each instrument. The flight and ground
telemetry systems are responsible for delivering these
instrument data streams to investigators through the Science
Mission Operations Center (SMOC).
Level-0.5 Data: It is proposed that all the Level-0
instrument data be placed into a format that is widely
recognized by the space plasma physics community. The suggestion
is based on the desire to archive IMAGE telemetry data in a form
that will be recognized by researchers in this field for many
years to come. The Level-0 instrument data streams do not
constitute such a recognized format. This data format allows no
change to the binary values of the Level-0 data, only a
reorganization of Level-0 data, as desired, to achieve a more
logical order.
Level-1 Data: The IMAGE Mission is committed to deliver
summary observations to the NSSDC in Common Data Format (CDF).
Summary instrument data in the CDF structure is Level-1 data. By
definition, Level-1 data is a summary of Level-0 data,
containing only that subset of information that the instrument
leads deem necessary for use in surveying their instrument
observations and identifying interesting periods of time for
more detailed analysis.
Level-1 Displays: More than one graphic display of
Level-1 data will be necessary. At a minimum, low resolution,
multi-panel suitable graphic images and higher resolution
single-panel graphic images will be needed. These two types of
graphic images will be produced in a web-suitable format, e.g.
GIF or JPEG, in the SMOC and made available for quick viewing
over the web. Graphic displays produced from the Level-1 data
files by the CDAWeb and other NSSDC utilities are also
considered to be Level-1 Displays. In addition to different
graphic image sizes, some instrument summary data may be
suitable for display in more than one organizational format. An
example would be the RPI survey data, which will support
angle-distance and frequency-distance displays.
Three data formats have been proposed for Level-0.5: Level-0,
CDF, and IDFS. The proposal that Level-0 be used for Level-0.5
is a proposal to exclude implementation of a Level-0.5 format.
Each of these formats carry with it advantages and
disadvantages, therefore, any decision will lead to a necessary
set of tasks and compromises.
Please keep in mind that the statements below represent my
current knowledge of the pro and con arguments regarding the
current alternatives. Corrections and additions to this listing
will be made at the upcoming meeting.
Pro
- This format is defined by the individual instrument leads
and will necessarily already exist when IMAGE
flys.
- We can add document files that specify the particular
information and structure held in the Level-0, in order to
document the format sufficiently well to allow future
researchers access to the data.
- RPI Level-0 data doesn't make sense to translate into
Level-0.5, because it will require proprietary processing
before it will make any sense to people outside of the
instrument team.
Con
- The act of adding documenting files to the Level-0 data in
order to facilitate future access to IMAGE data by
non-IMAGE Team researchers defines a new data archive
format that is not in wide usage by the space plasma
research community. We ought not be making up any new
archive format, when formats are already
available.
- This format for the raw data is not self documenting or in
any way structured to support non-IMAGE Team access for
either the short term or long term.
- We will have to develop the necessary IDL or other
language programs for writing Level-1 data, for viewing,
and for analyzing data.
Pro
- This format allows us to organize instrument data streams
into a logical ordering of the bytes of
information.
- The format is in use and has been used in many other space
plasma physics missions, including the Viking, Freja,
Interball, and Prognoz missions on which SwRI was not a
participant.
- SwRI is the origin of this format and their participation
on the IMAGE team insures some level of support for its
use.
- The "generic software" has been developed to read the IDFS
data and perform certain kinds of processing, as well as,
support the federation of distributed data base systems
that use the "generic software" and archive IDFS formatted
data.
- The generic software will run on any platform running a
UNIX or UNIX-like operating system
- The generic software that has been developed for the IDFS
can be used to produce all of the defined Level-1 data
products with the exception of RPI.
Con
- We are already obligated to learn how to write and read
the CDF format. We shouldn't introduce yet another format
in our SMOC and investigator data systems.
- The generic software that has been developed for the IDFS
format cannot be used to write the Level-1 data files for
the RPI data.
- IDL routines have not yet been identified to read and
write IDFS formatted data.
- In general, we will have to develop the necessary IDL or
other language programs for writing Level-1 data, for
viewing, and for analyzing data.
- The generic software is not applicable to all data sets
for use in writing Level-1 data files.
- The generic software has yet to be tested on a platform
running a non-UNIX operating system (Windows, DOS, NT, or
MacOS).
- IDFS specifications look heavily oriented toward storing
conventional imagery data coming from particle
detectors/spectrometers/UV sensors. It is a concern that
fitting RPI data into IDFS might be a time consuming
effort, especially considering IDFS library interfacing
problems with RPI team post-processing software in
Java.
- IDFS is a proprietary binary format which comes with a set
of library functions written in C language to operate with
it. It does not look possible to read IDFS files directly
(i.e., without library functions) because of proprietary
solutions enforcing platform independence of the format.
Therefore the library routines which "know" how to deal
with the data must be an integral part of any application
operating on IDFS files. And if an application is not
written in C or C++, a problem of porting or interfacing
appears. Porting means rewriting the library functions in
different language (say, IDL language), where interfacing
means arranging calls to IDFS library via cross-language
interface. Neither solution is fun to
implement.
Pro
- This format allows us to organize instrument data streams
into a logical ordering of the bytes of
information.
- The format is in use and has been used in many other space
plasma physics missions.
- NASA/GSFC/NSSDC is the origin of this format and their
participation on the IMAGE team insures some level of
support for its use.
- IDL routines already exist for reading and writing CDF
formatted data.
- Using this format means not having to learn and support
another data archive format for the IMAGE mission.
Con
- Although there are many other missions that have used CDF,
it appears that CDF usage may have been mandated by
NASA.
- It wasn't specifically designed to archive raw telemetry
data and has never been used by any mission for that
purpose.
- We will have to develop the necessary IDL or other
language programs for writing Level-1 data, for viewing,
and for analyzing data.
- CDF appears not to be suitable for RPI Level-2 data,
because the format requires array sizes to be fixed in
size at the maximum possible dimension expected for each
data variable. This requirement will waste significant
space, because maximum array sizes would often not be
needed. It is also the case that normal maximum array
sizes could end up being too small in some cases to hold
all of the measurements. RPI Level-2 data is the first
format for this instrument that is useable by anyone. That
means that RPI Level-1 data must be produced from Level-2
data.
- If this were to be implemented where each CDF record was
just another data packet, then this approach doesn't get
you past (Level-0 for Level-0.5 Con #1) having to define
the packets in another document.
The only other way to implement it would be break the
packet down into it's various measurands, and save these
measurands in byte arrays, which still leaves you with the
task of converting/calibrating these into engineering
values.
- [Bodo Reinisch/Ivan Galkin] Suppose we can find a way to
import IDFS files into IDL. Then adopting IDFS means for
us a commitment to IDL development environment. We have
full IDL license in house, and IDL is flexible enough to
support plasmagram/echomap displays and the interactive
mode, so we do not seem to have problems working with the
RPI data. However, our processing techniques have to be
ported to IDL language as well. This type of commitment to
a visualization tool is quite serious and somewhat
contradicts our general policy to work with universally
available environments like Java. IDL is available for
many platforms, however the license is too expensive for
ordinary researchers to afford. (At this time we have only
one IDL developer seat.) Java tools seem to continue
gaining momentum and already show unmatched advanced
features coming from the best existing projects in the
field.
Curators
Dr. E. V. Bell, II, ed.bell@nasa.gov, +1-301-286-1187
NSSDC, Mail Code 690.1, NASA Goddard Space Flight Center, Greenbelt, MD 20771
Dr. D. R. Williams, dave.williams@gsfc.nasa.gov, +1-301-286-1258
NSSDC, Mail Code 690.1, NASA Goddard Space Flight Center, Greenbelt, MD 20771
|
