CU-QMW-UR-0002 UK CDHF: AMPTE UKS High Resolution Data 5 February 1997 1 0

The UK CDHF for STP AMPTE Database: UKS High Resolution Discussion Paper

S.J. Schwartz
Astronomy Unit
Queen Mary & Westfield College
Mile End Road,
London E1 4NS
and
The UK CDHF Team
e-mail csc_support@qmw.ac.uk

Contents

• Introduction
• Issues to be Addressed
• Status of Data and Teams
• Draft UKS High Resolution Database and Activities
  • Science Products
  • Rationale and Comments for Discussion
• Summary
• Notes for Tables
• About this document ...

Introduction

This document follows from a splinter meeting of the UK ISTP meeting held on 6 December 1996 at RAL. The splinter meeting consisted of representatitives of the UK AMPTE community. With the AMPTE Prime Parameter Database well in hand, the meeting concentrated on high resolution products. This discussion paper is designed to summarise that meeting and to propose a draft high resolution database for discussion purposes.

Issues to be Addressed

Some issues which need further consideration include:

Data Products

Which ``high resolution'' data products should be included?

Format & Access

What is the format of the product and how will it be accessed? Possibilities include CDF, IDFS, or others. Access could include online storage at the CDHF, off-line storage at the CDHF, on-the-fly generation at the CDHF, a mixture of CDHF and direct requests to the PI teams, off-line generation at the CDHF, or others.

Timescale

The timescale should be in the near future, i.e., months, in order to be timely and not interfere with other activities. Also, it will be helpful to make some products available as soon as possible to enable effective use of the data and to provide immediate feedback on the development of the CDHF.

Cluster/Phoenix Preparation

The construction of the high resolution facility should be seen as part of the preparation for Cluster/Phoenix, e.g., as a testbed for data formats, archiving, cataloguing/access, etc.

Resources

The resources necessary for the high resolution phase need to be found. CDHF resources will probably come from the Cluster/Phoenix preparation. That at PI and other institutions should also be part of those institutions' resources for Cluster/Phoenix, although the activity should complement rather than hinder the hardware/instrument preparation.

Partial Holdings

There may be a balance between complete product generation and incremental archival deposition as requests are filled and resources permit.

Status of Data and Teams

MAG

The IC team propose a single batch generation of all high resolution data. The CDHF will convert this data into CDFs and archive them. Validation will be done on a case by case request basis. The raw data exists in a stable format and on a stable medium.

ION

The MSSL team propose to convert the raw data into IDFS format, as they plan to do for their Cluster data. The raw data exists in a stable format and on a stable medium. The conversion of raw data stored in IDFS into science data products requires a layer of software. The source and location where this software will be run is open for discussion, but will clearly require a substantive MSSL input.

ELX

The electron data is handled both by QMW and RAL. The raw data is in a stable format and medium, although some ``cleaning'' is required. The final archived format for raw data is TBD but will probably be the existing data files.

CORR

The raw data exists mostly on tape and is volatile.

WAV

The data exists on tape only and is volatile.

IRM

The IRM side is progressing, although validation tests of the science products with MPE is not yet at an advanced state. The CDHF would plan to concentrate in the first instance on products which mirror those in the UKS high resolution database.

Draft UKS High Resolution Database and Activities

Science Products

Table 1 sets out a possible AMPTE UKS scenario for high resolution data and data products. It is only a discussion scenario at present, and does NOT reflect agreement by any party to supply the necessary resource or consent.

Rationale and Comments for Discussion

The various products and activities set out in Table 1 for discussion purposes represent some initial thought on the use of the High Resolution Database. Some explanation of the ideas behind this table is useful here by way of explanation and to provoke further discussion.

1. Proposed On-line data products are designed to:
   1. Provide higher time resolution for, e.g., magnetic field, parameters already held in the Prime Parameters
   2. Provide more detailed phase space information for particle data
   3. Provide relatively robust products (e.g., by supplying kinetic particle data at half-minute resolution)
   4. Require relatively modest storage requirements
   5. Be suitable for relatively rapid browsing despite the detail of information (at a few pixels for each record, energy-time spectra covering an hour or so can be accommodated on a single plot).
   6. Require relatively little further processing by the end user
   7. Be amenable to simple processing by the end user (e.g., simple summing over an array dimension
   8. Have straightforward IRM counterparts where feasible
2. Most products are envisaged to be delivered to the end user in science units as CDF files. This is purely a matter of convenience as the CDHF is set up to deal with ISTP/CSDS CDF files. Support will need to be provided, both within QSAS and perhaps on an ad hoc basis, to ensure that the end users can extract the data easily from these files for use within their home institutions.
3. As a consequence of these considerations, these on-line products fall short of a complete high resolution facility. The highest resolution facility therefore requires off-line capabilities and a more knowledgeable approach at the instrument level by the end user
4. The high resolution magnetometer data is effectively a final science product in science units
5. The proposed electron parameters include:
   1. Energy spectrograms and possibly pitch angle distributions at half-minute resolution for on-line access. The pitch angle distributions will be in equal cosine bins, so summing over all bins will also generate, to within a constant factor, the energy spectra. Nonetheless, is provided separately to facilitate easy and rapid browsing of the electron kinetic properties.
   2. Pitch angle distributions at arbitrary time and pitch angle resolution on request by CDHF-held software off-line processing.
   3. Electron moments calculated at arbitrary time resolution and with various user-specified options for dealing with low-energy portions and other data reduction possibilities for specific purposes.
   4. Software to achieve the above exists at both RAL and QMW, but will need to be integrated into the CDHF infrastructure. Additionally, some CDF design work is required to hold the resulting data products.
6. The proposed ion parameters include:
   1. On-line partial densities for FTR and SW modes integrated over two of the three dimensions (energy, polar angle, azimuthal angle) at half minute resolution. These products are well-suited to rapid browsing. Partial densities are chosen to enable end-user summation over all or part of the range (e.g., to find the density of ions with energies greater than some value). Unlike the electron data (which is assumed to be gyrotropic and can be put into equal pitch angle bins, the ion data is fundamentally three-dimensional in velocity and is sampled in energy/angle bins of varying sizes.
   2. Off-line generation of phase space distribution functions.
   3. Generating these products from MSSL-supplied IDFS files will require a layer of IDFS software to extract the relevant ``sensor'' data and assemble the scientific data product. In order to provide a reliable product, MSSL support in constructing this layer is envisaged.
7. Only one Correlator product is readily identifiable, although its availability and suitability has not yet been assessed. There are no correlator products in the AMPTE Prime Parameter database.
8. the AMPTE Prime Parameters include a wide range of Wave products, and no further wave products are obvious candidates for inclusion in the high resolution database.

Summary

Further progress will require iteration between the CDHF and the PI teams. Particular attention needs to be paid to ensure that the UKS high resolution facility meets the scientific need and can be put into place within resource constraints. The basic structure shown in Table 1 also serves as a starting point for other services and products.

  
Table 1: Strawman AMPTE UKS High Resolution Archive and Activities

  
Table 2: Strawman AMPTE UKS High Resolution Archive Sizes

Notes for Tables

1. The total words includes 2 words to hold the timetag for each data record.
2. Archived 30s particle products are suggested as a compromise in terms of storage and should serve as a useful dataset for browsing kinetic details.
3. Is there a need for ion moments other than those in the PP? In the case of the electrons, there are various options to fill the missing low-energy portions of the distribution which make the electron PP moments representative rather than definitive.
4. The storage in Table 2 for On-line data is 6Mb per day of AMPTE UKS (5 hours), totalling 900Mb over the mission lifetime. These figures exclude CDF overheads, etc.

About this document ...

The UK CDHF for STP AMPTE Database: UKS High Resolution Discussion Paper

This document was generated using the LaTeX2HTML translator Version 96.1 (Feb 5, 1996) Copyright © 1993, 1994, 1995, 1996, Nikos Drakos, Computer Based Learning Unit, University of Leeds.

The command line arguments were:
latex2html -split 0 AMPTE_DB_hi_res_discussion.

The translation was initiated by Steven J Schwartz on Thu Feb 6 11:42:48 GMT 1997