Monitoring TRACE Data Input and Processing at the EOF and in Palo Alto by Richard Nightingale, 6-Oct-2000 edited by Thomas Gaeng & RN, 30-Jan-2001 The tasks for monitoring the TRACE data collection follow in this paragraph. The rest of this note explains the tasks in more detail and how to do them. 1) The TRACE "raw_vc2" and "raw_fits" data sets on both cherokee at the EOF and vestige in Palo Alto need to be monitored daily, Monday through Friday, for the previous day(s)'s data ingestion, processing, and mirroring between these two sites. 2) Missing data need to be identified and then requested from the person overseeing the data coming into our system from the NASA FOT-DPS computers. At the present time the DPS point of contact is Louis Habersham, . The request should also be sent to the data team, where it will also be archived, at . Responses received back from Louis on missing or lost data should also be sent to "trace_data", if not copied already in the reponse. 3) Missing or partial passes of "raw_vc2" data need to be monitored until they come in to the EOF or are determined to be lost. The "raw_fits" hourly files need to be checked for missing files at the EOF and in Palo Alto to make sure the reformatting program is working properly at both locations. This will also indicate if the mirroring is working between the sites. 4) Information on the arrival of late data of over 3 days or on reformatting and/or mirroring problems, and questions on the above, should be sent to Richard Nightingale, . Mirroring problems should also be copied to Sam Freeland and Neal Hurlburt . The TRACE data flow from the NASA FOT-DPS system into the ~dps directory on disk6 of cherokee at the GSFC TRACE EOF. Here the inventory program "do_tr_inventory.pro" finds the new data from each TRACE downlink pass, moves it to the "raw_vc2" subdirectory based on the date and time of the data, renames the file, and produces an inventory of the images found. The reformatter program "do_tr_reformat.pro" then reformats the data into hourly TRACE quicklook files of compressed fits image data with extended headers. Bad-dated file names can get created by "do_tr_inventory.pro" when the first image of the "raw_vc2" data file has a bad date, which sometimes occurs during the transmission of the file from the spacecraft through the ground station to GSFC. Twice a day at ~2:00 and ~14:00 the new "raw_vc2" data are mirrored to the "raw_vc2" subdirectories on vestige in Palo Alto and then the new data are reformatted into hourly TRACE fits files, covering the previous day or so at the ~14:00 job and the last 3 days at the ~2:00 job. Vestige is the main TRACE data fileserver, from which data are mirrored to the other TRACE Data Centers or obtained via the internet or tape. It is probably most expedient to check the reformatted data first. At the EOF the data files are located at "/hosts/cherokee/disk5/raw_fits/tri/week*" and in Palo Alto they can be found at "/hosts/vestige/archive/sci/trace/tri/week*". All of the hourly data files for the previous ~24 - 48 hours, or so, should be checked for missing hours of data. Missing hours from previous days need to have been noted and then checked to see if they have arrived yet. The quicklook data at the EOF should normally be current to within ~3 to 6 hours. The DPS people, who work one shift weekdays only, have been doing a good job of keeping us supplied with data recently, but sometimes the data are delayed more than a few hours, especially early in the morning and on Mondays. The reformatted, archived data on vestige should be the same as that on cherokee, up to the previous mirroring time. New data on vestige that are more than 3 days old will not normally be reformatted until 3 week old data are reprocessed. Data can be missing for one of the following reasons. 1) The TRACE instrument may not be on or operating, e.g., the spacecraft and/or instrument might be in safehold. 2) The instrument may be on, but not taking data, e.g., the memory may be full and the instrument is waiting for the next downlink. In both of these cases the image frame counter, found in the housekeeping data and on the health form web page, will not be advancing. This type of data gap is usually found within a downlink pass of data and will show up inside an inventory listing, unless one or more passes of data are missing. 3) The downlinked data might have been lost in transmission from the spacecraft to the ground station or from the ground station to the DPS. Sometimes the last portion of a pass is lost in transmission at the ground station. Missing Data for the above reasons will be missing from both the "raw_vc2" and reformatted data files. 4) There may be missing reformatted data even with the corresponding "raw_vc2" data present if the data have not yet been reformatted or there has been a reformatter error causing the processing to stop. Instrument off and safehold times can usually be found in the trace planner email. The instrument frame counter and memory utilization are plotted on the health pages. These, as well as other housekeeping, can also be checked by looking at daily plots of the housekeeping item, such as the frame counter, 'ikdpfrmn'. This can be done in IDL within the "tsw" environment by the following command line, using the dates of interest, one day at a time: quick_hkplot,'1-jan-00 0:00','2-jan-00 0:00',mnem='ikdpfrmn',psym=3 Normally the thick, plotted line is moving upward with time over the day, but will be horizontal when the frame counter is not advancing. If it is flat over a whole hour or more, i.e., from 02:00 to 04:00 or 04:30, then those hours are missing. When a whole pass of data is missing, the housekeeping data, which are contained in the "raw_vc0" and "raw_vc1" data, may also be missing. In these cases there is a gap in the plotted line. Missing housekeeping data may arrive at a later time. Likewise the instrument memory utilization can be plotted using the mnemonic 'iksqrg1e', where flat tops indicate that the memory is full and the frame counter should be constant. An example of the command line would be quick_hkplot,'1-jan-00 0:00','2-jan-00 0:00',mnem='iksqrg1e' (visualization of used memory is better by leaving off the ",psym=3" option on the plot). One can obtain the start and end times more exactly for a missing data period by looking at the inventory list and within the inventory files. These are found in the weekly "raw_vc2" subdirectories under the "inv" subdirectory for the week of interest. Each inventory file name contains the start time of the first image in the file. After the header of the inventory file each line contains information on one image, including the start time for that image. The inventory file can be searched for suspected data gaps. An example of a recent missing data gap in an inventory file after 02:47:14, as found on vestige, is shown below: % more /archive/newops/trace/raw_vc2/week20000903/inv/20000903_0122_G060_V01.inv ############################################################################### #Input File: /disk6/people/dps/TRACE_G060_QL_2000-09-03T01-22-50Z_V01.DAT1 #TR_INVENTORY_TELEM.PRO V2.7 Run at: 4-Sep-2000 19:33:37.00 # #Byte-Offset Data Source Image-Time Sequence Data-Product Image # into file Type Area Seconds from Number Length in Date/Time # Index 24-May-68 Packets # 48 999 0 1018660970 11782 0 3-SEP-00 01:22:50 91460 1 1 1018660973 11872 73 3-SEP-00 01:22:53 165420 1 1 1018661000 11945 11 3-SEP-00 01:23:20 176144 1 2 1018661000 11956 84 3-SEP-00 01:23:20 261908 1 0 1018661015 12040 23 3-SEP-00 01:23:35 284704 1 1 1018661015 12063 137 3-SEP-00 01:23:35 424044 1 1 1018661018 12200 73 3-SEP-00 01:23:38 497932 1 1 1018661045 12273 13 3-SEP-00 01:24:05 510628 1 2 1018661045 12286 113 3-SEP-00 01:24:05 626180 1 0 1018661060 12399 23 3-SEP-00 01:24:20 648940 1 1 1018661060 12422 137 3-SEP-00 01:24:20 788476 1 1 1018661063 12559 72 3-SEP-00 01:24:23 862124 1 1 1018661090 12631 15 3-SEP-00 01:24:50 877084 1 2 1018661090 12646 187 3-SEP-00 01:24:50 1068396 1 0 1018661105 12833 23 3-SEP-00 01:25:05 . . . 39706428 1 1 1018665946 1678 72 3-SEP-00 02:45:46 39780012 1 1 1018665986 1750 16 3-SEP-00 02:46:26 39795648 1 2 1018665986 1766 182 3-SEP-00 02:46:26 39981184 1 0 1018665991 1948 23 3-SEP-00 02:46:31 40004048 1 1 1018665991 1971 137 3-SEP-00 02:46:31 40143944 1 1 1018665995 2108 73 3-SEP-00 02:46:35 40217696 1 1 1018666034 2181 17 3-SEP-00 02:47:14 ==> 40234252 1 2 1018666034 2198 235 3-SEP-00 02:47:14 40474300 1 1 1018670816 2433 3 3-SEP-00 04:06:56 40477076 1 2 1018670816 2436 3 3-SEP-00 04:06:56 40479860 1 1 1018670821 2439 4 3-SEP-00 04:07:01 40483740 1 0 1018670863 2443 140 3-SEP-00 04:07:43 40626264 12 1 1018670863 2583 26 3-SEP-00 04:07:43 40652216 1 1 1018670871 2609 5 3-SEP-00 04:07:51 . . . 83640240 1 1 1018678055 12098 12 3-SEP-00 06:07:35 83652512 1 2 1018678055 12110 101 3-SEP-00 06:07:35 83755684 1 0 1018678060 12211 23 3-SEP-00 06:07:40 83778412 1 1 1018678060 12234 136 3-SEP-00 06:07:40 83917472 1 1 1018678064 12370 72 3-SEP-00 06:07:44 83990996 1 1 1018678103 12442 13 3-SEP-00 06:08:23 84003336 1 2 1018678103 12455 101 3-SEP-00 06:08:23 84106360 1 0 1018678109 12556 23 3-SEP-00 06:08:29 84129036 1 1 1018678109 12579 136 3-SEP-00 06:08:29 84267992 1 1 1018678112 12715 73 3-SEP-00 06:08:32 ### error ###: apid has changed within a file from 60 to 0 ### error exit ###: { 0 14 580 -680525759 8192} 84341792 888 0 0 0 0 0 ## File size is 84341814 * * Note about data types in inventory list files: * (all negative data types are diagnostic engineering data!) * * -16 = jitter data * -1 = simple data dump in new format * -14 = data dump in old format * -12 = IP register data * -10 = special DTB data dump * * 0 = uncompressed normal data * 1 = jpeg compressed normal data * 8 = 8-bit packed normal data * 12 = 12-bit packed normal data * 999 = incomplete file * 888 = signifies the end of file * The error message at the end of the inventory file is normal because the program encounters the non-data information added to the end of the file at the ground station. The next data and inventory files start at 06:09. Recent "raw_vc2" data and inventory files can be read at the EOF when logged onto one of the EOF computers and can be read at Palo Alto when logged onto vestige. When missing data periods are found in both of the "raw_fits" and "raw_vc2" databases or in the "raw_fits" data with the frame counter seen to be advancing during the period, indicating that data images were taken, then a request should be made for the missing data from the DPS person. The request should specify the date and time of the missing data and include a "cc" to . Please forward responses to the request to "trace_data", if not copied on the response. An updated weekly status of the missing TRACE data hours can be found on the TRACE web pages. A student usually runs a student written program on Mondays to find missing hours of TRACE data for the previous week, and earlier, in the reformatted database on vestige. A missing data list and a reprocessed data list are produced. Up to the present time Richard has then determined the status of the missing data, as above, and informed the student of the status of each missing hour, i.e., no data taken, data lost in transmission, "raw_vc2" data available but not processed yet, or missing data requested (from the DPS person). The student then updates his web page for the missing data and Richard puts it online along with a student generated TRACE data quality plot. The data quality is indicated by the daily averaged percentage of the total data packets received at the EOF for each image, i.e., packets not lost in transmission and not discarded by the DPS because of CRC errors. Both web pages can be found under "Operations" and then "TRACE Health Monitoring" and "Data Quality", or they can be located directly at: http://vestige.lmsal.com/TRACE/Operations/ http://vestige.lmsal.com/TRACE/Data/Missing/ http://vestige.lmsal.com/TRACE/Data/Missing/trace_good_images.gif Under the revised monitoring plan Thomas Gaeng will perform the above ~daily monitoring of both databases and keep Richard informed of his findings of missing data via the "trace_data" email. Questions should also be directed to Richard. Richard will continue to work with the student on keeping the web pages updated using the missing data information supplied by Thomas. ------------------------------------------------------------------------------- example session on cherokee: -> log in as trace or source TRACE setup via: source /tsw/setup/setup_trace -> check raw_fits data: ls -CFlt /hosts/cherokee/disk5/raw_fits/tri/ ls -CFg /hosts/cherokee/disk5/raw_fits/tri/week20010114 | more ==> check for data gaps (i.e., jumps in sequential file numbers) -> if gaps are found check inventory files - e.g.: more /hosts/cherokee/disk6/dps/raw_vc2/week20010114/inv/20010120_1701_G060_V01.inv ==> check file size number before and after '888' data type at end of file. If the numbers differ a large number, then bad data was encountered and the inventory process stopped before finishing the inventory list. (The lost may be recoverable.) -> for data plots start idl: (plots are auto-scaled => make sure that there are no zero data points in the plot) IDL> quick_hkplot,'10-jan-01 0:00','11-jan-01 0:00',mnem='ikdpfrmn',psym=3 IDL> quick_hkplot,'10-jan-01 0:00','11-jan-01 0:00',mnem='ikdpfrmn' ==> same plots because there was no data gap IDL> quick_hkplot,'13-dec-00 0:00','14-dec-00 16:00',mnem='ikdpfrmn',psym=3 IDL> quick_hkplot,'13-dec-00 0:00','14-dec-00 16:00',mnem='ikdpfrmn' ==> ",psym=3" plot shows line gap while other plot shows thin line that connects the data ==> ",psym=3" option is good for frame counter plots IDL> quick_hkplot,'1-dec-00 0:00','2-dec-00 00:00',mnem='iksqrg1e' ==> for better memory plots leave out ",psym=3" option -------------------------------------------------------------------------------