The following is an outline of the things that should be routinely checked by TRACE "health monitors". Please note any curiosities in the ops log for others to read about and investigate if necessary.

1. Read the ops log for most recent entries

 

2. Examination of the page snaps – below I list some key items to be looked at. Keep in mind that a brief and frequent look at the key items is better than no look at all. The "minipage explanation" describes each value in detail and as time permits should be added to your health assessment.
• Correlate page time w/ most recent pass time. Sometimes we will miss a snap if there is no real-time data connection from the station or if there is a problem with the EGSE or connection w/ ITOS. Examine the ITOS connection log if needed.
• Spacecraft bus voltage usually runs at 31.5 volts unless in eclipse season where it will fall as low as 26.0 volts
• DPU current will usually run at ~.90 to 1.1 Amps with peaks up to 2.2 Amps
• Examine remainder of voltages and temperatures for appropriate ranges as defined in the "minipage explanation"
• Balanced diodes indicate TRACE is still in Fine Sunpoint Mode. D1 and D2 (East/West) will be roughly equal (2-3 DN) as will D3 and D4 (North/South). D1 and D2 usually have a stronger signal and run a few DN higher than D3 and D4. Sometimes you will see a snap that is taken while TRACE is moving to a new target. You can verify this by looking at the wedge positions on subsequent page snaps. Also, a QS move will cause the diodes to become unbalanced so keep this in mind if TRACE is taking rapid exposures between quadrants.
• Sequence status should be running (0x8000) and sequence name should match the timeline for the page snap time period
• DHC status will usually be 0x48 (running) and error status will be 0x3800 (que program enabled)
• Keep a mental note of the recent DHC warning and error counts. At the time of this writing, we have seen the warning count slowly incrementing (26), indicating possible memory hits to DHC tables. We plan to reload soon. The error count will increment for a variety of reasons most of which are one time problems. If we see a continuous stream of errors, notify Dnyanesh Mathur or Mike Levay, as this would indicate a more serious problem.

 

3. Guider Average Error Plots
• These values represent the error signal being sent from the guide telescope to the Image Stabilization System (ISS). Normally they do not vary by more than a couple DN. There are some calibration sequences (GT and PZT calibrations as well as flat field sequences) that will show a greater deviation. Anomalous spikes should be noted for others to look into.
• Note on plots: On many of the plots, particularly the ISS plots, you will be able to get a feel for the recovery VC1 (stored engineering data). When VC1 is missing, a stratght line will connect the previous and latest data points rather than displayind the normal jittery ISS pattern. Often data will come in later or with a future playback.

 

4. Command Count, Frame Number and Sequence Register 1E
• A steadily increasing command count and frame number indicate the timeline execution is progressing nominally. Should these values flatline then most likely observing has terminated for some reason. Look for a sequence status of 0x0000 and examine the timeline plan.
• Sequence register 1E is used by the control computer to keep track of remaining memory in the spacecraft bulk storage, chron partition. This partition can hold 181 Mbytes of image data however most sequences are written to pause after the 170 Mbytes threshold has been crossed. A nominal plot will show a saw-tooth pattern of steadily increasing memory storage interrupted by a drop at each station pass. Should this plot be flatlined or zeroed for a long duration, it is likely that the sequence has stopped.

 

5. Main Current Plots
• An active sequence of multiple wavelengths will show DPU current spikes up to 2.2 Amps
• Currently, up to three telescope operational heaters are active drawing from 0.6 to 1.8 Amps
• The motor current may spike up to 1.5 Amps depending on when the telemetry is sampled however you will likely see routine spikes of 0.6 Amps, characteristic of the quadrant selector

 

6. Other current plots – the only item of interest here is a plot of the s/c bus voltage, varying on an orbital basis between 31.3 and 31.6 volts

 

7. Spacecraft Battery Plots – These plots do not need to be routinely checked unless TRACE is in eclipse season. The following notes are characteristics of eclipse season:
• Battery state of charge (SOC) will dip to near 80%
• S/C bus voltage will vary from 31.5 volts (fully charged) to 26.0 volts (at deepest eclipse)
• Battery current will discharge and charge at values up to +/-6 Amps
• Shunt current (excess power) will vary from 0 (battery discharging) to 5 Amps (fully charged battery)
• Battery temperatures tend to cool when charging. Last season we saw average temperature values of ~4-7 degrees, increasing towards the end of eclipse season. As long as the battery does not go below 0 degrees, it will be fine. A spacecraft operational heater will kick in at 0 degrees.

 

8. Temperatures – below are nominal ranges for instrument temperatures

 

6 Basic Temp Plots	Deg C	Orbital Delta	Comments
Primary Mirror	10-15	0.5	high thermal mass of telescope and s/c structure provides stability
MgF2 Frame	15-30	10	located in front of telescope w/ much solar exposure
Camera Electronics	10-15	1	rear of telescope; stable due to surrounding components
DPU Electronics	20-35	0.5	Depending on sequence cadence, this parameter can run warm at times
DPU +5 Converter	25-40	1	Depending on sequence cadence, this parameter can run warm at times
CCD Header	-50 to -60	5	warms during eclipse season due to increased Earth’s albedo; ignore the top sensor at –40 – this sensor is only valid for room temperature values
9 SC Only Plots	10 - 30	2-5	these thermistors were added by the spacecraft team and provide temperature values for when the instrument is off

9 SC Only Plots

10 - 30

2-5

these thermistors were added by the spacecraft team and provide temperature values for when the instrument is off

 

9. Voltage Plots – these values should not deviate by more than a half-volt from prescribed values

 

10. Anomaly Tracking – these programs run once or twice a day to scour stored engineering telemetry looking for known anomalous conditions
• IPERR_SEA – searches for DHC errors and identifies offending error code. In the example below, the error code is 8075 - No end of transfer on dma bus b, having to do with the amount of time being too short to completely read out an image transfer. This is a common error seen time to time but should not be seen as a stream of errors.

Date / Time Count Err OpCod Sys DMA Addr SeqID

1-JUN-99 09:08:29 152 8075 162 34c3 107 6ac 115d

 

• Focus Check – we have seen that the focus mechanism has slipped 2 to 3 places relative to the analog sensor value since launch. Center is defined at the position where center indicator reads 1 just prior to reading 0 on the next step. In the example below, the center is defined as position 98. At launch, this value was set to 100 and since then has been recorded as low as 97.

Date	Time	Position Counter	Analog Snesor	Center Indicator
28-MAY-99	06:23:39	98	216	1
28-MAY-99	06:23:44	97	72	0
28-MAY-99	06:24:54	97	65	0
28-MAY-99	06:24:59	98	201	1

 

• Quadrant Selector Reset Check – Spacecraft safing algorithms will reset the quadrant selector should it detect that the motor current monitor exceeds 0.09 Amps for 50 seconds. When this happens an event message is generated and telemetered during a station pass. This routine scours telemetry for the time of reset, offending sequence and quadrant positions. A plot is generated and should be examined to verify that the mechanism reset properly. Valid QS positions are described in the minipage explanation. The example below lists all resets over a five day span:

Quadrant Selector Reset Check Ver 2.0 Program Run: 26-MAY-99 22:11:07 UT
Search Times Are: 21-MAY-99 22:10:30 to 26-MAY-99 22:10:32
Reset Times

22-MAY-99 05:56:59 HPW.flarewatch 195(10) UV(20) 195(10) UV(20) 195(9)

22-MAY-99 07:44:29 CCK.four_tops 171(28) 195(10) 171(28) 195(9) 284(2)

22-MAY-99 08:17:29 BNH.fdm.loop 171(28) ????(24) 195(11)

22-MAY-99 20:57:19 HPW.flarewatch 195(10) UV(20) 195(10) UV(20)

22-MAY-99 23:13:54 HPW.flarewatch 195(10) UV(20) 195(10) UV(20) 195(10)

23-MAY-99 03:29:49 HPW.flarewatch 195(10) UV(20) 195(10) UV(20)

26-MAY-99 04:03:14 STD.CDSsynoptic 284(2) UV(19) 171(28)

 

11. TRACE Sequence Summary by Sequence Run – this program will output gif files for web viewing for a sampling of images per sequence run. Often this routine lags behind or does not have image data for processing as data is sometimes delayed at the station. Browser is a better option for assessing image quality but hard to do remotely.

 

12. Per Image listing for the last 24 hours – this is a helpful routine which lists statistics on image data received
• It is a good check to see that in fact image data has been received from the station and has been processed properly by the DPS and EGSE.
• A quick scan should be made of the maximum pixel intensity (IMAX) keeping in mind that the Safeframe execution begins incrementing at 3500 DN. An evaluation of the AEC performance can be made by watching to see that for higher average intensities (Iavg) the exposure tie is lowered. Most 171/195 exposures do not exceed 20 seconds while some 284 images can be exposed for up to 2 minutes (extreme and rare at that).
• A scan of the QS positions should be made to see that in fact they report valid positions. There have been several times in the past where the QS has been in a wrong position for a given wavelength. Currently there is no automated method for detecting this except when the QS gets "stuck" and reset.

 

13. Make a log entry describing any discrepancies you have observed or none if you find all is OK. The point is to let others know that in fact someone has been assessing the health of the instrument.

Note on TRACE General Health Monitoring Page

This page is home to many of the above described health items plotted in 24 Hr, 10 and 60 day timespans. Occassionally, these items should be examined for longer term trends. An increase in noise in the voltage or current readings could be the sign of degradation and possibly alter the way we operate the instrument. Also apparrent are seasonal variations in temperatures due to the changing distance from the sun, eclipse season or changes in the heater setpoints. CCD dosimetry images are stored here as well.