TRACE Safehold Event, 10/10/99

The purpose here is to document TRACE Instrument power-up activities, oversights, and problems associated with the Safehold Event on 10/10/99.

On October 10, 1999 at 21:15:19, the TRACE spacecraft incurred its first safehold event of the mission. The cause of safehold initiation was due to a CPU watchdog timeout (S/C Event Code 101) in the Spacecraft Computer System (SCS). The likely cause of this event was a Single Event Upset (SEU), or radiation hit, to the CPU. Supporting evidence to the hardware upset theory include the fact that TRACE entered the South Atlantic Anomaly (SAA) at ~21:05 UT. Software problems were ruled out based on the lack of evidence found in the Mode Transition Log Table in the SCS. When the software detects a problem, it first writes supporting evidence to the Mode Transition Log Table for later examination and then suspends the clock pulse to the CPU watchdog.

Normally this watchdog timer is reset by the clock signal sent from the CPU. If the watchdog does not see this signal, it times out after 5 seconds and power cycles the +5 volt power supply to the Spacecraft Computer System (SCS). When power cycled, the SCS initializes to boot mode. Just as the processor watchdog does not see a timing pulse, the Attitude Control Electronics (ACE) box no longer receives its timing pulse and watchdogs of its own send the spacecraft to the 'safehold' attitude (~15 degree coning action of the +Y axis about the sunline). In addition, the ACE box sends a hardware command to the Spacecraft Electronics (SPE) box, which in turn will open the non-essential bus relay, thus powering off the telescope.

The safehold situation was detected by the FOT during a blind acquisition at 1057 UT on 10/11 after notification from ground station personnel at Poker Flat of several negative acquisitions. The Instrument team was notified shortly thereafter and anomaly resolution and reconfiguration began. Recovery activities concluded during the 2301 UT pass on 10/12.

22:45 Wallops

Missed due to station misconfiguration

23:21 McMurdo

Ephemeris loaded and ACS commanded to Inertial Sunpoint Mode

TI_DPU_on - DPU turned on but patches were not loaded ⁽⁷⁾

Current was 0.83 Amps until DHC reset and mechanism power off commands were sent; current

then dropped to ~0.63 Amps

Safing actions were not enabled until patches were loaded next pass

00:56 McMurdo

TI_cc_patches ^(7,8) - CC version went to 1:34 as expected

TI_th_init ⁽²⁾ - modified for deadband settings of 100/101 (16.01-16.33 C) and duty cycles of 1 for zones

1,2,3,6; zones 4,5 remained at 88/94 (11.92-14 C); heaters were not turned on however until next

day

TI_me_on - QS did not go to 20/UV as expected but instead 10/195, temperature was 13 C so not an

issue. This incorrect move is not so unusual considering the QS history.

TI_lt_on - nominal; proc leaves the ISS loop open

11:13 Poker Flat

TI_dh_setup - the patches were skipped out of this proc due to a misunderstanding and considering the fact that they were not loaded until much later during L&EO. In the future it's OK to load at this point.

TI_dhc_ldtabs1 - proc halted before bad column tables due to a short pass ~8 minutes

14:34 Poker Flat

TI_dhc_ldtabs1 - completed

TI_dhc_ldtabs2 ⁽⁸⁾- completed

TI_th_heaters_on - completed; initial current draw was ~2.7 Amps and did not cycle until they were in the deadband

16:12 Poker Flat - late AOS by 5 minutes

Commanded S/C to Fine Sunpoint Mode

TI_dhc_patches - verified CC version 1:2D; bug in this proc waiting for DHC version in wrong place - fixed

20:24 McMurdo

TI_dhc_patches - only sent "enable que" and "return version" number commands

TI_cm_on ⁽⁶⁾ - executed OK but filter wheels and quad selector ended in wrong spots ⁽¹⁰⁾

TI_focus_101299 ⁽³⁾ - verified focus by stepping through center from position 112

TI_ccd_protdis/protenb - enabled CCD Safety algorithm in DHC

Uplinked target, PZT offset, AEC tables; wavelength table contained a bug but was fixed ⁽⁹⁾

Closed ISS loop - /ICISCLSL

Uplinked safing RTS # 54 ⁽⁴⁾

Enabled Action Point # 44 and RTS # 53 ⁽⁵⁾

Disabled Gyro # 1; nominal configuration is to run on #'s 2 & 3

22:01 McMurdo

Loaded Sequence and Frame base load files, T99285sld.01 and T99285fld.01

Started STD.focus_wluv sequence - /ICSQSTR NSEQ=1, PRIORITY=0, SQ1H=0000, SQ1L=0X0xed1

23:01 Wallops

Loaded timeline for day 286

1. The SCS Bulk Memory Storage Task does not run in boot mode. Do not expect to receive engineering data during this time. For this event, no data was recorded from 21:15:19 UT on 10/10 until 14:22 UT on 10/11.
2. The procedure TI_th_init had to be modified to include most recent thermal deadbands and duty cycles.
3. New procedure TI_focus_101299 was written to instruct the Control Computer of the location of the focus mechanism. The mechanism was then stepped to verify the focus position.
4. The FOT must load the following RTS to the SCS before normal operations resume:

• # 51 Eclipse Exit
• # 52 Eclipse Entry
• # 53 Quadrant Selector Reset

• # 54 Stop Instrument Sequence when in ISP

1. The FOT must also activate the following Action Point and RTS:

• # 44 Stop Instrument Sequence when in ISP
• # 53 Quadrant Selector Reset

1. The DHC must be setup prior to turning the camera on. When the camera turns on, it sends some 'funny' data to the DHC, usually causing a DHC error. However if the DHC is not there to respond, it can get into a strange state itself. In addition, the power surge from the camera can hang the transceiver on the DHC side when not powered. See MxL for more info in this area.
2. The procedure TI_cc_patches is written to inspect the version number of the CC and load appropriate patches, skipping some earlier ones. This caused confusion as the base software version is considered 1:26 however two EEPROM patches were made before launch bringing the version number to 1:2A at powerup.
3. The following loads/patches have been accounted for in the power up procedures:

• Bad Pixel Tables - T99106bp1.01, T99106bp2.01; uplinked on 4/28/99 and inserted into TI_dh_ldtabs_2
• ISS Patches added to TI_cc_patches

1:2F - Changes default PZT positions after open/close loop commands to X=61, Z=58

1:32 - fix a problem in the CC s/w controlling the ISS, which should end the difficulty we've had

• Other patches in TI_cc_patches
• RCP012c - Patch to correct ccd location calculation
• RCP012d - Patch to correct ccd offset calculation
• RCP012e - Patch to put IMax, IMin Values into IMXV and IMNV
• Patch 01:31 - Patch to the quadrant selector table to make 180 deg moves go past the 0 position
• Patch 01:34 - Patch to correct ADC Offsets for summing modes

9. The following CC loads are considered "latest and greatest:"

• Wavelength Table - T99285wld.02

• Target Table - T99285tld.01

• AEC Table - T99285ald.01

• PZT Offset Table - T99285pld.01

• It is expected that frame and sequence loads will change in the future so they will not be documented here

10. Camera turn on, TI_cm_on, includes the taking of two pictures to setup the amplifiers. The wavelength table should first be loaded so that the mechanisms will go to the proper positions.
11. Temperatures drop significantly during safehold since the instrument operational heaters are off. Both during launch and during this event, only the camera electronics dipped low enough to cause a survival heater to kick on. The camera electronics reached -11 C before the survival heater kicked on. The primary mirror dipped to -3 degrees as well. Most other temps were in the 5-25 degree range.