10. Appendix E: Frequently Encountered Problems (FEP)
Listed below are some problems that have been seen with Mosaic. In some cases, we have known solutions; in other cases, the solutions are more magical.
10.1. Mysterious messages appear and pictures cannot be
taken.
Occasionally, the
Arcon controllers get into a weird state. See section 4.4 for recovery
procedures.
10.2. Data Doesn't
Show Up on Reduction Computer.
Occasionally, and we don't yet understand why or when, the Mosaic reads out but the data fails to transfer to the reduction computer. If you act now, you may be able to save the picture. Restart the DCA on tan/emerald. Type "recover" on the acquisition computer (rush/rust) in the IRAF data-taking window. [We have only seen this problem very occasionally when the Arcons are restarted multiple times.] Unfortunately, "recover" generally will not salvage a readout that hangs after only a portion of the image has been readout.
10.3. "Echo" Images in adjacent CCD: "Cross-Talk"
Due to electrical cross-talk between the CCD pairs controlled by a single Arcon controller (each Arcon runs 2 of the 8 CCDs), the data stored for one of the CCDs in each pair has an echo of the other. Specifically, the stored pixel values for the affected CCD are the true values plus a very small fraction of the counts measured in the paired CCD. The effect is most noticeable for saturated and nearly saturated stars (>20,000 ADUs), whose echoes appear at the 10 to 30 ADU level in the second CCD. For Mosaic-1 we see echoes in CCDs #1, 3, 6, 8 arising from the CCDs #2, 4, 5, and 7 respectively. CCDs 2, 4, 5, and 7 show no sign of a matching echo from their paired CCDs. The Mosaic-2 has similar cross-talk between amplifiers, but in a more complex matching due to the 16-amp readout. For both instruments these echoes can be effectively removed with the CCDPROC "xtalkcor" option turned on. While the magnitude of the cross-talk is generally stable, it does change on relatively long time-scales. We have a library of past xtalk coefficients used for the correction process available in IRAF (in mscdb$) and at http://www.noao.edu/noao/mosaic/mosaic/calibs.html. It is possible to calculate your own set of new coefficients from your own data using the xtcoeff task in the mscred package of IRAF. One can also adjust the provided files slightly and reprocess, checking the residuals until they are smaller than the noise in the image. A more complete discussion of how to determine your own coefficients will be included in a future update or you may contact Buell Jannuzi at jannuzi@noao.edu for more information.
Figure 10.3.1: A sample "echo" appearing in CCD #3 from a saturated star in CCD #4.
10.4. Bad
Rows in All CCDs
We are occasionally seeing images in which one or more bad rows appear in all the CCDs. An example is shown below in Figure 10.4.1. The cause is unknown but common to both Mosaic systems (KPNO and CTIO). Not much can be done with the images to fix them, but a subsequent median, say from a dithered sequence, ought to reject outlying pixels.
Figure 10.4.1: A region of a mosaic image from CCDS 3 and 4 (bottom) and 7 and 8 (top) showing a series of bad rows. CCD 3 has a bad column down the middle. The bad rows are about a quarter of the way from the bottom of CCDs 3 and 4, and a quarter of the way from the top of CCDs 7 and 8.
10.5. Calibrating the CCD Voltages at the Start
of a Block
CAUTION: The following procedure should only be performed by NOAO technical staff. A typing error can easily leave the Mosaic system in an unusable state!
At the start of a run, or if the ambient temperature has changed more than about 10 degrees C, it is necessary to calibrate the Arcon boards. If you don't do this, then subsequent Arcon restarts will be take an extra 5 minutes or so.
From the ARCON Console window, type:
arsh> all video calibrate input
This may take up to 6 minutes. Then type the following sequence:
arsh> b1 video report
arsh> b2 video report
arsh> b3 video report
arsh> b4 video report
[The "b" numbers refer to the Arcon box numbers.] After each command, review the line that starts with "coarseinput(0)". There are 4 numbers, for which 2 will be exactly 0 when running in 8 channel mode. The other 2 numbers should be around 3000.
Now, edit the file on RUSH called /data1/4meter/.arshrc or /data1/36inch/.arshrc, which, near the bottom has a set of lines like:
b1 video calibrate input 3092307732053218
b2 video calibrate input 3085306230513208
b3 video calibrate input 3027311230343047
b4 video calibrate input 3264325430463031
Change the non-zero numbers for each of the 4 lines and restart the Arcons. The next time the Arcons are started, the automatic CCD voltage calibrations will begin with these approximate values, and should take less time to process.
10.6. The ADC does
not seem to be reacting fast enough!
The ADC GUI does not respond instantaneously to the motion of the ADC prisms. There is a time delay. If you restart ARCONS the GUI will occasionally not even show the normal positions of the prisms, but as soon as you turn on either of the non-null modes and/or move the telescope, the display will update. Similarly, you might start an exposure and hear the shutter open before you see the ADC GUI update the positions of the prisms. Be assured the prisms are moved before the shutter opens, it is just the updating of the GUI that is lower priority for the CPU of rush/rust.
10.7. Warning message "Warning: 'ared' package not defined" when starting
ARCONS.
When I start-up ARCONS, I see a warning message "Warning: 'ared' package not defined", do I need to worry about this? No, you can ignore this warning.