Return to SQIID Web Page
NB: The UPSQIID package is not an officially released and supported IRAF package; all queries should be directed to KMM at NOAO (merrill@noao.edu; 520-318-8319). Although specifically targeted for SQIID, the routines are suitable for other image data as well. See APPENDIX B for more information.
The NOAO SQIID Infrared Camera produces simultaneous images of the same field in the J, H, K, and narrowband L passbands, using individual 512X512 quadrants of ALADDIN InSb arrays. The observations are generally background (photon statistics) limited. Typical observing programs include:
These three kinds of observations are distinguished because they require somewhat different data reduction strategies. This document describes a set of IRAF programs (created by Michael Merrill at NOAO) designed to facilitate the reduction of SQIID datasets. These programs reside in the IRAF package "upsqiid". They have a number of imperfections in the user interface (especially a large number of irrelevant parameters) and currently do not have help files.
We will define a "dataset" to be the set of direct observations of a given field. These may be dithered observations of a single target or a mosaic of a larger region. In the extreme case, the dataset may contain only a single exposure (and only the first two steps listed below would be required). The basic path for the reduction of SQIID dataset can be described as follows:
Users are cautioned that IR image datasets often present a greater data reduction challenge than optical CCD images both due to the superior performance of optical CCD detectors (lower dark current, readout noise, and pixel to pixel sensitivity variations) and especially due to the extreme background limited nature of most IR observations. The results at each step in the process should be carefully examined and problems understood before proceeding. Many problems can be solved by the exclusion of bad images from the datasets.
The SQIID system generates FITS images with the following naming convention:
where "base_name" ("n1_") was supplied by you at the time of observation, "channel_id" is j,h,k or l, and the number "XXX" is sequentially numbered (it was automatically incremented for each exposure until a new "base_name" was declared, typically at each new object).
See APPENDIX A.2 for further information on image naming conventions and data restoration.
To facilitate the handling of the large numbers of image sets typically obtained during a SQIID observing run, the SQIID software is designed to operate on IRAF "@list" files (inside IRAF, one instructs the system to read data from the file "list" by prepending "@" to it; "@list" tells the task to read list to get the image names.) The task 'chlist' is provided to create list files. (See APPENDIX A.3 for further discussion of the "chlist" tasks.)
To use 'chlist' the image names must end in an integer. For example,
will create a list file name "jlist" (note that the output is to the standard output) with 20 image names starting with "raw001, raw005, ..., raw077". The increment between names is the hidden parameter 'delta' which defaults to 1. Directory pathnames can be prepended to the baseid.
To facilitate data processing of image lists, the RECHANNEL, SQSECTIONS, SQPARSE tasks have been provided to assist in producing names of alternative channels from a given channel image name:
RECHANNEL produces a list of images from an specific channels from a list
of another channel.
RECHANNEL:
image = Image list
channel = Image color: |j|h|k|l|
(section = "") Image subsection [xmin:xmax,ymin:ymax]
SQSECTIONS functions like the IRAF standard SECTIONS task, allowing one to
count images resulting from a list or image template, output assorted subsets.
"ch_id" allows one to produce a specific channel output from any channel input.
SQSECTIONS:
images = Image template
(option = "root") Option (nolist, fullname, root, section)
(nimages = ) Number of images in template
(ch_id = "") Returned channel_id (jhkl)
SQPARSE:
image = Image name
(root = "") Returned filename root sans channel
(chroot = "") Returned filename root with channel
(ch_id = "") Returned channel_id (jhkl)
(alt_ch = "") Returned alternative channel(s)
(extn = "") Returned image extension
(section = "") Returned image section
(verbose = no) Print results
Since the individual channels of SQIID illumine a single quadrant of a four quadrant array and the best quadrant of each device is selected, the relative orientation on the sky is channel specific. Until the image saver task itself is modified to properly orient the channels (North up and East to the left) in the saved images, this task needs to be performed after the images are saved to disk.
[NOTE: As of September, 2000, the saver task correctly handles the orientation issue during the saver process. Images saved to disk (and automatically displayed) are properly oriented with North up and East to the left.]
The SQIID software can handle this several different ways. However, it is important to keep track of the orientation issue as you process data. The channels are oriented as follows:
J H K L
W N S E
S N W E E W N S
E S N W
One can process all the data, producing sky frames, darks, flats, and masks, for each channel "as is". One can then use the orient parameter to select within the basic processing tasks (USQPROC, PATPROC, MOVPROC) to determine the final orientation:
The CHORIENT task allows one to orient images one at a time, using the image name template from one channel to optionally orient all channels. The image header keyword "ORIENT" is added to signal that the orientation process has occured. One can either orient in place, without changing the name of the image, or, using the "newid" string to append an identifier between the image root and the channel id.
CHORIENT:
image = Image name
(channels = ".") Orient these channels (eg, "j", "jhk";
"." == named image channel only
(verbose = no) Print results
(newid = "") ID append to existing name?
Starting with image = n1_001j.fits:
By convention, the channel ID always remains at the end.
III. DARK, FLAT, and SKY Frames
Proper data reduction requires accurate solutions for the small additive effects of internal illumination and charge generation (DARK frames), the large additive effects of sky illumination (SKY frames) and the multipicative effects of position dependent pixel sensitivity (FLATFIELD frames).
The creation of DARK, FLAT, and SKY calibration frames are the first step in the data reduction process. The DARK frames are simple to obtain and process but the FLAT and SKY frames are more difficult to create and are crucial to the quality of the final images. Compared to optical band CCD observations, infrared observations are extremely background limited. Furthermore, the background in the near-infrared is variable at many temporal and spatial scales.
System SKY frames are established from observations with the same integration time and near in time and place to the source frames. This is done by interspersing frames of off-source (ideally) blank sky fields in the sequence of source observations. Many of these sky frames, each at different position, are then median filtered to remove celestial sources to generate a SKY frame, which is measure of the temporally stable illumination on the array, including dark current and fixed pattern noise.
System FLATFIELD frames are established using the sky as a uniform external illumination source. The background in the IR results in sky levels with S/N of better than 100 in a single 3 minute exposure. By taking the median of a reasonable number of blank fields (or target fields without large objects) obtained at different times during the night and at different locations on the sky, a satisfactory measure of the system flatfield can be derived.
Generally, DARK frames (taken with the internal cold dark slide in place) are stable over a night (at about 1 electron/sec) and are probably stable over an entire observing run. Changes in dark current can accompany changes in the ALADDIN array temperatue. Since the SQIID dark current has both a base level and a time-dependent component, a dark frame must be created for each exposure time (that you intend to use to determine flatfields or for dark subtraction) during the observing run (it is assumed here that the observer has obtained such dark exposures).
DARK frames are created with the task "SQIID". For example,
The task "usqdark" has a number of parameters which should be set as indicated in the "lpar" listing below. This format will be used henceforth to document the parameter default settings and the parameters of interest to the user (which are indicated with an * in the leftmost column). Parameters which feed directly into the IMCOMBINE task set apart.
USQDARK:
* input = Input raw dark images
* output = Output IMCOMBINED dark image
* (stat_calc = yes) Calculate statistics?
* (statsec = "[300:700,400:600]") Image section for calculating statistics
* (logfile = "STDOUT") Log file name
* (verbose = yes) Verbose output?
# IMCOMBINE parameters =========================================================
(common = "none") Pre-combine common offset: |none|median|mode|
* (reject_opt = "minmax") Type of rejection operation
* (comb_opt = "average") Type of combine operation: |average|median|
(lthreshold = INDEF) Lower threshold for exclusion in statistics
(hthreshold = INDEF) Upper threshold for exclusion in statistics
(blank = 0.) Value of output pixel when all are rejected
(expname = "") Image header exposure time keyword
* (nlow = 1) minmax: Number of low pixels to reject
* (nhigh = 1) minmax: Number of high pixels to reject
(nkeep = 0) Min to keep (pos) or max to reject (neg)
(lsigma = 3.) Lower sigma clipping factor
(hsigma = 3.) Upper sigma clipping factor
(pclip = -0.5) pclip: Percentile clipping parameter
# IMCOMBINE parameters =========================================================
See APPENDIX A.4 for a discussion the input image parameter.
The 'statsec' default should generally be satisfactory. The 'reject_opt'
should probably be "minmaxreject". The 'comb_opt' should probably be "average"
if the number of dark exposures is less than or equal to 5 and "median" if
greater than 5. One does not normally threshold DARKS or bring the data to a
common zero-point offset (thus, 'common = "none"'). If you want to keep a
record of the USQDARK operations set 'logfile' to a filename (else set to "").
Setting "verbose = yes" increases the amount of information reported
(including statistics). Setting "stat_calc = yes", assures that complete
statistics will be reported.
USQDARK calls "images.immatch.imcombine". See APPENDIX A.6 for more
information on IMCOMBINE.
b) FLAT Frames
The pixel to pixel sensitivity (i.e., the flatfield) is also generally
stable over the night and perhaps several nights. Unlike optical CCD
observations, images of the interior of the dome or the twilight sky are
not practical for the flatfielding of infrared images. Single filter
infrared observations of faint sources were typically limited by variations
in the background during chopping but the use of the single detector meant
that "flatfielding" was not required. With the field of view of infrared
array detectors now larger than the sources of interest, the sky background
may be relatively easier to compensate for (see below) but since infrared
sources are often much fainter than the sky background very precise flat
field calibration is required.
The background in the IR results in sky levels with S/N of better than 100
in single exposures (order 30 sec at J, H, K and order 1 sec at L band).
By taking the median of a reasonable number of blank fields (or target
fields without large objects) obtained at different times during the night
and at different locations on the sky, a satisfactory measure of the detector
flatfield can be derived. A list of suitable observations, carefully edited to
exclude fields with sources larger than the dithering range, should be prepared.
FLAT frames are created with the task "usqflat". For example,
Since direct illumination of the array is possible (remember that IR secondary mirrors are usually undersized), observations near bright sources (such as the Moon), which have atypical illumination, should not be used to determine global flatfields. Observations of illuminated targets inside the dome and during twilight will also have illumination atypical of nighttime observations, and should be used with caution.
There are several problems with the twilight sky:
Operationally, we create a FLATFIELD by subtracting a DARK frame (taken with the internal cold dark slide in place) from appropriately selected sky frames, normalize the result by dividing by its the median value within the central region, and median filter the ensemble to produce a FLATFIELD with high S/N. The SQIID system response is stable and is very flat across the arrays, with a odd-even column modulation of +/- a few percent. Consequently, the flatfield for each channel should be stable at the percent level under normal illumination and global flatfields can be constructed which are viable for extended periods of time.
The task "usqflat" has a number of parameters which should be set as indicated in the "lpar" listing below:
USQFLAT:
* input = Input raw sky images
* output = Output normalized flat_field image
* darkimage = Input dark_count image ("null"==noaction)
(ref_flat = "") Reference normalized flat_field images
(statsec = "[300:700,400:600]") Image section for calculating statistics
* (lo_reset = 0.1) Lower limit for exclusion in output
* (hi_reset = 2.) Upper limit for exclusion in output
(logfile = "STDOUT") Log file name
(verbose = yes) Verbose output?
# IMCOMBINE parameters =========================================================
(common = "none") Pre-combine common offset: |none|median|mode|
* (prenorm = "median") Pre-IMCOMBINE frame divisor:|none|median|mode|
* (imscale = "none") IMCOMBINE scale option: |none|median|mode|
(comb_opt = "median") Type of combine operation: |average|median|
(reject_opt = "none") Type of pixel rejection operation
(mclip = no) Use median, not mean, in clip algorithms
(pclip = -0.5) pclip: Percentile clipping parameter
(lthreshold = INDEF) Lower threshold for exclusion in statistics
(hthreshold = INDEF) Upper threshold for exclusion in statistics
(blank = 0.1) Value if there are no pixels
(weight = "none") Image weights
(expname = "") Image header exposure time keyword
(nlow = 1) minmax: Number of low pixels to reject
(nhigh = 1) minmax: Number of high pixels to reject
(nkeep = 0) Min to keep (pos) or max to reject (neg)
(lsigma = 3.) Lower sigma clipping factor
(hsigma = 3.) Upper sigma clipping factor
(rdnoise = "0.") ccdclip: CCD readout noise (electrons)
(gain = "1.") ccdclip: CCD gain (electrons/DN)
(snoise = "0.") ccdclip: Sensitivity noise (fraction)
(sigscale = 0.1) Tolerance for sigma clipping scaling correction
(grow = 0) Radius (pixels) for 1D neighbor rejection
# IMCOMBINE parameters =========================================================
See APPENDIX A.4 for a discussion the input image parameter.
Since 'prenorm' and 'imscale' are complimentary parameters which determine how the image scaling is accomplished, one of them should be set to "none". The 'prenorm' parameter controls scaling operations before submission of the data to the IRAF 'imcombine' task. The IRAF 'imstatistics' task is used to determine the selected statisitic for each image within the image section selected by 'statsec' which is used to scale the image by division. The 'imscale' parameter controls the scaling operations after submission of the data to the IRAF 'imcombine' task. Imcombine samples the data within 'statsec' for the 'imscale' selected statistic, finds the average statistic for all the images and scales each image to the average value prior to combination. Setting 'reject_opt = "minmax"' might be an appropriate choice. The data which survive the pixel rejection and threshold algorithms are then combined according to 'comb_opt'. The 'usqflat' task then divides the result by the average statistic to produce the FLAT.
The 'statsec' default should generally be satisfactory. The 'reject_opt' should probably be "minmaxreject". The 'comb_opt' should be "median". One does not normally threshold FLATS or bring the data to a common zero-point offset (thus, 'common = "none"'). The parameters 'lo_reset' and 'hi_reset' are used to range the FLAT to be between two values. Data below 'lo_reset' are set to lo_reset; data above 'hi_reset' are set to hi_reset.
If you want to keep a record of the USQFLAT operations set 'logfile' to a filename (else set to ""). Setting 'verbose = yes' increases the amount of information reported (including statistics).
USQFLAT calls "images.immatch.imcombine". See APPENDIX A.6 for more information on IMCOMBINE.
The background illumination in the near infrared is far from stable with time or location. In the near IR (especially at J) moonlight (direct or reflected off clouds) is a factor, and the night sky emission (especially at H, K, and L) is a function of temperature and humidity. Although the 10 to 30 percent variations in background caused by these factors do not strongly limit the S/N of observations (except at K and L for large changes in temperature), they greatly complicate both the creation of mosaics of large regions and accurate surface photometry of objects with extents comparable to the SQIID's field of view. For such observing programs, it is best to obtain a sufficient number of exposures (and intermixed sky exposures if necessary) to create a sky frame for that specific observation. For programs with single or a few observations of many targets, a sky calibration based on the average over the night combined with the fitting of a surface to the final image is the best that can usually be accomplished. It is useful to remember that variable airglow can cause the sky background at H can vary by a factor of 2 and J can vary by 40% on hour time scales.
After examinination of the individual images a list of suitable images should be prepared as described above. The selection of suitable images may require that the individual images be dark subtracted and flatfielded prior to examination on the image display (see section IV). However, raw images must be used as input to the "usqsky" task. SKY frames are created with the task "usqsky". For example,
The task "usqsky" has a number of parameters which should be set as indicated in the "lpar" listing below:
USQSKY: * input = Input raw sky images * output = Output sky image (darkimage = "null") Input dark_count image ("null"==noaction) (statsec = "[300:700,400:600]") Image section for calculating statistics * (logfile = "STDOUT") Log file name * (verbose = yes) Verbose output? # IMCOMBINE parameters ========================================================= (comb_opt = "median") Type of combine operation: |average|median| (norm_opt = "zero") Type of pre-combine operation: |zero|scale|none (norm_stat = "median") Pre-combine common statistic: |none|mean|median (reject_opt = "none") Type of pixel rejection operation (mclip = no) Use median, not mean, in clip algorithms (pclip = -0.5) pclip: Percentile clipping parameter * (lthreshold = INDEF) Lower threshold for exclusion in statistics * (hthreshold = INDEF) Upper threshold for exclusion in statistics (blank = 0.1) Value if there are no pixels (weight = "none") Image weights (expname = "") Image header exposure time keyword (nlow = 1) minmax: Number of low pixels to reject (nhigh = 1) minmax: Number of high pixels to reject (nkeep = 0) Min to keep (pos) or max to reject (neg) (lsigma = 3.) Lower sigma clipping factor (hsigma = 3.) Upper sigma clipping factor (rdnoise = "0.") ccdclip: CCD readout noise (electrons) (gain = "1.") ccdclip: CCD gain (electrons/DN) (snoise = "0.") ccdclip: Sensitivity noise (fraction) (sigscale = 0.1) Tolerance for sigma clipping scaling correction (grow = 0) Radius (pixels) for 1D neighbor rejection # IMCOMBINE parameters =========================================================
See APPENDIX A.4 for a discussion the input image parameter.
Setting 'darkimage = "null"' avoids subtraction of the DARK prior to image combination. You can enable dark subtraction by setting 'darkimage' to the appropriate DARK (DARK will be added to the result after image combination). The data are offset to the average of the statistic selected by 'common' prior to image combination. Setting 'reject_opt = minmax' might be an appropriate choice. The data which survive the pixel rejection and threshold algorithms are then combined according to 'comb_opt'.
The parameters "lthreshold" and hthreshold" should be set to include the range of all good pixels in all images. For example, the range 0 to 20000 would certainly be safe. Setting these top closer tolerences may improve the precision of the common median determination.
If you want to keep a record of the USQSKY operations set "logfile" to a filename (else set to ""). Setting "verbose = yes" increases the amount of information reported (including statistics).
USQSKY calls "images.immatch.imcombine". See APPENDIX A.6 for more information on IMCOMBINE.
IV. Processing of Individual Images
Each image (except for dark exposures and blank sky frames) needs to be processed prior to combining the collective datasets into dithered images or mosaics. It is most efficient to create a list of images for each dataset (see "chlist" above). The processing applied is:
This processing is done with the task "upsqproc" (basic process). For example,
The DARK subtraction is implicit in the SKY removal given the sequence of steps listed above. The expression "%raw%p%" assumes that all of the file names in the list "limages" start with the string "raw" (directory information is skipped so that only that part of the pathname after the final "/" survives the substitution process). The output files will have file names starting with the string "p" in place of "raw". The list "limages" then may be edited with a text editor or a new list created with "chlist" for use in the next stages of the reduction.
The task "upsqproc" has a number of parameters which should be set as indicated in the "lpar" listing below:
UPSQPROC:
* input = Input raw images
* output = Output image descriptor: @list||.ext||%in%out%
* flatimage = Input flat field image name
* blankimage = Input blank field image name
scaleimage = Input scale image name
(opt_norm = "none") Normalization: |none|mean|median|mode|
(sec_norm = "[300:700,400:600]") Image section for calculating norm
(result_norm = 0.) Resultant value after normalization
(int_time = 5.) Integration_time
(delay_time = 3.) Delay_time
(rescale = no) Rescale scaleimage to make sky?
(fixpix = no) Run FIXPIX on data?
(badpix = "badpix") badpix file in FIXPIX format
* (setpix = no) Run SETPIX on data?
* (maskimage = "badmask") untransposed bad pixel image mask
* (value = 0.) pixel value for masked pixels
* (orient = no ) Orient image with N up and E left
(lowerlim = INDEF) Lower limit for exclusion in stats
(upperlim = INDEF) Upper limit for exclusion in stats
* (verbose = yes) Verbose output?
* (logfile = "STDOUT") logfile name
See APPENDIX A.4 for a discussion the input image parameter and APPENDIX A.5 for a discussion of the UPSQPROC output image parameter.
The parameter 'blankimage' should be set to the SKY frame except for very short exposures (e.g. standard star observations) where the appropriate DARK frame may be used. The parameter 'scaleimage' is not presently relevant. For SQIID data, the parameter 'orient' could be set to "yes". This will orient the images to the usual North up, East left orientation. Normally, the bad pixel corrections controlled by 'fixpix' and 'setpix' are deferred until the images are actually combined. However, if you wish to mask bad pixels at this stage set 'setpix = yes' and 'maskimage' to an mask image appropriate to the channel selected and the orientation of the raw image. Masked pixels are set to 'value'. (see APPENDIX A.7 for how to make a pixel mask.)
The processing order is:
MOVPROC list processes SQIID raw image data using sky frames generated for each processed image. For a given image the running (moving) median from a subset of frames within selected list distance of each frame (excluding the given frame) is processed to produce a SKY frame. This mode is most appropriate for "quick look" type processing of data where the stability of the sky is either unknown or is known to be varying on a time scale shorter than the full data set (so that more traditional approach of grouping skys over long periods is inadequate). The number of frames which enter into the sky can be selected. To facilitate list processing, the data processing can begin and end at selected element in the list. Data which are off source can be optionally included in the running sky frame generation but withheld from full processing into final images. MOVPROC makes no distinction between on and off source frames when determining membership in the sky lists.
MOVPROC:
* input = Input raw images
* output = Output image descriptor: @list||.ext||%in%out%
* flatimage = Input flat field image name
* (blankimage = "") Input blank field image name: "compute"==compute
(fixpix = no) Run FIXPIX on data?
(badpix = "badpix") badpix file in FIXPIX format
(setpix = no) Run SETPIX on data?
(maskimage = "badmask") untransposed bad pixel image mask
(bvalue = 0.) Value if there are no pixels
* (orient = no) Orient image with N up and E left
* (verbose = yes) Verbose output?
* (logfile = "STDOUT") logfile name
* (include = 0) Number of included images in blankimage subset
* (improc = 0) Number of images to process in group
* (imskip = 0) Number of images to skip between process
* (first_proc = 1) List number of first image to be processed
* (last_proc = 1000) List number of last image to be processed
(darkimage = "null") Input dark_count image ("null"==noaction)
# IMCOMBINE parameters =========================================================
(norm_opt = "zero") Type of combine operation: |zero|scale|none|
(norm_stat = "median") Pre-combine common offset: |none|mean|median|mo
(comb_opt = "median") Type of combine operation: |average|median|
(reject_opt = "none") Type of pixel rejection operation
(statsec = "[300:700,400:600]") Image section for calculating statistics
(lthreshold = INDEF) Lower threshold for exclusion in statistics
(hthreshold = INDEF) Upper threshold for exclusion in statistics
(mclip = no) Use median, not mean, in clip algorithms
(pclip = -0.5) pclip: Percentile clipping parameter
(weight = "none") Image weights
(expname = "") Image header exposure time keyword
(nlow = 1) minmax: Number of low pixels to reject
(nhigh = 1) minmax: Number of high pixels to reject
(nkeep = 0) Min to keep (pos) or max to reject (neg)
(lsigma = 3.) Lower sigma clipping factor
(hsigma = 3.) Upper sigma clipping factor
(rdnoise = "0.") ccdclip: CCD readout noise (electrons)
(gain = "1.") ccdclip: CCD gain (electrons/DN)
(snoise = "0.") ccdclip: Sensitivity noise (fraction)
(sigscale = 0.1) Tolerance for sigma clipping scaling correction
(grow = 0) Radius (pixels) for 1D neighbor rejection
# IMCOMBINE parameters =========================================================
See APPENDIX A.4 for a discussion the input image parameter and APPENDIX A.5 for a discussion of the output image parameter.
MOVPROC processes data from list number "first_proc" to list number "last_proc". For each processed data frame from the list it creates a sky frame from the "include" number of images nearest the processed frame. This is equivalent to a running mean/median (selected by "comb_opt") which excludes the image being processed. The task is well suited to handle data with a regular pattern of "on" frames ("improc") interspersed with "off" frames ("imskip"). The parameters "patproc" and "imskip" together determine which of the data are processed into final images. Setting "imskip" to a number other than zero signals the task to avoid processing the "off" frames while still including them in the running sky frame.
The preferred setting for IMCOMBINE parameters is:
"norm_opt" = "zero" (additive corrections) "norm_stat" = "median" (median statistic) "comb_opt" = "median" (combine operation: median filter) "reject_opt" = "none
MOVPROC then would median filter ("comp_opt") the images which have been brought to a common median value ("norm_stat") by adjusting the median value within the "statsec" part of the image to a common value by changing adding/subtracting (norm_option) the appropriate constant prior to combining the images into a composite sky frame.
For SQIID data, the parameter 'orient' could be set to "yes". This would orient the images to the usual North up, East left orientation. Normally, the bad pixel corrections controlled by 'fixpix' and 'setpix' are deferred until the images are actually combined. However, if you wish to mask bad pixels at this stage set 'setpix = yes' and 'maskimage' to an mask image appropriate to the channel selected and the orientation of the raw image. Masked pixels are set to 'value'. (see APPENDIX A.7 for how to make a pixel mask.)
The processing order is:
If you want to keep a record of the MOVPROC operations set "logfile" to a filename (else set to ""). Setting "verbose = yes" increases the amount of information reported (including statistics). It is useful to remember that MOVPROC produces 4-byte real images.
For example, a 95 frame sequence of 5 off, 10 on, 5 off, 10 on, .... 5 on (ABBABBABBABBABBABBA where A is a set of five off-target sky frames and B is a set of 5 on-target object frames) could be processed by setting:
include=5
first_proc=6
last_proc=90
patproc=10
imskip=5
Any last_proc greater than 90 would achieve the same result. One gets 60
processed images corresponding to list numbers 6-15, 21-30, 36-45, 51-60, 66-75,
and 81-90.
PATPROC processes raw image data generated for representative data aquisition protocols which intersperse object (+) and sky (-) frame in known fixed repeating patterns. These patterns vary in the amount of time devoted to off target images:
all_on: + + + + + + all time on source field pair: +- +-+ - equal time on and off triad: +-+ +-+ +-+ 2/3 on and 1/3 off alt-triad: -++ -++ -++ quad: +--+ +--+ +--+ equal time on and off but less telescope motion alt-quad: -++- -++- -++- than pair
The data are normally proceed - sky subtracted, then flat fielded, etc, using the same protocol as UPSQPROC. However, the sky frames are generated from the data (sub)set as part of the data processing. The sky frames can come from all the data, all the off-source frames, or the off-source frame nearest in order time in the data sequence to a given frame. The data at each +|- pattern position can either be single frames or same-size groups of images. When observing conditions are sufficiently stable to permit one to benefit from a composite sky (improved noise and spatial structure rejection), PATPROC can be used to generate composite off-source frames for subtraction from the appropriate on-sky frames. Each group (size="multiple") off-field images is used to generate composite sky frames for each member of the (size = "multiple")0 group. The memeber of a given group of on-source frames is processed using the group sky frame nearest in time in the data sequence.
PATPROC:
* input = Input raw images
* output = Output image descriptor: @list||.ext||%in%out%
* flatimage = Input flat field image name
* skyimage = Input blank field image name
* (order = 1) Pattern # 1:++ 2:+- 3:+-+ 4:+--+ 5:-++- 6:-++ ?
* (multiple = 1) # of frames at each +/- pattern state?
* (seq_id = "none") Sequential imager id template? (eg, ".00"|"000")
(rescale_opt = "none") Rescale blank to object via: |none|mean|median|
(sec_norm = "[300:700,400:600]") Image section for calculating statistics
(save_blank = no) Save generated blank_frame?
(fixpix = no) Run FIXPIX on data?
(badpix = "badpix") badpix file in FIXPIX format
* (setpix = no) Run SETPIX on data?
* (maskimage = "badmask") untransposed bad pixel image mask
* (value = 0) pixel value for masked pixels
* (orient = no) Orient image with N up and E left
(verbose = yes) Verbose output?
* (logfile = "STDOUT") logfile name
# IMCOMBINE parameters =========================================================
(comb_opt = "median") Type of combine operation: |average|median|
(reject_opt = "none") Type of pixel rejection operation
(norm_opt = "zero") Type of pre-combine operation: |zero|scale|none
(norm_stat = "median") Pre-combine common statistic: |none|mean|median
(mclip = no) Use median, not mean, in clip algorithms
(pclip = -0.5) pclip: Percentile clipping parameter
(lthreshold = INDEF) Lower threshold for exclusion in statistics
(hthreshold = INDEF) Upper threshold for exclusion in statistics
(weight = "none") Image weights
(expname = "") Image header exposure time keyword
(nlow = 1) minmax: Number of low pixels to reject
(nhigh = 1) minmax: Number of high pixels to reject
(nkeep = 0) Min to keep (pos) or max to reject (neg)
(lsigma = 3.) Lower sigma clipping factor
(hsigma = 3.) Upper sigma clipping factor
(rdnoise = "0.") ccdclip: CCD readout noise (electrons)
(gain = "1.") ccdclip: CCD gain (electrons/DN)
(snoise = "0.") ccdclip: Sensitivity noise (fraction)
(sigscale = 0.1) Tolerance for sigma clipping scaling correction
(grow = 0) Radius (pixels) for 1D neighbor rejection
# IMCOMBINE parameters =========================================================
See APPENDIX A.4 for a discussion the input image parameter and APPENDIX A.5 for a discussion of the output image parameter.
The "blankimage" parameter is used to select the type of SKY frame generated:
vaid_image_name : use this image as basis for blank image
null : don't subtract blank image
make_all : make sky from all images
make_off : make sky from all off images
make_order : use sky from order selected off field nearest neighbor(s)
in the data sequence
The pattern "order" number identifies the fixed data acquisition pattern:
1: all_on: + + + + 2: pair: +- +- 3: triad: +-+ +-+ 4: quad: +--+ +--+ 5: alt-quad: -++- -++- 6: alt-triad: -++ -++
Each image in the sequence is processed using the sky frame nearest in time.
The parameter "multiple" extends the capabilities of PATPROC to cases where the input list to contain more than one image at each +|- pattern position. The "multiple" off-field images are grouped to generate composite sky frames for each member of the "multiple" group.
The "seq_id" parameter depicts the sequential imager id template (".00"|"000") for generating descriptive sky names for the case "blankimage" = "make_order". The images are assumed to be named according to the following convention:
image_name == imroot//seq_id//"."//imextn
seq_id == seq_mark//seq_number
For example, for an image name of 982507.012.fits, sed_id = ".000" or ".012" and for an image name of image012.fits, seq_id = "000" or "012". The number of digits and the sequence marker ("" or "." above) are important.
The preferred setting for IMCOMBINE parameters is:
"norm_opt" = "zero" (additive corrections) "norm_stat" = "median" (median statistic) "comb_opt" = "median" (combine operation: median filter) "reject_opt" = "none
PATPROC then would median filter ("comp_opt") the images which have been brought to a common median value ("norm_stat") by adjusting the median value within the "sec_norm" part of the image to a common value by changing adding/subtracting (norm_option) the appropriate constant prior to combining the images into a composite sky frame.
For SQIID data, the parameter 'orient' could be set to "yes". This would orient the images to the usual North up, East left orientation. Normally, the bad pixel corrections controlled by 'fixpix' and 'setpix' are deferred until the images are actually combined. However, if you wish to mask bad pixels at this stage set 'setpix = yes' and 'maskimage' to an mask image appropriate to the channel selected and the orientation of the raw image. Masked pixels are set to 'value'. (see APPENDIX A.7 for how to make a pixel mask.)
The processing order is:
If you want to keep a record of the PATPROC operations set "logfile" to a filename (else set to ""). Setting "verbose = yes" increases the amount of information reported (including statistics). It is useful to remember that PATPROC produces 4-byte real images.
For example, a 95 frame sequence of 5 off, 10 on, 5 off, 10 on, .... 5 on (ABBABBABBABBABBABBA where A is a set of five off-target sky frames and B is a set of 5 on-target object frames) could be processed by setting:
blankimage="make_order"
order=6
multiple=5
One gets 60 processed images corresponding to:
list numbers 6-10 processed with composite sky from list numbers 1-5
11-15 16-20
21-25 16-20
26-30 31-35
36-40 31-35
40-45 46-50
51-55 46-50
56-60 61-65
66-70 61-65
71-75 76-80
81-85 76-80
86-90 91-95
An 90-image list would have reused the 76-80 sky to process 86-90.
V. Building a Database from a Dataset
Following the calibrations applied to the individual frames discussed above, each dataset needs to be converted into a "database". This applies even to datasets consisting of a single exposure on a target. As with datasets, each database contains the observations of a single field or target in a single filter passband. The database construct will be used to bring the individual exposures together into the desired final image (dither or mosaic) with appropriate spatial and intensity offset registration.
The 'UPSQIID' package contains a set of image registration tools designed to facilitate image registration and combination. The image registration process naturally separates into the following steps:
The 'SQIID' package image registration tools are designed to operate from ordered image lists. The initial data flow through the system is determined to a large extent by the degree of spatial overlap among the images in a given data set. IR observations of extended regions (FOV greater than array FOV) are built up from spatial grids of pointed data sets which overlap on the edges. While the degree of spatial overlap is an uneasy compromise between the competing desire to map efficiently and the need to have objects in common along the edges, these data do not in general share a common overlap region. IR observations of high sensitivity are built up from pointed data sets which are heavily overlapped and share a (generally large) region in common. (The process of producing such data multiply overlapped with relatively small spatial offsets between frames is commonly called "dithering".) Off course, actual data collection may involve a complex hiearchy spatial offsets (e.g., grids of dithered data) and might require cobination of registration techniques. The dataflow for purely WIDE or DEEP observations through the 'UPSQIID' package image registration tools proceeds as follows after generation of the appropriate image list:
WIDE DEEP
tile spatially ordered image usqmos
interactive spatial registration center xyget
link registration xylap
intensity match zget zget
make composite image nircombine nircombine
These tools generate a database which links the untouched processed data (generally either stored as mosaic of tiled images using USQMOS, or as lists of images) via a prescription which describes how each image is the be transformed, shifted, intensity offset, and masked to make a composite image. The input images (in the database mosaics or lists) are not themselves modified during this process; rather, each tool in turn inserts the necessary information into the database text files. When required, temporary copies of the images are created, modified, and discarded at task completion. One can edit the prescription as required to meet the special needs of each data set. Tools are provided for merging registration data which share a common image into larger databases. Ultimately, the images are combined according to the database prescription using the powerful IRAF task 'imcombine' (see APPENDIX A.6 and its IRAF help page for a discussion of IMCOMBINE).
The user is warned that while the toolkit exists and is filled with (most) of the requisite tools, the high-level tasks which would automate the registration task have not been written - you are still in for a lot of work! The user is advised to read APPENDIX A.9 for guidelines on "keeping track of your reductions".
The 'SQIID' task 'usqmos' can be used to gather observations of a single object for a given filter into its own database. Each database consists of a text file and an image containing a mosaic (with no overlap) of the images in the database. Hence the disk storage requirements grow during this stage of the processing. If possible, retain the individual images and calibration frames on the disk since iteration of the processing is often necessary and usually involves the recreation of the FLAT or SKY frame and the reprocessing of the individual images. (See APPENDIX A.9 for assistance in "keeping track of things".)
Note: 'usqmos` calls the 'noao.nproto.irmosaic' task which is also part of the 'noao.imred.irred' package loaded by 'UPSQIID'. USQMOS is really just a pre and post processor for IRMOSAIC, initializing the 'UPSQIID' registration database files. See APPENDIX A.11 and the help page for IRMOSAIC for more details.
NB: Given the 512X512 image size for SQIID data, this step has become impractical for grids above 4 X 4 so that some other mechanism will be required to operate on larger grids.
A dataset of processed images is assembled into a database with the task 'usqmos'. For example,
The task 'usqmos' has a number of parameters which should be set as indicated in the "lpar" listing below:
USQMOS:
* input = Input images
* output = Output image
* nxsub = Number of subrasters in x
* nysub = Number of subrasters in y
(trim_section = "[*,*]") Input image section written to output image
* (null_input = "") List of missing input images
* (corner = "ll") Starting corner for the mosaic
* (direction = "row") Starting direction for the mosaic
(raster = no) Raster scan?
(median = no) Compute the median of each subraster?
(median_secti = "[*,*]") Input image section used to compute median
(subtract = no) Substract median from each subraster?
(oval = 10000.) Mosaic border pixel values
(nimcols = INDEF) The number of columns in the output image
(nimrows = INDEF) The number of rows in the output image
(opixtype = "") Output pixel type?
(tran = no) Apply image transform to mosaiced images?
(task_tran = "geotran") transform task: imlintran|geotran
(db_tran = "") name of database file output by GEOMAP
(co_tran = "") name of coordinate file input to GEOTRAN
(geom_tran = "linear") GEOTRAN transformation geometry
(max_tran = yes) Offset GEOTRAN to save maximum image
(interp_tran = "linear") GEO(IMLIN)TRAN interpolant
(bound_tran = "nearest") GEO(IMLIN)TRAN boundary
(const_tran = 0.) GEO(IMLIN)TRAN constant boundary extension valu
(flux_tran = yes) conserve flux during GEO(IMLIN)TRAN
(save_tran = no) Save the intermediate GEO(IMLIN)TRANed images?
(save_dbmos = no) Save the IRMOSAIC database file?
(logfile = "STDOUT") Log file name
(infofile = "") Output information file name
The input image list is specified as above (it wants to be an @list). The output file name (without extension) will be the rootname of both the database mosaic image (with the extension .imh) and the database text file (with the extension .dbmos). The images must all be the same size. Setting 'infofile' to other than "" names the output database file accordingly.
The 'nxsub' and 'nysub' parameters specify the dimensions of the database mosaic. When there will be unfilled cells in the mosaic, a list of the missing images must be provided in the parameter 'null_input'. For example, if the dataset contains 10 images, a 3 by 4 mosaic could be specified. Then 'null_input' should be "11,12" or "11-12". If their are no null images, set 'null_input = ""' (a " " will cause the task to abort!). The image numbering refers to the postion of the image in the input list.
The images in the input list will be tiled into a mosaic with 1-pixel boundaries (value determined by 'ovalue') according to the selected prescription. The parameter "corner", which indicates where to start tiling the images from the input list and the parameter "direction", which indicates the direction (row or column) to proceed with the tiling operation, should be set to the values which describe the way you have ordered your data in the input list. A natural choice might be 'corner' = "ll"' (lowerleft) and 'direction = "row"'. Then the images in the database mosaic would be numbered starting with the lower left image as "1" and proceeding by row. For example,
9 10 11 12 5 6 7 8 1 2 3 4are the image numbers in the above example. The options to subtract the median and to transform the image should be set to "no" since these steps are done elsewhere. Setting 'median = yes' calculates and reports the image statistics. However, occasionally (for reasons unknown), 'irmosaic' can take an inordinately long time to perform these calculations.
The parameters 'task_tran', 'db_tran', 'co_tran', 'geom_tran', 'max_tran', 'interp_tran', 'bound_tran', 'const_tran', 'flux_tran' are used to geometrically transform the images prior to storage in the mosaic. Normally one saves the data as is and defers image transforms to later stages. (See APPENDIX A.12 for further discussion.)
Note that if the task aborts for any reason, copies of the raw database file produced by 'irmosaic' (they have "db" prepended to the "output" name) should be deleted before proceeding, since the 'irmosaic' task appends data to any existing database by that name.
When assembling the database for the other channels, it is important to realize that the data at each channel must be tiled into its USQMOS mosaic at the same locations. Subsequent 'UPSQIID' registration tasks assume that the mosaics are equivalent accept for channel. Use of the "@list:channel_id" convention assures that such will be the case.
VI. Specifying the Relative Spatial Offsets
To combine a database of images into a single, final image, the relative offsets between the individual images in the database must be determined. IRAF does not yet have a general image registration task and many of the tools which are available (see APPENDIX A.11) are tailored to tiling images sets taken in a fixed grid pattern with minimal overlap (The images are not combined: each successive image covers up its neighbors in the overlap region.) See also the help pages for the tasks 'proto.imalign' and 'proto.imcentroid' for descriptions of system tasks which are useful for multiply-overlapped images which share an object in common.
The interactive determination of the offsets is accomplished by identifying common sources in the images in the database mosaic and requires an IRAF image display tool (e.g., 'ximtool' or 'saoimage'). Since registration of grids of images with minimal overlap between adjacent images (taken to cover large area) and heavily overlapped images which share a common overlap region (taken to reach faint magnitudes) require different strategies, there are two sets of tools to accomplish this task:
These tools produce a common database which can be used by subsequent steps in the registration process. In addition, the tool 'mergecom', discussed below, allows you to merge databases from both types of data into a common database. (e.g., to bring the second members of dithered pairs for grid data in registration with the first members of the pair.)
Grid data are most easily handled using the 'center' task discussed in APPENDIX A.10. To use this task, the images must be ordered in the USQMOS image so that overlapping images are adjacent to one another. If data are in dithered pairs on an M x N grid, place the first members of each pair in an appropriate order in an M x N grid within the USQMOS image and the second pair in the same order either above it in the same image (to make a M x 2N image) or in a second USQMOS image. Display the USQMOS image "usqmosl" and enter the 'center' task by typing:
Using the image cursor (see APPENDIX A.10) successively mark images which appear in adjacent frames: the data must be paired such that each member of the pair corresponds to the position of the selected object in an adjacent frame. The overall order in which the object pairs is selected is unimportant. Avoid very bright stars (which might be saturated) and stars at the edge of the subrasters (whose position can be influenced by the one pixel gap between adjacent sub-rasters. Hopefully, all edge pairs contain at least one object within their common overlap region.
The completed file, called "usqmosh.ctr.1" by default, becomes the basis file for the 'xylap' task, which collects and averages the spatial offsets between adjacent frames (calling 'getcenters') and links them into a common map (calling 'linklaps'). For, example, you can access the 'xylap' by typing:
XYLAP employs many of the conventions of the IRALIGN/IRMATCH1(2)D IRAF tasks discussed in APPENDIX A.11. Task parameters for 'xylap' are as follows:
XYLAP:
* mosimage = usqmosh MOSAIC image name
* basisfile = usqmosh.ctr.1 File produced by CENTER||IRMATCH
* (linkfile = "default") File with selected XY linkage paths
* (mos_info = "default") Images info file from IRMOSAIC|USQMOS
* (nx_sub = INDEF) Number of input images along x direction
* (ny_sub = INDEF) Number of input images along y direction
* (nxrsub = INDEF) index of x reference subraster
* (nyrsub = INDEF) index of y reference subraster
* (guess = no) Replace null links with average values?
* (new_origin = yes) Move origin to lower left corner?
* (trimlimits = "[0:0,0:0]") trim limits on the input subrasters
(verbose = yes) Verbose output during LINKLAPS?
* (outfile = "") Output file name
(tran = no) Request GEOTRAN mosimage before IMCOMBINE?
(db_tran = "") name of database file output by GEOMAP
(geom_tran = "linear") GEOTRAN transformation geometry
(max_tran = yes) Offset GEOTRAN to save maximum image?
(interp_tran = "linear") GEOTRAN interpolant
(bound_tran = "nearest") GEOTRAN boundary
(const_tran = 0.) GEOTRAN constant boundary extension value
(flux_tran = yes) Conserve flux upon GEOTRAN?
(interp_shift = "linear") IMSHIFT interpolant (nearest,linear,poly3,poly5
(logfile = "STDOUT") Logfile name
mosimage
name of the 'usqmos' image to be registered
basisfile
'basisfile' is a text file listing the coordinates of objects in the
input image one object per line in the following format:
1) the x and y coordinates of the object in the first subraster
2) the x and y coordinates of the same object in the second subraster
3) the x and y coordinates of the next object in the first subraster etc.
The ouput of the 'center' task, or any other file with these attributes,
can be used.
link_file = "default"
By default ("default") the linkage path for each subraster is computed
along two orthogonal paths and the average linkage is used: one path
proceeds along the row to the reference subraster row, then along the
column; the other path proceeds along the column and then along the row.
If 'link_file' is given a linkage filename, that file is used to
determine the linkage paths for each subraster. Since arbitrary
linkage paths are allowed, one can accommodate problem images which
are not linked on sufficient sides. You need to walk each piece to the
same final link (one path per line), giving the path position of each
image you trespass on the way from each subraster to the number of the
reference subraster declared by 'nxrsub' and 'nyrsub'.
Here are three sample formats for xylap linkfiles for the 3 x 4 mosaic
in the 'usqmos' example above and nxrsub = 2 and nyrsub = 2:
Tile order: 9 10 11 12
5 6 7 8
1 2 3 4
file1 file2 file3
1 2 6 1 5 6 1 2 6
2 6 2 6 2 6
3 2 6 3 7 6 3 2 6
4 3 2 6 4 8 7 6 4 8 7 6
5 6 5 6 5 6
6 6 6
7 6 7 6 7 6
8 7 6 8 7 6 8 7 6
9 10 6 9 5 6 9 10 6
10 6 10 6 10 6
11 10 6 11 7 6 11 7 6
12 11 10 6 12 8 7 6 12 8 7 6
File1 proceeds along the row to the reference subraster row, then along
the column; file2 procceds along the column, then along the row; file3
contains a variety of linkages to avoid missing links.
nx_sub = INDEF, ls ny_sub = INDEF
The number of subrasters along a row and a column of the input image.
This will defaults to the entire image. If you are only registering
part of the image (e.g., the lower half of the dithered M x 2N example
above) you need to enter the appropriate numbers here.
nxrsub = INDEF, ls nyrsub = INDEF
The column and row index of the reference subraster. This will
default to the central subraster.
guess = no
If 'guess = yes', links which are absent from the database are
estimated on the basis of the average value for the link along the
row or column where the link is missing. Setting 'guess = no'
prevents the use of average links when none exists. (By default,
orthogonal paths are computed and the average is used; if one path
exists and the other doesn't, one does not want the average!).
new_origin = yes
Setting 'new_origin = yes' tells the task to find the size of the
combined image in the database and shift the implied origin of the
database to include all of the images by putting the database origin
in the lower left corner of the dataset. 'new_origin = no' does not
reset the origin from wherever it happens to be.
trimlimits = "[1:1,1:1]"
The number of columns or rows to trim off each edge of each input
subraster before inserting it in the output image in section notation
([left:right,top:bottom]). The default is to trim 1 column or line at
each edge. These trims differ with each channel (see the SQIID
manual for appropriate choices. The input images are not touched;
rather the database is marked to exclude these edge regions from the
final image.
tran = no
Setting 'tran = yes' requests that the task transform the database
coordinates according to the 'geotran' conventions and to mark the
database so that the images will be geotransformed during the
combination process. Setting 'tran = no' leaves things alone.
'db_tran', 'geom_tran', 'max_tran', 'interp_tran', 'bound_tran',
'const_tran', 'flux_tran' are parameters employed by GEOTRAN (see
APPENDIX A.12) and should be left at their default values.
They are ignored when 'tran = no'.
interp_shift = "linear"
The type of interpolant used to shift the subrasters. Uses IMSHIFT
convention ("linear", "nearest", "poly3", "poly5" and "spline3" are
allowed); "linear" is the perferred choice.
outfile = ""
Outfile is the name of the database file generated. If 'outfile = ""'
or 'outfile = "default"', this file is named "mosimage.xylap".
As the XY offset data are being collected, XYLAP ('getcenters' task) signals the presence of mis-matched object pairs as follows:
During linkage operation XYLAP ('linklaps' task) signals missing links and the action taken:
#DBL Note: no data for link | r 2,3 | #DBL Note: using row_ave para_laps 3 for link | r 2,3 | #DBL Note: no data for link | c 2,1 | #DBL Note: using row_ave perp_laps 1 for link | c 2,1 | #DBL Note: no data for link | c 2,2 | #DBL Note: using row_ave perp_laps 2 for link | c 2,2 | #DBL Note: null links not used in pairsNote: XYLAP, which uses the output from the CENTER task is set up to operate on mosaics no larger than 10 in X or Y. (IRAF scripts do not handle arrays well and larger arrays eat up too much space on the stack.)
Many datasets are either irregularly or multiply-overlapped and cannot be put into a simple grid. The 'xyget' task is designed to accommodate datasets which share an object in common. In many instances, the entire dataset will share a common object and 'xyget' can be run once on the entire dataset. In other cases, subsets of data will share a common object and the dataset can be broken into a series of multiply-overlapped images. Then 'xyget' can be run on each subset and on pairs of images which link each subset and the 'mergecom' task (discussed below) can be used to merge the data into a single database for the entire subset. For example,
xyget usqmosl "1-10|1" starting from USQMOS image "usqmosl"
xyget @sqlist "1-10|1" starting form SQIID image list "sqlist"
will permit the user to interactively indentify a number of point sources in image 1 and then a single source which appears in each of images 1 to 10. Prior to running this task it is advisable to examine the database mosaic image to identify a single point source which is present on all of the images. During the manual registration process one can reject (exclude from further consideration as part of the registration database) specific frames by typing "x" (instead of spacebar) when the offending frame is up for review.
Selected XYGET parmeters are discussed below:
XYGET:
* images = sqmosh MOSAIC image name | @list of images to compare
* frame_nums = "1-10|1" Selected frame numbers within MOSAIC|@list
getoffset = yes Do you want to get frame offsets?
answer = yes Do you want to continue?
compute_size = yes Do you want to [re]compute image size?
* (mos_info = "default") Images info file from IRMOSAIC|USQMOS
* (ref_info = "default") Reference info file from IRMOSAIC|USQMOS
(outfile = "") Output info file - default: images.xycom
* (trimlimits = "[0:0,0:0]") trim limits on the input subrasters
(coord_in = "") Input initial coordinate file
(in_shifts = "") Initial shift file between ref and images
(bigbox = 11.) Size of coarse search box
(boxsize = 7.) Size of final centering box
(tran = no) Request GEOTRAN images before IMCOMBINE?
(db_tran = "") name of database file output by GEOMAP
(geom_tran = "linear") GEOTRAN transformation geometry
(max_tran = yes) Offset GEOTRAN to save maximum image?
(interp_tran = "linear") GEOTRAN interpolant
(bound_tran = "nearest") GEOTRAN boundary
(const_tran = 0.) GEOTRAN constant boundary extension value
(flux_tran = yes) Conserve flux upon GEOTRAN?
* (zscale = no) DISPLAY using zscale?
* (z1 = 0.) minimum greylevel to be displayed
* (z2 = 1000.) maximum greylevel to be displayed
* (verbose = no) Verbose reporting
images
The name of the USQMOS mosaic image or the image file "@-file"
you want to register. An "@-file" is file containing a list of
image names, (one per line) - the filename is preceded by "@" to tell
IRAF to look for images within the file. NB: When submitting an
"@-file" for images stored within an USQMOS image, you need to give the
full name of the USQMOS mosiac database in 'mos_info' instead of using
the default.
frame_nums
A specially formatted string which tells which images to register
and what image to register them against. The images are identified
by their path order in the USQMOS database (MOS_XXX lines, where
XXX is the path number). The format for 'frame_nums' consists of up
to two to three fields separated by "|". (Note, because of the use
of the "|" delimiter, the 'frame_nums' string must be encloded in
quotes). The first field indicates the range of images to be
registered; the second field indicates the number of the reference
image these images are to be referenced to; the third field (when
present) gives the name of the 'usqmos' image where the reference
image can be found. By default, the images are all from the same
database; inclusion of the third field allows you to register images
against another image not in the current database. Since the
'SQIID' registration database files assign a single line which begins
with "COM_XXX", where XXX is the order of the image in the list these
frame_numbers will often be refered to as COM# (see APPENDIX A.13).
RANGE SYNTAX
The syntax for the range portion of the frame_num string consists of
positive integers, '-' (minus), '*', ',' (comma), and whitespace.
The commas and whitespace are ignored and may be freely used for
clarity. The remainder of the string consists of sequences of three
fields. The first field is the beginning of a range, the second is a
'-', and the third is the end of the range. The following examples
illustrate the range syntax:
range result
1-10 1 2 3 4 5 6 7 8 9 10
1,5,9 1 5 9
1-3,5 1 2 3 5
* 1 to the number of 'usqmos' subrasters read from the database
Warning: since this number is the original size of the
USQMOS mosaic, it may not include all the images in a merged
dataset.
For example:
'frame_nums = "1-4|2"' registers images 1,2,3,4 against image 2 in
the same mosaic
'frame_nums = "1-4|2|mosh2' registers the same images against imagex
2 in the mosaic mosh2
mos_info = "default"
Name of the 'usqmos' database file associated with the 'images'
parameter. 'mos_info = "default" points to the file "images.dbmos"
ref_info = "default"
Name of the 'usqmos' database file associated with the indirect image
reference field in the 'frame_nums' string noted above.
'ref_info = "default" points to the file "third_field.dbmos"
outfile = ""
Outfile is the name of the database file generated. If 'outfile = ""'
or 'outfile = "default"', this file is named "images.xycom".
trimlimits = "[1:1,1:1]"
The number of columns or rows to trim off each edge of each input
subraster before inserting it in the output image in section notation
([left:right,top:bottom]). The default is to trim 1 column or line at
each edge. These trims differ with each channel (see the SQIID
manual for appropriate choices. The input images are not touched;
rather the database is marked to exclude these edge regions from the
final image.
tran = no
Setting 'tran = yes' requests that the task transform the database
coordinates according to the 'geotran' conventions and to mark the
database so that the images will be geotransformed during the
combination process. Setting 'tran = no' leaves things alone.
'db_tran', 'geom_tran', 'max_tran', 'interp_tran', 'bound_tran',
'const_tran', 'flux_tran' are parameters employed by GEOTRAN
(see APPENDIX A.12) and should be left at their default values.
They are ignored when 'tran = no'.
The parameter 'getoffset' will be prompted for at runtime and set to "yes". The parameter 'trimlimits' excludes masked regions on the edges of the detectors. The values for this parameter have changed over time during the development of the SQIID. Appropriate values are included in the appendix. Since the interactive offsets are determined for the H band images which will be used as the basis of the final transformation, the parameter 'tran' must be set to "no".
The scaling of the image display during the interactive phases of the 'xyget' task may be automatic if the parameter 'zscale' is "yes", otherwise the images will be scaled to fill the range between the values of the parameters 'z1' and 'z2'. It is often best to set the parameter 'verbose' to "yes".
Since the relative geometries of the detectors remain constant over an observing run (in fact, between instrument disassemblies), once the H band relative offsets are known the offsets fo images in the other passbands can be automatically determined. The necessary calibrations reside in files as documented in the APPENDIX A.12. The task "xyadopt" will put the necessary information into the database text files for the J, K, and L databases. For example to handle the J channel,
will obtain the relative offsets from the file "sqmosh.xycom" (which is created by "xyget") to produce the registration file "sqmosj.xycom" appropriate for the J channel image.
XYADOPT uses geomap output files appropriate for the observing run and converts assembly instructions of a given channel to those needed to combine another channel. The database is transformed so that the XY locations are OK, and the images are set-up to be geotran`ed into the coordinate system of the target channel prior to combination.
Selected XYAPOPT parameters are described below:
XYADOPT:
* images = IRMOSAIC image name | @list of images to compar
* frame_nums = "*|1" Selected image numbers within IRMOSAIC
* basis_info = Lap_basis info file from GETCOMBINE|IRCOMBINE
answer = yes Do you want to continue?
(mos_info = "default") Images info file from IRMOSAIC|MKMOS
* (trimlimits = "[0:0,0:0]") trim limits on the input subrasters
(outfile = "") Output information file
* (tran = yes) Request GEOTRAN images before IMCOMBINE?
* (db_tran = "") name of database file output by GEOMAP
(geom_tran = "linear") GEOTRAN transformation geometry
(max_tran = yes) Offset GEOTRAN to save maximum image?
(interp_tran = "linear") GEOTRAN interpolant
(bound_tran = "nearest") GEOTRAN boundary
(const_tran = 0.) GEOTRAN constant boundary extension value
(all_frames = no) Use entire frame_nums range (else nsubraters)
(flux_tran = yes) Conserve flux upon GEOTRAN?
* (verbose = no) Verbose reporting
images
The name of the SQMOS mosaic image you want to register. Although
an @list indicated a file containing image names (one per line) is
accepted, the software has not been fully tested in that mode.
frame_nums
A specially formatted string which tells which images to register
and what image to register them against. Same syntax as discussed
in XYGET above. Since the 'sqiid' registration database files assign
a single line which begins with "COM_XXX", where XXX is the order of
the image in the list these frame_numbers will often be refered to
as COM# (see APPENDIX A.13).
basis_info
Name of the database file produced by either 'xylap' or 'xyget' for
the channel you whose registration you wish to adopt
trimlimits = "[1:1,1:1]"
The number of columns or rows to trim off each edge of each input
subraster before inserting it in the output image in section notation
([left:right,top:bottom]). The default is to trim 1 column or line at
each edge. These trims differ with each channel (see the SQIID
manual for appropriate choices. The input images are not touched;
rather the database is marked to exclude these edge regions from the
final image.
outfile = ""
Outfile is the name of the database file generated. If 'outfile = ""'
or 'outfile = "default"', this file is named "images.xycom".
tran = yes
Setting 'tran = yes' requests that the task transform the database
coordinates according to the 'geotran' conventions and to mark the
database so that the images will be geotransformed during the
combination process. Setting 'tran = no' leaves things alone.
'geom_tran', 'max_tran', 'interp_tran', 'bound_tran', 'const_tran',
'flux_tran' are parameters employed by GEOTRAN (see APPENDIX A.12) and
should be left at their default values. They are ignored when
'tran = no'.
db_tran
Name of the GEOMAP database file which transforms the current
channel image to that of the adopted channel. See APPENDIX A.12
for a discussion of these files. (Note: since the path to the
associated co-ordinate file, ".geoco" is found within the 'db_tran'
".geodb" file, they both must be in the current directory to be found
properly.)
In the current example, the parameter 'basis_info' should be set to the H band ".xycom" file and the parameter "'rimlimits' must be set for each detector (see the description of XYGET above). The parameter 'tran' must be set to "yes". The transformation coordinates are in the appropriate ".geodb" file as described in APPENDIX A.12. This filename must be specified in the parameter 'db_tran'. (Note: since the path to the associated co-ordinate file, ".geoco" is found within the ".geodb" file, they must be in the current directory to be found properly.)
XYGET/XYLAP and MERGECOM allow you to operate in larger domains. If you assemble the data in smaller mosaics, you can pull them together using XYGET to link the individual mosaics and MERGECOM to put it together into a single prescription. As noted above, XYGET will accept pieces from different mosaics. The format can be summarized as follows:
"Range in included image "COM# | reference COM# | mosaic name of reference" the final "|" and field is left out if they are from the same mosaic if the reference USQMOS database is mos_name.dbmos, set 'ref_info = "default"' else set 'ref_info' = name of USQMOS database for reference image
Collect the names of the separate registration database files (same channel only, one per line) into a file (e.g., usqmosh.merge) and use 'mergecom' to merge them into a single file (e.g., usqmoshall.xycom) as follows:
Note: You must feed the information to mergecom in the proper order when you organize the list of filenames: each reference frame must be found within the list of frames which has accummulated from the files earlier in the list.
Selected MERGECOM parameters are discussed below:
MERGECOM:
infofiles = @usqmosl.merge files produced by XYGET|XYLAP|IRCOMBINE
mergedfile = usqmoshall.xycom name of resultant composite image
compute_size = yes Do you want to [re]compute image size?
answer = yes Do you want to continue?
(subset = no) Keep only COM_000 and mos_name COM's?
(mos_name = " ") mosaic name for saved COM's
(renumber = no) renumber secondary referenced frames?
(verify = no) Verify alias before proceeding?
infofiles
List of registration files you wish to merge. Either a comma
delimited inline list or an "@list" filename with the files listed one
per line. Remember to feed the information to mergecom in the proper
order when you organize the list of filenames:
Each reference frame must be found within the list of frames
which has accummulated from the files earlier in the list.
mergedfile
Name of resultant composite file.
compute_size = yes
After all files have been accepted into the database, you will be
prompted:
Do you want to [re]compute image size? (yes)
Answer "yes" if you wish to have the task to find the new size of the
combined image in the database and shift the implied origin of the
database to include all of the images by putting the database origin
in the lower left corner of the dataset. Answer "no" if you want
to leave things as they are.
subset = no
If some of the 'infofiles' include references to reference images
outside the dataset which you do not want to include in the
'mergedfile', set 'subset = yes'. The task will then only include
those images which have the rootname 'mos_name'.
mos_name = " "
Root name of the 'usqmos' mosaic image which you wish to include in
the 'mergedfile'. Ignored if 'subset = no'.
renumber = no
Set 'renumber = yes' if you want to automatically renumber COM# after
the first file, so they cannot conflict with earlier COM#. Each new
file is renumbered to begin where the previous file left off.
verify = no
MERGECOM makes the reference frame found in the first file the
master reference frame for the entire dataset (its the COM_000 image).
Each subsequent file is searched for this image. If it cannot find
the image, it will search for a match between the reference image in
the current file and any image in the accumulated list of images.
If 'verify = no' it automatically begins this search. If 'verify =
yes', it prompts you:
If you answer "no", the task ends; else it continues.
When you are satisfied with the result, you can use XYADOPT on the merged file to create merged registration database files for your other channels.
For a variety of reasons, including changing sky background, the processed images will generally not perfectly match inside their regions of common overlap. The 'zget' task is used to determine the relative intensity offsets between images you wish to combine into a composite image. If the images share a sizable overlap region which is not dominated by high surface brightness, intensity (z) registration can be deferred to the final stage in the image combination process, the 'nircombine' task itself.
However, for the more general case, where images do not share a proper common overlap region, the ZGET task is generally able (albeit slowly!) to get the job done. You must specify the linkage between overlapping images you want to use to accomplish this registration. The task uses the registration database to determine the overlap regions for each image pair. Copies of the images are geotransformed, fractionally shifted, and masked. The statistics (selected by 'common') within the overlap regions of each image pair are determined and the prescription is them followed to accumulate the net offset of each image compared to the first image in the prescription. Currently, the entire overlap region is included in the statistics.
Following our example for the mosaic "usqmosl", you can use 'zget' by typing:
Selected ZGET parameters are discussed below:
ZGET:
* match_name = name of resultant composite image
* infofile = "" file produced by GETCOMBINE
* image_nums = Selected image numbers|ref_id
answer = yes Do you want to continue?
compute_size = yes Do you want to [re]compute image size?
(outfile = "") Output information file name
(gxshift = 0.) global xshift for final image
(gyshift = 0.) global yshift for final image
(goverlap = "") Global overlap section (overrides all else)
(gsize = "") Global image size (overrides all else)
(trimlimits = "[2:2,2:2]") added trim limits for input subrasters
* (setpix = yes) Run SETPIX on data?
* (maskimage = "badmask") bad pixel image mask
(fixpix = no) Run FIXPIX on data?
(fixfile = "badpix") badpix file in FIXPIX format
* (common = "median") Pre-combine common offset: |none|adopt|mean|med
* (lowerlim = -1000.) Lower limit for exclusion in IMSTATS
* (upperlim = 60000.) Upper limit for exclusion in IMSTATS
(interp_shift = "linear") IMSHIFT interpolant (nearest,linear,poly3,poly5
(bound_shift = "nearest") IMSHIFT boundary (constant,nearest,reflect,wrap
(const_shift = 0.) IMSHIFT Constant for boundary extension
* (svalue = -10000000.) Setpix value (<< floor)
(floor = -200.) Rejection floor for imcombine
(ceiling = 60000.) Rejection ceiling for imcombine
(ovalue = -1000.) Rejected pixel value for new image?
(size = no) Compute image size?
(save_images = no) Save images which were IMCOMBINED?
(verbose = yes) Verbose output?
(format = yes) Format output table
match_name
Name of the resulant composite image
infofile
'UPSQID' registration database file corresponding to the image you
wish to assemble. THe database files output by XYLAP, XYGET, ZGET,
or MERGECOM and any of their transformations via XYADOPT are acceptable.
Remember that you can edit COM-line prescription contained in these
files when necessary to get your desired result.
image_nums
'image_nums' tells the task how to determine the intensity offsets:
it tells which images to register and what image to register them
against. It can either be a specially formated string or an @list file.
The specially formatted string uses the same syntax as discussed
in XYGET above. The format can be summarized as follows:
"Range in included image "COM# | reference COM# | mosaic name of reference"
When a filename is given (with "@" prepended) the file must contain
a complete prescription of how to connect each image with its neighbor.
The format for each line is as follows:
"COM# of included image | COM# of reference"
You have considerable freedom of choice in assembling this file. The
only requirement on this file is that each image be connected to a
neighbor somewhere:
Pick a starting point (which only appears to the right of the "|").
Build up paths to each image (one per line).
Every image (except the starting point) must appear once and only
once on the left side of the "|".
Consider the following spatial orientation:
7 8 9
4 5 6
1 2 3
A simple files might be:
file1 file2
2|1 2|1
3|2 3|2
4|1 4|1
5|4 7|4
6|5 8|7
7|4 9|8
8|7 5|8
9|8 6|9 or 6|5 or 6|3
where file1 is a straight forward implementation and file2 would avoid
an unfortunate overlap between 4 and 5.
outfile = ""
Outfile is the name of the database file generated. If 'outfile = ""'
or 'outfile = "default"', this file is named "match_name.zcom".
trimlimits = "[2:2,2:2]" (Not currently used by ZGET)
The number of columns or rows to trim off each edge of each input
subraster before inserting it in the output image in section notation
([left:right,top:bottom]). The default is to trim 2 column or line at
each edge. This compensates for the 2 pixel border trashed (i.e. made
up) by the IMSHIFT operations. The input images are not touched; rather
the database is marked to exclude these edge regions from the final
image. However, the images the trimmed portions of the image will be
excluded during the image overlap statistics. Normally, 'trimlimits'
are cumulative in the database. This 'trimlimits' is not. It just
communicates to the task generating the overlap regions the message
that the edges are flawed so don't count them as useful during the
overlap computations.
common = "median"
'common' selects the statistic you want to use to determine the z
offsets between images. It can be either "mean", "median", or "mode".
lowerlim = -1000.
Lower limit for exclusion in IMSTATS during intensity offset operations.
When using 'setpix = yes' you must be sure to exclude the mask regions,
which will be set to 'svalue' from the statistics. You might consider
choosing a higher value to improve image statistics by restricting the
range. However, be sure to keep this limit safely below the lowest
real data in your image!
upperlim = 60000.
Upper limit for exclusion in IMSTATS during intensity offset operations.
You might consider choosing a lower value to limit the effects of
bright stars and improve image statistics by restricting the range.
setpix = yes
Since the over-lap region includes the edges of the frame, it is best
to mask the images when computing intensity offsets and combining the
images into a composite image. 'setpix = yes' will mask the images
using the mask 'maskimage' before calculating overlap statistics.
ZGET works on image copies. The original images remain untouched.
maskimage = "badmask"
Name of the channel-specific mask image. See APPENDIX A.7 for a
discussion of SQIID image masks. Remember to use the mask appropriate
to the current channel.
svalue = -10000000.
The value given to masked pixels. The large negative number shown is
assures that any output pixel which is generated through subsequent
image shift and geotransformed from masked input pixels stays
recognizably negative compared to good pixels so that such pixels
can be thresholded from the final image. When using such a 'svalue',
you must remember to exclude this value by choice of lowerlimit from
statistics and save such images as real images. To avoid loss of
precision, you should use 'imreplace' to replace these values with
safe values before you save them using 'wfits' with image scaling.
(Saving images via 'wfits' using 'bitpix = -32' option will save the
images in IEEE 32bit format without image scaling.)
size = no
Since one has presumably determined the final image size during the XY
registration process, 'size = no' is the preferred choice. If you
set 'size = yes', the task we redetermine the image size and edit
the prescription accordingly (loosing your registration with other
images). If the task cannot find the proper information it will
ask you for permission to do it:
Do you want to [re]compute image size? (yes)
Answer "yes" if you wish to proceed and "no" if you wish to exit.
save_images = no
Setting 'save_images = yes' preserves the temporary transformed,
shifted, and masked images generated by the task for examination of
pathological intensity offsets (you could use the 'imexamine' task).
Generally, you do not want to save the images and set
'save_images = no'.
verbose = yes
Set 'verbose = yes' if you want a more complete log of the ZGET
operations including in the output file.
format = yes
Setting 'format = yes' generates output COM-lines aligned into columns
so that they are easier to read. To accomplish this, the UNIX 'awk'
task is called. If this is not well supported on your system, set
'format = no'.
'interp_shift', 'bound_shift', and 'const_shift' are parameters employed by IMSHIFT and should be left at their indicated default values. 'floor', 'ceiling', and 'ovalue' are parameters which are currently unused by ZGET, but are used in the final combine operations in the task 'nircombine'.
If ZGET is crashing, saying things like "rmedian not found", or giving bizarre results, check the 'upperlim' and 'lowerlim' parameters to be sure you aren't "trowing the baby out with the bathwater". In particular, 'lowerlim' must not be INDEF (otherwise all those -1e8's generated by the SETPIX operation and choice of 'svalue' make it in and really mess things up!). The other possibility is that 'lowerlim' is set too high. Although you have subtracted background, H has a way of varying alot. Consequently the residual background can vary several hundred counts from that of the subtracted skyframe. If you have been saving the UPSQPROC output to file you can check there, or you can use IMEXAM on the USQMOS image to see what a better 'lowerlim' would be.
If you have residual problems intensity matching in the final image (due to ghosts on one image which are not on the other, the presence of bright stars, or whatever), try different paths to reach the offending images: link from above below or the other side. You just need to be sure that a linkage path to each image exists. Failing this, you can examine the combined image using the "l" and "c" line and column plotting option withim 'imexam' to find the offset error and to edit the ".zcom" files accordingly. (Note: since the intensity offsets within the COM-lines are not cumulative - they are only applied to the image on that line - the editing task is straight forward. See APPENDIX A.13 for a discussion of the registration database file format.)
VII. Creating the Final Images
Following the construction of the database files containing the offset and transformation information, the final images are created with the task 'nircombine'. NIRCOMBINE follows the submitted registration prescription and transforms (GEOTRANSFORM), spatially shifts (IMSHIFT), intensity offsets IMARITH), and masks (USQMASK) copies of the input database images and presents these copies to the IRAF 'imcombine' task along with the integer spatial offsets necessary for assembling the final image. See APPENDIX A.6 for a discussion of 'imcombine'.) In this task all of the transformations are actually applied. Also, the few bad pixels in the detectors can be corrected for either by exclusion from the dithered final image or by interpolation in a mosaic image. For example,
Whenever you run 'nircombine' it checks to see if there is a common overlap region for the dataset. If there is no overlap region, it issues a warning:
Selected NIRCOMBINE parameters are described below:
NIRCOMBINE:
* match_name = name of resultant composite image
* infofile = "" file produced by XYGET|XYLAP|XYADOPT
answer = yes Do you want to continue?
compute_size = yes Do you want to [re]compute image size?
* (outfile = "") Output information file name
* (trimlimits = "[2:2,2:2]") added trim limits for input subrasters
* (register = yes) Maintain input image origin and size
* (common = "median") Pre-combine common offset: |none|adopt|mean|med
* (comb_opt = "median") Type of combine operation: |average|median|
* (reject_opt = "none") Type of pixel rejection operation
* (lthreshold = -200.) Rejection floor for imcombine and stats
* (hthreshold = 65000.) Rejection ceiling for imcombine and stats
* (blank = -1000.) Value if there are no pixels
* (setpix = yes) Run SETPIX on data?
* (maskimage = "badmask") bad pixel image mask
* (svalue = -100000000.) Setpix value (<< lthreshold)
(fixpix = no) Run FIXPIX on data?
(fixfile = "badpix") badpix file in FIXPIX format
(to_scale = 0.) Scale to value (0 = noscale) via to_name
(to_name = "INT_S") Image header to_name keyword
(make_stack = no) Imstack images prior to imcombine?
(apply_zero = yes) Apply zeropoint to images prior to imcombine?
(invert_zero = no) Invert database zeropoints prior to application
(save_images = no) Save images which were IMCOMBINED?
(do_combine = yes) IMCOMBINE (else IRMATCH) into final image?
* (verbose = yes) Verbose output?
(size = no) Compute image size?
(interp_shift = "linear") IMSHIFT interpolant (nearest,linear,poly3,poly5
(bound_shift = "nearest") IMSHIFT boundary (constant,nearest,reflect,wrap
(const_shift = 0.) IMSHIFT Constant for boundary extension
(gxshift = 0.) global xshift for final image
(gyshift = 0.) global yshift for final image
(goverlap = "") Global overlap section (overrides all else)
(gsize = "") Global image size (overrides all else)
(weight = "none") Image weights
(mclip = no) Use median, not mean, in clip algorithms
(pclip = -0.5) pclip: Percentile clipping parameter
(nlow = 1) minmax: Number of low pixels to reject
(nhigh = 1) minmax: Number of high pixels to reject
(lsigma = 3.) Lower sigma clipping factor
(hsigma = 3.) Upper sigma clipping factor
(sigscale = 0.1) Tolerance for sigma clipping scaling correction
(grow = 0) Radius (pixels) for 1D neighbor rejection
(expname = "") Image header exposure time keyword
(rdnoise = "0.") ccdclip: CCD readout noise (electrons)
(gain = "1.") ccdclip: CCD gain (electrons/DN)
match_name
Name of the resulant composite image
infofile
'UPSQIID' registration database file corresponding to the image you
wish to assemble. THe database files output by XYLAP, XYGET, ZGET,
or MERGECOM and any of their transformations via XYADOPT are acceptable.
Remember that you can edit COM-line prescription contained in these
files when necessary to get your desired result.
outfile = ""
Outfile is the name of the database file generated. If 'outfile = ""'
or 'outfile = "default"', this file is named "match_name.src".
trimlimits = "[2:2,2:2]"
The number of columns or rows to trim off each edge of each
input subraster before inserting it in the output image in
section notation. The default is to trim 2 columns or lines at each
edge. This compensates for the 2 pixel border trashed (i.e. made up)
by the IMSHIFT operations. The input images are not touched; rather
the database is marked to exclude these edge regions from the final
image. However, the images the trimmed portions of the image will be
excluded during the image overlap statistics. Normally, 'trimlimits'
are cumulative in the database. This 'trimlimits' is not. It just
communicates to the task generating the overlap regions the message
that the edges are flawed so don't count them as useful during the
overlap computations.
register = yes
The new imcombine task trims the edges of the output image at the
top and right to the miminal size required. Consequently final image
size can very with channelx. Setting 'register = yes' restores images
to the full size requested in 'infofile'. Setting 'register = no'
leaves the output image of IMCOMBINE alone.
common = "median" (mean|median|mode)
If your dataset contains images which share a common overlap region,
you can use 'common' to select the statistic you want to use to
determine the z offsets between images. It can be either "mean",
"median", or "mode". If you wish to adopt previously determined
intensity offsets (e.g. from ZGET) from 'infofile', set
'common = "adopt". (feeds 'zero' parameter in IMCOMBINE)
comb_opt = "median" (average|median)
Type of combining aperation performed on the final set of pixels
(after offsetting, masking, thresholding, and rejection). The choices
are "average" and "median" (only in IRAF 2.10). THe median uses the
average of the two central values when the number of pixels is even.
reject_opt = "none" (none|minmax|ccdclip|crreject|sigclip|avsigclip|pclip)
Type of rejection operation performed on the pixels remaining
after offsetting, masking and thresholding. The rejection choices
are:
none - No rejection
minmax - Reject the nlow and nhigh pixels
ccdclip - Reject pixels using CCD noise parameters
crreject - Reject only positive pixels using CCD noise parameters
sigclip - Reject pixels using a sigma clipping algorithm
avsigclip - Reject pixels using an averaged sigma clipping algorithm
pclip - Reject pixels using sigma based on percentiles
If earlier than IRAF2.10, the reject_option should be set to its default
("none"). "minmax" is a conservative choice. See APPENDIX A.6 and
the help page for IMCOMBINE for discussion of the alternatives.
Depending on the choice of 'reject_opt, the parameters from 'weight'
through 'gain' listed above may be better served by choices other than
those shown.
lthreshold = -1000.
Lower limit for inclusion in IMSTATS during intensity offset operations
and for inclusion in the final image in IMCOMBINE during threshold
operations. When using 'setpix = yes' you must be sure to exclude the
mask regions, which will be set to 'svalue' from the statistics. You
might consider choosing a higher value to improve image statistics by
restricting the range. However, be sure to keep this limit safely
below the loweset real data in your image! IMCOMBINE thresholds its
input pixels prior to any scaling, rejection, and combining; if INDEF
the thresholds are not used.
hthreshold = 60000.
Upper limit for inclusion in IMSTATS during intensity offset operations
and for inclusion in the final image in IMCOMBINE during threshold
operations. You might consider choosing a lower value to limit the
effects of bright stars and improve image statistics by restricting the
range. IMCOMBINE thresholds its input pixels prior to any scaling,
rejection, and combining; if INDEF the thresholds are not used.
blank = -1000.
'blank' is the value given output pixel locations where no input pixels have survived the threshold and rejection algorithms.
setpix = yes
Since the over-lap region includes the edges of the frame, it is best
to mask the images when computing intensity offsets and combining the
images into a composite image. 'setpix = yes' will mask the images
using the mask 'maskimage' before calculating overlap statistics.
NIRCOMBINE works on image copies. The original images remain untouched.
maskimage = "badmask"
Name of the channel-specific mask image. See APPENDIX A.7 for a
discussion of SQIID image masks. Remember to use the mask appropriate
to the current channel.
svalue = -100000000.
The value given to masked pixels. The large negative number shown is
assures that any output pixel which is generated through subsequent
image shift and geotransformed from masked input pixels stays
recognizably negative compared to good pixels so that such pixels
can be thresholded from the final image. When using such a 'svalue',
you must remember to exclude this value by choice of lowerlimit from
statistics.
verbose = yes
Set 'verbose = yes' if you want a more complete log of the NIRCOMBINE
operations including in the output file.
to_scale = 0.
Scale to value (0 = noscale) via to_name. This allows one to scale
images to a common integration time prior to IMCOMBINE. For example,
if to_scale = 10, based on the image header to_name keyword (default
is "INT_S", the WILDFIRE image header keyword for integration time in
seconds), data with INT_S = 10 are untouched and data with INT_S = 8 are
multipled by 10/8=1.25 after masking and spatial shifting but prior to
submission to IMCOMBINE.
to_name = "INT_S"
Image header to_name keyword
make_stack = no
Imstack images prior to imcombine? Explicitly incorporates creation
of an image stack (via IMSTACK) for submission to the IMCOMBINE task
to get around 102 image limit. IMCOMBINE implicitly does this
internally in IRAF 2.11EXPORT. This allows one to stave this image
stack for later review. THe RECOMBINE task can be used to interactively
adjust the relative intensity offsets wihin this stack and recombine
to see the result. This is more useful in tiled images than in fully
overlapped images.
apply_zero = yes
Apply zeropoint to images prior to imcombine? The apply_zero flag to
allows choice of:
(1) applying zero-point shifts via imarith prior to imcombine,
so that zeropoint adjusted results will be saved via
make_stack+ or save_images+.
(2) submitting intensity offsets file directly to imcombine,
so that zeropoint unadjusted results will be saved via
make_stack+ or save_images+.
invert_zero = no
Invert database zeropoints prior to application. IRAF V2.10EXPORT
April 1992 imcombine has an inconsistency in the sign convention on
applied zeropoint. The sign of the file based zeropoint assumes the
value to be subtracted, whereas the reported number is the value to be
added. These have been reconciled to "value to be added" in the
IRAF V2.11EXPORT release. zero_invert flag toggles these situations.
save_images = no
Save images which were IMCOMBINED? Useful fto see what the shifted
and masked images look like.
do_combine = yes
By default, 'do_combine = yes' and the images are combined into a
single composite image using all the available pixels. If you
would rather emulate the tiling operations of the IRALIGN/IRMATCH1(2)D
tasks and have each succesive image cover up the overlap regions with
prior images in the prescription, set 'do_combine = no'.
'interp_shift', 'bound_shift', and 'const_shift', are used internally by IMSHIFT and should remain at their indicated default values.
The query parameter "computer_size" should be set to "no". The trimlimits should be [2:2,2:2] to exclude the edges which will have been affected by the transformation.
Except at the edges (which were excluded earlier), the SQIID detectors have only a few bad pixels and fewer clusters of bad pixels. For dithered datasets, these pixels should be excluded from the assembly of the final images by setting the parameter "setpix" to "yes" and providing the name of the detector's the bad pixel mask (see the Appendix) in the parameter "maskimage". Masked pixels will be set to "svalue" if they are not overlapped by a good pixel. For mosaics and single images, the "setpix" option may also be used. However, the parameter "fixpix" may be set to "yes" instead ("setpix" must be "no") to interpolate over the bad pixels. These used the IRAF "fixpix" algorithm and requires a fixpix format filename in the parameter "fixfile".
When using SETPIX, the image mask must either be in the current directory or a valid IRAF pathname to it must be included in the 'maskimage' parameter name, e.g.:
mask if in current directory ../mask if one directory above data$mask if the pathname data has been defined and set in you IRAF loginIf GEOTRAN is requested ('do_tran = yes' in the 'infofile' database), both the ".geodb" and ".geoco" files, which are embedded in the 'infofile' database, must be in the current directory.
The parameters 'lthreshold' and 'hthreshold' should be set to include the range of all good pixels in all images. For example, the range -100 to 20000 would certainly be safe for images which have been optimally sky-subtracted. Setting these top closer tolerences may improve the precision of the common median determination. The parameters "lthreshold" and "hthreshold" also limit the range of pixels values which will be included in assembly of the final image. The limits should certainly be set to exclude pixels flagged as bad (i.e., 'svalue' after masking) and pixels with unreasonable values.
Separately assemble the individual processed frames of the images you wish to combine into a mosaics (one per channel) using USQMOS. When assembling the database for the other channels, it is important to realize that the data at each channel must be tiled into its USQMOS mosaic at the same locations. Subsequent 'SQIID' registration tasks assume that the mosaics are equivalent accept for channel. Use of the "@list:channel_id" convention assures that such will be the case. Defer image masking and image transformation until the final stage within NIRCOMBINE.
IF you have dithered frames etc, in addition to an NxM set of overlapped images, you can put them all into a single mosaic if you put the NxM image into the lower left corner of the mosaic. Then, when you use XYLAP,
set nx_sub = N
set ny_sub = M
set nxr_sub and nyr_sub to the position (within the confines of NxM but
with the proper co-ordinates from the larger array)
Potential grief is minimized if you keep the X dimension of the full array at N.
XYGET/XYADOPT and MERGECOM allow you to operate in larger domains, which are not necessarily organized into a simple grid. If you assemble the data in smaller mosaics, you can pull them together using XYGET to link the individual mosaics and MERGECOM to put it together into a single prescription. XYGET accepts pieces from different mosaics. The format is as follows:
"Range in included image "COM# | reference COM# | mosaic name of reference"
the final "|" and field is left out if they are from the same mosaic
if the reference USQMOS database is mos_name.dbmos,
set ref_info = "default"
else
set ref_info = name of USQMOS database for reference image
To assemble the datasets for other SQIID channels, assuming you have assembled the data at all wavelengths in the same XY order, you can use XYADOPT to create the prescription for the other colors, once you have a successful prescription for one color. Set the parameters for XYADOPT as follows:
set images to the basename of the new color
set frame_nums =
"Range in COM# of included images | #_of reference image in original color"
set mos_info = "default" or "name_of_newcolor_usqmos_database"
set basis_info = the ".xycom" file for your complete original color success
set trimlimits to value approriate to new color
set tran = yes
set db_tran = name of appropriate ".geodb" file for transforming new_color to
original_color. Both the ".geodb" and ".geoco" files must be in the
current directory where you are assembling everything
answer "no" whenever you are prompted by XYADOPT/ZGET/NIRCOMBINE "Do you want
to resize image?". You want to stick with the origin determined by you
first successful channel image.
Where images do not share a proper common overlap region, use the ZGET task to determine the intensity offsets. You must specify the linkage between overlapping images you want to use to accomplish this registration. The task uses the registration database to determine the overlap regions for each image pair. Copies of the images are geotransformed, fractionally shifted, and masked. The statistics (selected by 'common') within the overlap regions of each image pair are determined and the prescription is them followed to accumulate the net offset of each image compared to the first image in the prescription. Currently, the entire overlap region is included in the statistics. Set the parameters for ZGET as follows:
set match_name to the name of the image you wish to create for the channel
set info_file to the name of the complete ".xycom" file for the channel
set image_nums to the name of the file containing the linkage paths you
want to follow in assembling the image
set setpix = yes
set maskimage to the bad pixel mask for the channel
set common = "median"
set lowerlim to a value safely below the lowest data in your images (but not
INDEF!)
answer "no" whenever you are prompted by XYADOPT/ZGET/NIRCOMBINE "Do you want
to resize image?". You want to stick with the origin determined by you
first successful channel image.
Use NIRCOMBINE to assemble the final image using the XY(Z) regustration database table generated above.
set match_name to the name of the image you wish to create for the channel
set info_file to the name of the complete ".zcom" file for the channel
If your images share a proper common overlap, use the ".xycom" file and
set common = "median"
set trimlimits = "[2:2,2:2]" to trim off the "rough edges" generated by
IMSHIFT
set register = yes to preserve image size for all channels
set common = "median" if you have a common overlap region
set common = "adopt" if you want to use the ".zcom" info from ZGET
set setpix = yes
set maskimage to the bad pixel mask for the channel
set common = "median"
set svalue = -1e8 to make sure bad pixels and there progeny are thresholded
set lthreshold to a value safely below the lowest data in your images (but not
INDEF!)
set blank to a number recognizably below your lowest data
answer "no" whenever you are prompted by XYADOPT/ZGET/NIRCOMBINE "Do you want
to resize image?". You want to stick with the origin determined by you
first successful channel image.
Assembling grids beyond order 6x6 can be a daunting task. The larger the grid, the more likely there will be weak or missing links in the registration process. At some point, one needs to worry about cumulative effects. Furthermore, XYLAP, which uses the output from the CENTER task is set up to operate on mosaics no larger than 10 in X or Y. (IRAF scripts do not handle variable arrays well and larger arrays eat up too much space on the stack.) The techniques summaried above are well suited for minimally-overlapped spatial grids up to 6x6 in size. What is the best way to register larger images? The answer begins with it depends ...!
You have a number of choices, since the database generated by XYLAP/XYGET does not really care where the individual frames are. Here are some recipes for linking datasets composed of dithered pairs of a 3x3 grid (arranged as separate 3x3 grids arranged vertically into a 3x6 mosaic), which is extensible to larger situations:
1) The plan ahead option (probably too late for this!). Often, I will include a key piece of an adjacent frame inside my 3x6 (make a 3x7 with the additional pieces in the top row - remember that some of the entries to USQMOS/IRMOSAIC can be "null". Then I can use XYGET followed by MERGECOM to handle the overlapped piece and bring them all to the same universe. You can still do this by creating a new mosaic with the same number of frames in the X direction as your original. Then you can reuse your CENTER about and run XYLAP again with nx_sub and ny_sub set to 3. Then you run XYGET on the additonal piece to get the link to the next 3x3 and use MERGECOM to put it all together.
2) Use the option on XYGET (created for folk, like myself, who don't always plan ahead!) which allows the matched frames to be in separate USQMOS mosaics. Match frame pairs for each USQMOS that way and use MERGECOM to put it together.
3) Make a USQMOS of NIRCOMBINEd frames. Here you need to get crafty, since USQMOS/IRMOSAIC only accepts pieces of the same size. You can use MKPATTERN in the 'noao.artdata' package to make an image big enough to fit any of the pieces and then imcopy each NIRCOMBINE frame to the larger template. You should use IMREPLACE to set all the bad and border pixels to that large negative number the 'setpix' option uses (e.g. -1e8; they have been set to 'blank' by NIRCOMBINE.). Then use USQMOS to make up new mosaics for each channel (each ordered in the same way) and CENTER/XYLAP or XYGET on your favorite channel to link them. Use XYADOPT to set up the other channels. Don't use the 'setpix' option (you took care of that when you did NIRCOMBINE; now you don't have a proper mask image) or the 'tran' options (you've already done that during NIRCOMBINE) along the way in XYGET, XYLAP, XYADOPT, ZGET or NIRCOMBINE.
Which way to go is a toss up. Option 1&2 rely on a single overlapping frame for the link, while option 3 let's you use several. However, if the 3x3's aren't put together well (because of weak or missing links), option 3 might lead to contradictions at the edges.
Remember to feed the information to MERGECOM in the proper order when you organize the list of filenames:
As usual "lpar taskname" lists and "epar taskname" edits the parameter file for "taskname". Lines which are shown inside parenthesis are hidden parameters, and are not prompted when you run the task. You can set them using epar or include at the end of tour command line (following any prompted parameters you wish to include). Within the listings shown here some parameters (mode and assorted "list" parameters - parameters which appear on the left of the file listing without prompts on the right) are not shown, since they internal task parameters which cannot be changed by you and only appear because the CL language requires them to appear external to the task.
(The "list" type variables which appear in an IRAF script file refer to pointer structures which are used internally to an IRAF script procedure, but because of the nature of the beast must appear externally. They show the temporary values that they had the previous invocation and will update.)
Within this writeup, we have marked parameters which are of special interest to the user with an asterisk "*" in the left-hand column.
A lot of the tasks found in this package create intermediate images/files (rescaled images, dark substracted images ...). When everything is all right, these images/files are deleted before the task is exited. Thus you are never really aware that they were ever created. If for some reason a task crashes on you, then you may be left with a few of these 'temporary' images/files in your directory. They are the images/files with names starting with _T. Typing "cleanup" after a task crashes, deletes these images/files (if any). Additional temporary files are generated within the IRAF named directory "tmp" and can be accessed via the "tmp$" alias (e.g. cd tmp$; dir tmp$; head tmp$*).
The SQIID system generates FITS images with the following naming convention:
"base_name"//XXX//"channel_id"//.fits (e.g., n1_081j.fits)
where "base_name" ("n1_") was supplied by you at the time of observation,
"channel_id" is j,h,k or l, and the number "XXX" is sequentially numbered
(it was automatically incremented for each exposure until a new "base_name"
was declared, typically at each new object).
To facilitate the handling of the large numbers of image sets typically obtained during a SQIID observing run, the UPSQIID software is designed to operate on IRAF "@-list" files (inside IRAF, one instructs the system to read data from the file "list" by prepending "@" to it - "@list" tells the task to read list to get the image names.) The task "chlist" is provided to create list files. (Alternatively the task "nlocal.ranges" can be used.) To use "chlist" the image names must end in an integer. The first useful tool is called CHLIST and its parameter list is shown below. It is used to create list of images of a given color. First_image is the name of the first image name to be in the list. Number is the total number of images in the list. Delta is the increment size. This number is added to the previous image name in the list to create the next image name of the list.
CHLIST: first_image = First image in sequentially numbered images number = Number of images in output list (delta = 1) (id_color = "none") Location of color indicator? (end|none|predot|s As indicated, you can include directory pathnames. Set delta = 1 for sequential image names. If you were to give IRAF the following command (with delta = 4): chlist raw001 2 > jlist then jlist would read raw001 raw005. If you were to give IRAF the following command: chlist raw003 5 > klist then klist would read raw003 raw007 raw011 raw015 raw019. If you were to give IRAF the following command (with delta = 1): chlist raw/n1_001j.1 3 > jlist then jlist would read raw/n1_001j.1 raw/n1_001j.2 raw/n1_001j.3.
CHLIST has no intrinsic output file; that is why you have to use redirection. If you don't the output is printed on the terminal screen (STDOUT) but no saved. Note that CHLIST acts totally independently of what images you have in your directory. It does not check whether the images exist or not; they just create a list of names.
The input image string parameter for UPSQIID tasks can be the usual "comma delimited command line lists" or "@-file" type lists. One can also use wildcards within image pathnames to operate on all images which match the template (the IRAF "section" command is invoked).
For example, assuming the following images exist:
raw/n1001 raw/n1002 raw/n1003
and that jlist contains:
raw/n1001
raw/n1002
raw/n1003
Then, "raw/n1001,raw/n1002,raw/n1003"
"@jlist"
"raw/n100?.*.imh"
are equivalent j-image inputs.
Within the XYGET/XYADOPT tasks, the input to images should be the name of the USQMOS mosaic rather than the list. (Although an alternative path exists for submiting image lists rather than the name of the mosaic, they are not fully debugged. If you submit the list, you need to give the full name of the USQMOS mosaic database rather than "default", because default says "append .dbmos to the image name".
The input and out image string parameters in UPSQPROC can be the usual "comma delimited commanfd line lists" or "@-file" type lists. In addition, the IRAF substitution operator "% % %" can be used (in the same way as imrename works) to generate a new image name based on the old name with string substitution. If the output string begins with a period, the string is appended to the input names. To operate on only one image named in input and output the output name needs a prepended "," to get things to work. Consider the following examples:
Let the file inlist contain im1 and the file outlist contain out1
im2 out2
im3 out3
Then the given command lines produce the corresponding outputs as shown:
COMMAND LINE RESULTANT ASSIGNMENTS
"upsqproc @inlist @outlist" im1 -> out1
im2 -> out2
im3 -> out3
"upsqproc @inlist %im%out%" im1 -> out1
im2 -> out2
im3 -> out3
"upsqproc im1,im2,im3 out1,out2,out3" im1 -> out1
im2 -> out2
im3 -> out3
"upsqproc @inlist .ds" im1 -> im1.ds
im2 -> im2.ds
im3 -> im3.ds
"upsqproc im1 ,out1" im1 -> out1
Note: directory information is skipped over during the "% % %" substitution so that only that part of the pathname after the final "/" survives the substitution process.
Many of the UPSQIID image tasks (usqdark, usqsky, usqflat, nircombine) call the IRAF task "images.imcombine", which contains many parameters (most of which are only useful for the very discerning and sophisticated!). In order to make these parameters available to the UPSQIID package user, I have had to replicate them in the appropriate UPSQIID tasks. See the manual page for "imcombine" for more complete description of the parater choices