The NEWFIRM Quick Reduce Pipeline and Data Analysis Tools

Last update: 13 October 2010

These notes provide a quick guide to the NEWFIRM Quick Reduce Pipeline and other data reduction software and tools for use at the telescope. NEWFIRM is currently operating at the Blanco 4m telescope at Cerro Tololo, and these notes have been updated for operations at CTIO.

Table of contents:


Quick reference information:


The NEWFIRM Quick Reduce Pipeline (QRP) - an introduction

The NEWFIRM Quick Reduce Pipeline (QRP) operates on computers at the CTIO 4m telescope, and provides basic, automated data processing for NEWFIRM during the course of an observing run. It is designed to provide basic reduced image products and data quality feedback to the observer on a reasonably short timescale as the observations are being taken. The QRP is not intended to produce science-quality reductions. It takes several shortcuts to speed data processing, such as using only single-pass sky subtraction. Moreover, the best possible calibration data may not be available at the time a given observation is taken. For example, frequently an observer will not take darks with exposure times appropriate for their science observations until after the science data are taken. In this case, the pipeline will fall back on earlier darks from a library, using something with the closest available exposure time.

The QRP is triggered at the end of each NEWFIRM exposure. However, in general NEWFIRM observations are taken in sequences of multiple exposures that are defined with the NEWFIRM Observing Control System, or NOCS. Examples include sequences of darks or flats, or dithered sequences of scientific observations. The QRP will not process observations taken with 'TEST' sequences (as created by the NEWFIRM script-generating GUI), or focus sequences (see the discussion of the nffocus routine below for information on analyzing focus data).

Much of the QRP processing does not begin does not begin until the last exposure in an observing sequence is completed. For night-time science observations, the pipeline applies dark and flat calibrations, sky subtraction (usually but not always based on a running median), determines a WCS and photometric calibration relative to 2MASS stars in the field, resamples images to a common pixel grid, and combines them into a stacked mosaic. The QRP will also process and combine darks and dome flat sequences.

Processing of the data starts immediately upon completion of a sequence. For many observing programs, the typical cadence of exposures during night-time science observations will be of order one image per minute. At this data rate, the pipeline will typically take somewhat longer to process the data than the time needed to complete the observing sequence itself. However, if you are observing with a much quicker cadence (e.g., a long series of 15 second exposures), the pipeline may not be able to keep up.

The products of the Quick Reduce Pipeline are not archived or saved in any other way. If you want to keep the QRP-produced FITS products for your own use, this is your responsibility. The pipeline review web pages that you can examine during your run will also be removed at the start of the next observer's run. Note also that these review pages are also behind the mountain firewall, and are thus not accessible to people outside the mountain internet domain.

A more detailed description of the QRP and list of some known caveats is given below.


How to observe in a "pipeline friendly" manner

There are a few important steps that you can take to ensure that your observations are done in a way that ensures good performance from the Quick Reduce Pipeline.

Take at least some calibrations before you start to observe: The QRP operates in pseudo-real time, processing your observations after the completion of each observing sequence. Therefore, it needs to have appropriate calibration images - specifically, darks and dome flats - available at the time the observations are taken. The QRP maintains a calibration library, and will look back in that library for the most appropriate calibrations taken as recently as possible. However, if they are not available (e.g., flats for the particular filter you are using), then it will simply skip this processing step. Or, it may use rather old and inappropriate calibration files (e.g., darks with the wrong exposure times) as its "best guess". The best thing you can do is to take some calibrations before you observe, if at all possible. Note that on the second (and subsequent) nights of your run, the QRP can fall back on calibrations from the first night.

Note that the NEWFIRM science pipeline, which will process your data after your observing run is over, is not restricted to using calibrations taken before your science observations. It will comfortably use darks and flats taken after your night of observing, or on the next day of your run, etc. However, it is always good infrared observing practice to take flats and darks as close in time as possible to when you make your science observations.

Dither, don't linger: The NOCS observing sequences that you create with the NGUI script generator will in principle allow you "linger" at a given position in an observing sequence, taking more than one exposure per dither position, by setting "NumObs" to a value greater than 1. At present, however, the QRP does not process such data correctly, and in particular the sky subtraction will be badly corrupted. We will change this in the future, but in the meanwhile (and, in fact, in general) we would recommend that instead you use Coadds (set "coadds" to >1) instead of "NumObs". This will take several exposures and coadd them on board the instrument before writing to disk. This will also improve your observing efficiency - what's not to like?

Keep dithering: The pipeline uses a running median sky subtraction procedure as long as a minimum of 4 dithered exposures are taken in a sequence. For shorter sequences, it will use simple pairwise image subtraction, and you will see pronounced negative residuals in your images. From the point of view of sky subtraction, longer sequences with more dither positions are better - more than 9 if possible, but the longer, the better. If your science requires shorter dither sequences, then by all means go ahead and use them. However, if you plan to take many dithered exposures on a given field, it is better to do so in longer observing sequences rather than to break them up into many short sequences of a few frames each.

Offset sky fields: If you are observing a large, extended object that fills a significant fraction of the NEWFIRM field of view, or perhaps if you are observing a very crowded field, then you may wish to "chop" (perhaps frequently, perhaps occasionally) to a relatively blank field that is offset a significant distance from your target. The pipeline can process such data, constructing its background images using only the frames taken pointing at the offset field. However, for this to work, the pipeline needs to know which frames are the "object" exposures and which ones point at the offset "sky" field. This is only possible if you observe using the appropriate type of sequence generated with the NGUI tool. The offsets to blank sky are done using the RA and Dec Offset distances in the Telescope Configuration section of the NGUI. There are a variety of script options that will observe in this manner, including DeepRich, ModMapRich, and QuickMapWithSky. You should consult with Ron Probst or someone else familiar with the instrument to discuss which is best for you. Any of these sequences, however, will record metadata in appropriate header keywords which will tell the pipeline which frames are "object" and which are "sky". Note that it is possible, but not advisable, to create your own customized offsetting procedure using Map offsets in other NOCS/NGUI scripts (e.g., DeepSparse or QuickMap), rather than the Telescope Configuration offsets. However, the pipeline will not correctly process these data.

One special case is the "4-shooter" or "4Q" observing mode, which can be invoked by selecting the "4Q" dither or map option within most scripts. This is designed for observations of an extended object which fills a good portion of one NEWFIRM quadrant but not the whole 4-detector NEWFIRM field of view. 4Q mode places the target near the center of one quadrant, and then cycles the target through each of the four quadrants as many times as you wish, optionally applying other (usually smaller) dither offsets so that the target positions do not exactly repeat from cycle to cycle. 4Q-mode correctly records in the image headers the information about which detector includes the target, and the NEWFIRM pipeline will correctly process these data, constructing sky frames for each detectors using only those exposures which do not include the target. This has proven to be a very effective and efficient way to observe extended objects that are still smaller than 14 arcmin or so in diameter (i.e., the field of view of a single NEWFIRM detector).

Test exposures: If you want to take a short test exposure for some reason (e.g. to check instrment aliveness, telescope pointing on a bright star, check the sky brightness, etc. ), use a Test sequence from the NGUI script generator. Observations taken with the Test sequence will not get processed -- the QRP will not do a good job processing single test exposures anyway.


Observer setup information

NEWFIRM observers generally use the screens for the computer ctiozm to operate the instrument and examine incoming data. These screens get very crowded, so we recommend using another computer (or at least a different desktop on ctiozm) for examining pipeline output. There is another computer in the 4m control room, ctioa8, which would be suitable, or you could work from a laptop.

On the ctiozm console (data-taking computer):

On another computer (e.g., ctioa8, or from a laptop):


Checking the Pipeline status

You can use the command plstatus To check the operational status of the NEWFIRM Quick Reduce Pipeline from an xterm window logged in as observer@newfirm. If the pipeline is functioning properly, you should expect to see this:

[observer@newfirm ~]$ plstatus

Pipeline node status report: Sat Mar 15 15:24:44 2008
Current node is nfpipe-01

Node        Connected   NodeMgr     Available Pipelines           
------------------------------------------------------------------
nfpipe-01   Yes         Running     NEWFIRM:
                                       wcs, mtd, gcl, flt, drk,
                                       fil, dts, ndp, gos, sfl,
                                       mkd, swc, sky, sgc, stk,
                                       sdk
------------------------------------------------------------------
nfpipe-02   Yes         Running     NEWFIRM:
                                       wcs, drk, flt, fil, gos,
                                       sfl, sky, sgc, sdk, swc
------------------------------------------------------------------

If, instead, the machines are up but the pipeline is not running, you should see this:

[observer@newfirm ~]$ plstatus

Pipeline node status report: Sat Mar 15 15:24:44 2008
Current node is nfpipe-01

Node        Connected   NodeMgr     Available Pipelines           
------------------------------------------------------------------
nfpipe-01   Yes         No
------------------------------------------------------------------
nfpipe-02   Yes         No
------------------------------------------------------------------

If "Connected" reads "No" then the pipeline machine itself is not up.


Examining the pipeline data products

Before observing, you should use the DHS GUI you should specify the path for a directory where you would like the QRP FITS data products to be delivered (see setup instructions above). We recommend that you choose a directory other than that where your individual raw images will be written. For example, using a "QRP" subdirectory would be appropriate.

You can inspect the results of the QRP processing in two ways: via the data product review web pages that are generated by the pipeline, or by direct examination of the FITS data products with your favorite image processing tools (e.g. ximtool, ds9, IRAF, IDL, etc.).

Pipeline review web pages:
The QRP produces web pages that provide easy access to pictures of your reduced data, as well as various diagnostic information generated by the pipeline. You can find these QRP review web pages at the following URLs:

Note that you will need to reload/refresh this web page in your browser periodically, as it will be updated as new data are processed and added to the QRP review log.

You will see a log of data that the QRP has processed for your run so far, with various information about each observing sequence, including:

In addition, this log reports the ranges spanned by some data quality parameters that are measured by the pipeline, such as:

Clicking on the image filenames for a given sequence takes you to a pipeline review page with information about data from that sequence. There are review pages for processed, combined dark frames and dome flats, as well as for night-time on-sky observations. For night-time science observations, the QRP review pages provide:

For dithered on-sky observing sequences, preview images are shown for each individual exposure, as well as for a stacked mosaics of the observations. The stacked images are shown in the second column of thumbnail images. If, during the course of an observing sequence, the telescope motions exceed a particular hardwired tolerance (currently 8 arcmin, but this may change), the QRP will break the sequence into multiple, independent stacks that will all appear in the 2nd column of thumbnail images.

FITS data products:
When processing is complete for a sequence, FITS files of the combined image (the "mosaics" or "stacks") will be delivered to the directory that you specified (see above). The names of the files start with NFQR (for NEWFIRM Quick Reduce). FITS files are delivered only for on-sky observations, not for dome flats or dark calibrations.


DQQUERY - monitor seeing, transparency and sky brightness

dqquery is a command that displays measurements of the seeing, transparency, and sky brightness made by the NEWFIRM Quick Reduce Pipeline. The pipeline creates a catalog of sources from which the average sky brightness (ADU) and seeing (arcsec) are determined. The catalog is matched against 2MASS sources and a magnitude zero point is estimated. Naturally, if the exposure contains few or no stars then the seeing and zero points may be indeterminate or poorly determined. The matching to the 2MASS catalog may also fail for similar reasons or due to large errors in the telescope coordinates.

The measurements are recorded in a pipeline database which is queried by dqquery. The results are written when the command is first executed and then every 5 minutes thereafter until the command is terminated. A control-C termination will print a couple of lines of errors which can be ignored. The fact that a database query is made means that it is possible for the database to be unavailable during periods when the pipeline is busy recording data. So just wait or try again.

To use dqquery, open an xterm window on the newfirm computer (e.g., via ssh observer@newfirm-ct). There, execute the command dqquery. The output is illustrated below, along with an example of the program output. The listing gives a number of recent measurements with most recent first. Depending on the filters used there may be information from different filters. There are a few user selectable parameters, particularly the number of recent measurements desired.

DQQUERY example:

[observer@newfirm ~]$ dqquery -h
Usage: /shared/pipeline/V1.0/MarioSrc/bin/dqquery [OPTIONS]
Display the seeing, photometric zeropoint, and background levels
for recent exposures.
    -a, --average 

One useful application of dqquery is to dump a log of data quality measurements for all observations from a long stretch of time (e.g., for the last night). You can do something like this:

[observer@newfirm ~]$ dqquery -n -r 500 > dqquery_log.txt

The -n option turns off the automatic looping display, while -r 500 dumps the data quality information for the last 500 images that the pipeline has processed. At present, there is no simple way to request DQ information just from a particular night or range of times, but it is easy to just dump measurements from a large number of files and then edit the output to restrict to the desired image names or data/time range.


NFFOCUS - NEWFIRM focus evaluation IRAF task

From within an IRAF session the task nffocus may be used to analyze a focus sequence. It has been designed to make measurements and select the best set for estimating a best focus. There is a description of nffocus under the IRAF help system:

cl> phelp nffocus

This task has been designed to work conveniently with the standard NEWFIRM focus recipe. In this recipe a sky offset exposure is first taken followed by a sequence of exposures on the focus field where the focus is changed in uniform steps. After completion of the sequence the task only needs to identify the set of exposures. This can be done by specifying either a single image number or image name within the sequence or by a list of focus sequence images. In the latter case it is convenient to make a file containing the list and specify this to task with the @file syntax.

cl> nffocus 17142         # Image number
cl> nffocus obj17142      # Root name is chosen by the observer
cl> nffocus @focusj.list  # Prepared list of images

The task will initially make catalogs of the sources from the sky subtracted focus field images. This will take a few minutes. Repeating the task will simply reuse these catalogs so it is quick to return to the graphical analysis. In this graphical analysis you can delete sources, view spatial trends, and visually confirm the automatic best focus recommendation. There are many options but the initial FWHM verses focus plot is often all you need along with the 'd' key to delete points to clean up the outliers. The '?' will give you a quick summary of the available cursor keys and commands.


Quick Reduce Pipeline: details and caveats

The QRP is designed to quickly process data taken at the telescope to provide feedback to the observer about the progress of their observing program. It also reports various quality measurements to help the observer judge the observing conditions. The QRP intentionally takes some shortcuts in order to ensure faster processing and quick feedback to the observer.

The processing steps that the QRP applies to on-sky science observations are:

The QRP makes several shortcuts with respect to the full processing done by the NEWFIRM science pipeline. We comment on some of these here, and also on some general processing issues that apply to both the QRP and the science pipeline.