DPP | Software | Library | Surveys | NSA | Education | LSST | NVO

Data Products and Software at NOAO—Some Changes Coming

Over the past several years, NOAO’s effort in the areas of software and data management have expanded to encompass archives, automatic data reduction pipelines, and web-based services for data analysis and visualization. This complements and builds on our previous strengths in exportable user data reduction software epitomized by the widely-used IRAF system. The first release of our end-to-end data flow system, which will enable Virtual Observatory research, is planned for early next year. As we gear up to provide new ways for the community to access NOAO data products and services we are also announcing changes in how IRAF software and services will be provided.

The NOAO Data Products Program (DPP) data flow system combines new data storage, data reduction pipelines, portals to integrate data access and tools, and a transport system to link these together. This system is NOAO’s first step towards establishing a data center that is relevant in the Virtual Observatory era. This effort has been the major undertaking of the DPP group for the past few years. The committees that provide oversight or advice on the NOAO program have endorsed this vision of the future of NOAO’s data-related services. Its value lies not only in NOAO’s participation in creating an integral piece of the Virtual Observatory, but also in simply allowing the multiple uses of NOAO data.

The data transport system, which has been running in a prototype version for almost a year, replaces our tape-based Save-the-Bits system, capturing the data streams from the NOAO telescopes and feeding them via network into the archive. The NOAO Science Archive (NSA) itself, which is being developed and will be operated in partnership with the National Center for Supercomputing Applications, will handle the physical storage of this raw data, its associated metadata, and ultimately, pipeline-reduced data. Data from the NOAO telescopes will be available to everyone after the proprietary period, nominally 18 months, but shorter in some cases. Metadata will be viewable and searchable immediately and is an important feature of our accountability to users and the community.

Pipelines to automatically reduce the data from the CCD Mosaic Imagers and the new wide-field IR imager, NEWFIRM, will also begin operation at about this time. These pipelines will characterize the data and document its quality, giving researchers some knowledge of the usefulness of the available data prior to retrieving it. The reduced images, and ultimately the catalogs produced from them, will become available to all.

The NOAO Data Portal is an integrated interface to a combination of data, tools, and services. Through it, users will be able to discover and retrieve data from the NSA as well as external archives. The portal also provides users with a set of tools to visualize and analyze data and metadata, as well as enabling users to query the archive data holdings. Tools to help visualize time-domain information will be an early emphasis of portal development. The portal will also provide access to some IRAF processing tools as web services. Future releases will greatly enhance the capabilities of the portal, including the ability to combine image and catalog information.

As we go into the operations phase for these new capabilities we are also making changes in the way we operate IRAF to make it more efficient and self-supporting. By “IRAF operations” we mean keeping up with changes in systems that host IRAF and in direct user support. We are moving towards an “open source” model (although IRAF sources have always been freely available), which relies more on community efforts than in the past. To do this we will establish a web site providing forums for people to ask AND ANSWER questions, and post announcements, as well as searchable access to the large archive of email from our help desk and an updated FAQ facility. We hope to now make it more attractive for users to contribute bug fixes and enhancements, and we will be relying on the participation of those users in maintaining support for the user community. In some ways there will only be small changes (the IRAF project has always maintained a web site and ftp tree), and in others big changes (by moving to a forum-based help desk). The core IRAF group at NOAO will coordinate and participate in this open source approach, answering expert level questions and contributing to the code base, as partners with the community.

Beyond operations there is the question of new developments in IRAF. A traditional mission of the IRAF project was to provide exportable data reduction tools for NOAO data. These were written so that they could also be used with data from other observatories. The high citation rate for IRAF and the wide use of this software in the community illustrates the great success of IRAF for providing general data reduction tools for astronomers, students, and amateurs on a variety of data. While this mission is being phased out, there will still be occasional developments when they are needed for other aspects of our program.

As the traditional mission is phased out we see a new mission to provide analysis and visualization through web services and the virtual observatory. Therefore, the development of traditional exportable data reduction packages at NOAO will be largely replaced by web and archive services, some of which will be based on IRAF.

Developments in the core IRAF development platform and system will continue at a low level with emphasis on the needs of the pipeline, archive, and virtual observatory projects. We will continue to issue updates as they are needed for new versions of supported operating systems and compilers. There may also be specific collaborative projects within and outside of AURA. More community driven developments will be addressed through the open source model where the community participation is essential.

We hope that no one stops using IRAF because of the changes to our operations model. In fact, we hope that the community will embrace the open source model and continue to communicate and contribute. We even expect that new IRAF exportable packages, such as those for Gemini instruments and interfaces such as a Python (PYRAF) and an error-handling CL (ECL), will continue to be developed as well as new tools to interact with IRAF through web services. We are interested in keeping the lines of communication open in this period of transition and will provide forums at science and technical meetings such as the AAS. We, in the Data Products Program, are excited about the new opportunities for scientific discovery that the Virtual Observatory will create, and we anticipate working with the community and other VO developers to ensure that the users of NOAO facilities benefit from its development.