Previous Article Next Article Table of Contents


Global Oscillation Network Group (1Dec95) (from GONG, NOAO Newsletter No. 44, December 1995) The Global Oscillation Network Group (GONG) Project is a community- based activity to operate a six-site helioseismic observing network, to do the basic data reduction and provide the data and software tools to the community, and to coordinate analysis of the rich data set that is resulting. GONG data are available to any qualified investigator whose proposal has been accepted, however active membership in a GONG Scientific Team encourages early access to the data and the collaborative scientific analysis that the Teams have already initiated. The GONG Newsletter provides status reports on all aspects of the Project and related helioseismic science. Overview The GONG network is completely deployed and operational, and the DMAC has network data available!!! [Photo not included] The Big Bear, California site with the Big Bear Solar Observatory "floating" in the background. It took 16 months of hard work from the first groundbreaking, at Learmonth, to the last camera alignment, at Udaipur, to complete the deployment of the network; not to mention the decade of design, development, and production that preceded it. Thank you, dear reader, for your patient support while the Project has been working to achieve this, and thanks to the Project staff, all of our collaborators, and hosts around the world for all of their effort on behalf of GONG. The deployment sequence was chosen to achieve a three-site network, with roughly 120° separation at the earliest possible moment, to get data into the hands of the members to help understand the performance as rapidly as possible. The duty cycle for the first GONG month of three stations was a quite respectable 70%, and the data have been pushed all the way through the DMAC pipeline. All of the data products--including our first-cut frequencies--are now available through the DSDS. If you do not already have an account with the DSDS, the simplest way to set one up is through our WWW server (http://helios.tuc.noao.edu). The Science Teams are shifting into high gear, and they are scheduling working sessions in Tucson to work toward three near-term goals for the presentation of their work: 1) a summary presentation at the Winter meeting of the American Astronomical Society in mid-January, 2) a special issue of the journal Science to appear in late Spring, and 3) a joint meeting with the AAS and its Solar Physics Division in June 1996. Deployment Deployment is behind us at last. It took about 10 months from the very beginning to the very end when Frank Hill and the Red Team departed from New Delhi in early October. There were many anxious moments (many more of those than real problems), many successes, and many good times along the way. The short version of the story is this: all of the stations arrived where we intended them to go, none were damaged beyond repair, and they were all working when we left. Now, the rest of the story. The activity officially began in Tucson last December when the Teide station was lifted by crane onto the back of a flat bed truck, beginning its personal odyssey to the Canary Islands. The stories it could tell, if only it could talk! There was the day that one end of the external cable raceway and the doorknob were crushed beyond recognition at the hands of some nameless stevedore somewhere between Los Angeles, California and Santa Cruz de Tenerife. During that same period, salt water flooded over the station, corroding every lock and hinge that hadn't already been crushed, almost (if not completely) beyond use. This, however, was hardly sufficient to discourage either Frank Hill and the rest of GONG's Red Team, or the truly excellent IAC staff at El Teide. The cable raceway was rebuilt by the observatory shops, the doorknob was replaced, and judicious application of bolt cutters, oil, clean water, and paint rendered the shelter indistinguishable from the condition it was in when it left the Arizona desert. The instrument itself went back together just like the manuals said it would, and the first data were obtained on 17 February. The installation process took six weeks, which was exactly what had been scheduled. Special thanks go out to Pere Pallé, Jesús Patrón, Otilia de la Rosa, Antonio Pimienta, and many others at the IAC who helped us make our first deployment a success. Next stop: Learmonth, Western Australia. Ed Stover and Rob Hubbard, the vanguard of the Blue Team, left Tucson on 15 March, and caught up with the shelter in Learmonth on 20 March. To our surprise the shelter was in exactly the same condition as the El Teide station. The cable raceway was crushed to the same unrecognizable pulp in exactly the same place, as was the doorknob. Were we dealing with the same evil stevedore, singling out the GONG project for this special attention, or were we missing something fundamental? While in Australia, Duane Miller, the blue team's instrument maker, saw a brief image on local television that provided the answer. The equipment that is used world wide to move standard shipping containers around ports has a hydraulic ram that conflicts in several places with some of the modifications we had made to the container to make it into a GONG instrument shelter. Although we had asked for special handling, our "big white box" just looked too much like a standard shipping container, and was routinely being treated as such. When the balance of the Blue team arrived in Learmonth, Master Sergeant Coffman, a member of the United States Air Force contingent at the observatory, gave us our official "in brief." The message was simple: drive on the left, pedestrians don't have the right of way, and almost everything that crawls or swims is deadly. If we would just keep our hands in our pockets and not leave our rooms, we would be safe. Coming from rattlesnake country, we were not so easily frightened. We did adopt the temporary site password deathadder, however, just as a reminder of our mortality. The bite of this particular serpent renders its victim dead within about 30 minutes in most cases. In the final analysis, the only death adder we saw was a dead death adder. The people and the critters alike were wonderful to us, and we had the second GONG network site up and running when the last of the team returned home on 28 April. Here the acknowledgments have to include John Kennewell, Alex Liu, Major Jeff Carson, Jenny House, and the rest of the Australian and USAF observatory staff. No rest for the wicked, however. After three weeks at home to get to know our families again, the Blue team was back in the air, preparing to install the Mauna Loa station. By now we had learned what pieces of the station should be removed prior to shipping, and the Mauna Loa station arrived in Hilo in far better condition than the first two. Most of the team left together this time, arriving in Hilo on 22 May. We had come to participate in a deployment in a tropical paradise, but instead we found ourselves--within a very few hours of making landfall--up at an altitude of 11,000 feet on a desolate volcano in the rain and fog. Every day we commuted two hours from Hilo to Mauna Loa Observatory, and two more hours home at day's end. We called our work site Mars, because it was slightly more colorful (though no less lifeless) than the moon. Our offerings to Pele, the goddess of the volcano, forestalled the next major eruption of Mauna Loa long enough for us to complete the task at hand. As proof that we do learn from our experiences, this deployment was completed in only four weeks. And just in case anyone out there is feeling too sorry for the Blue Team, Hilo may not be the greatest place in the world to look for beaches, but we found out where to drive to places that are. We actually suffered very little during our stay on the Big Island. Our heartfelt thanks in this case go out to Charlie Garcia, Eric Yasukawa, Darryl Koon, Russ Schnell and Judy Pereira for making this one a lot of fun. Next came Cerro Tololo and a chance for the Red Team to try their hand for the second time. The activity began with the departure of Frank and most of the others on 24 June. Once again, the support afforded our team by the local staff was nothing short of amazing. The shelter had arrived in excellent condition this time, and assembly proceeded very smoothly. Even the weather cooperated, yielding almost summer-like conditions for some days in the midst of the Chilean winter. It was reported that an unnamed instrument maker associated with the Red Team contingent lay prostrate on the ground for an extended period one day in an attempt to lure condors, but this cannot be confirmed. Nor can we categorically deny that his bunk mates staked him there. The end result was that the fourth GONG network site was left to the CTIO staff and the condors on 22 July, another four-week installation. All kinds of thanks go out to Oscar S a, Ricardo Venegas, and all of the great people working at our sister observatory in the Chilean Andes. The last hurrah for the Blue Team was Big Bear Observatory in southern California. Our adventure began on 24 July. We had had more than enough of airplanes and transoceanic flights by this time, so the team just loaded up our gear, piled into two cars in Tucson, and headed west on Interstate 10. Although we were faced with eight hours of some of the hottest interstate driving anywhere in the US (it was 50°C when we stopped for lunch in Blythe, California), the cool, clean air and spectacular scenery awaiting us at Big Bear Lake made it well worth the little bit of effort to get there. The shelter, too, had no boat rides to contend with, traveling on the same interstate about a day behind us. We saw it go by the window of the cafe at which we were enjoying a leisurely breakfast after a morning of shoveling sand in preparation for its arrival. No paint job was required this time. The shelter looked--not surprisingly--just like it had when it left Tucson a day earlier. What a fine place to work! The locals complained of the oppressive heat (it got up to almost 30°C one day), but those of us used to the desert in July and August in Tucson could hardly relate to their complaints. We had the instrument up and running in about three weeks, but several of us remained a few extra days to train the Big Bear staff in the operation of the instrument, and to breathe just a little more pine-scented air. We grudgingly returned home on 18 August, ending our deployment adventures. Many thanks for help and fond memories go out to Bill Marquette, Jeff Nenow, John Varsik, Karen Carlson, Randy Fear, and Melinda Hope. The final test of the Red Team began with the Udaipur deployment on 27 August. The Udaipur Solar Observatory staff at their new facility just outside of Udaipur already had the shelter and generator roughly in position when the team arrived. There were some delays initially, as two large install kits sent to India by air freight ultimately lagged behind the instrument shipment. The team and the USO staff did a great job of "getting along without" until the gear showed up about half way through the deployment. Although frustrating, these delays did allow the team to take a day off to explore the countryside and do some site seeing. They visited a beautiful Jain Temple near Udaipur, where one of our number was confronted by an angry monkey. During what seemed like an innocent photo opportunity, the primate dropped down from its perch and assumed a fighting stance reminiscent of an antagonist skilled in the martial arts. Sang Nguyen--himself having earned a black belt in Tae Kwon Do--instinctively adopted a threatening posture of his own which so intimidated the ape that it opted to stand down and live to fight another day. Many skills are needed on a well rounded GONG install team. Other than colds and flu that most likely accompanied the team all the way from Tucson, the balance of the deployment went smoothly. On 3 October, that instrument came on line, and the network was complete. As always, there are many folks to thank, but we should certainly name Arvind Bhatnagar, Ashok Ambastha, Sushant Tripathy, Sudhir Gupta, Naresh Jain, Raj Mal Jain, and the team's driver, Chaman Lal Tak. If we learned anything from our globe trotting experiences, it would include the following: o The only thing one can be sure about if ships are involved is that they will arrive late, and the people in country will know about it before we do. o Any GONG deployment team member is by nature and training more dangerous to his or her colleagues than any indigenous insects, reptiles, or sea creatures. o The weather will always be good until it needs to be. o When the going gets tough, trust the local people. They'll know what to do! Operations The highest priority of the new Network Operations group is keeping the data flowing. As one might expect, the most critical people in this regard are the dedicated folks working with us at the host sites. We have provided them with remote status boards, usually located in an area at the host site where the technical people spend most of their daylight hours. For example, at Big Bear, the status terminal is located in one of the offices in the dome, while in Learmonth, it's located in the radio (RSTN) building. What the local folks see is a video simulation of green, yellow and red lights indicating the status of the instrument. Green means the instrument is operational and all monitored parameters are within operating limits; yellow means a parameter has become marginal, but the bits are still flowing (or could if the Sun were out); red means that the site is no longer operational. The display also shows an intensity image derived from the digital data and updated every minute, and a magnetogram derived from the most recent hourly observation. A graphical user interface allows the on-site personnel to inquire as to the state of about 100 monitored instrument and weather parameters, and generally determine the exact source of trouble, should problems arise. What we have noticed so far, in nearly six months of operating instruments in the field, is just how good the people at our host sites are. We have had very few serious problems so far, and in many cases the scientists and technicians at the site have been able to fix things without our help at all! But when serious problems arise, the first step is to contact the GONG Network Operations Duty Responder (known locally as the "Duty Dude"). That is an individual within the Network Operations group, whose job it is to monitor the Internet, the phone, and fax machine for trouble and service requests. Only two of our sites are not currently on the Internet, and they expect to be soon. Thus, electronic mail has proved to be the best way to contact us and get the group here working on the problem. So what does the traffic from the sites look like so far? A review of the activity over the last 30 days looks something like this: Issue Service Requests Routine Correspondence 6 Tape problems 4 Modem problems 2 Network Connections 2 Computer hung 1 Image Display 1 Shelter Thermostat 1 The "Routine Correspondence" items include things like shortages of supplies, and other notes and comments on the routine operation of the instrument. We have had some problems with the modems at two of the sites, requiring in both cases some intervention by local experts. The so called "Network Connections" problems involve failure of the local-area network at two of the sites, which temporarily kept us from obtaining status and operations traffic over the Internet. (The modems are the fall back in this situation.) The "Hung Computer" was the SPARC station (not actually part of the data path), and was fixed with a reboot; the cause is still unknown. The biggest problem is data taping, and this one-month score card somewhat understates the problems that we have been having with tapes and tape drives. Two of the service requests during the last month relate to Exabyte tape drives that had to be replaced at two different sites with new units shipped from Tucson. Prior to this, we have had a number of other problems with the taping task, requiring operator intervention in most cases. These were also caused by tape related problems, though not the fault of the Exabyte drives themselves. Fortunately each site has four drives in service, and two more available as additional backups. For this reason, we have lost only about six hours of data from a single site as a result of these various failures since the first station came on line in March. Still, we're sure that taping issues are the biggest source of frustration for the on-site personnel, to say nothing of the operations group here! We are anxious to have a look at our data-taping software to see if we can't make it more robust to hardware failure, and see if we cannot take better advantage of the redundancy that we have at our disposal at each site. In addition to this sort of "passive" support of the GONG network (that is, sitting by and waiting for the phone to ring), we also have some active procedures in place. Every morning (Tucson time) someone remotely connects to each station, looks at the status, and reads any messages sent to us by our hosts, or by the operating software. Next, we download the last 24 operating hours of instrumental engineering parameters. These minute-by-minute values, such as power supply voltages and motor currents, are analyzed daily back in Tucson for unusual behavior, and become part of a data base allowing longer-term analysis for trends and signs of fatigue and imminent failure. Data Management and Analysis Center The DMAC has successfully integrated the algorithms for refining the limb geometry and extracting MTFs and for merging multi-site data using a MTF-derived weighting scheme into the data reduction pipeline. The DMAC reduced network data acquired by seven deployed network instruments (including the prototype in Tucson), test data acquired to verify the successful installation of six instruments, and test data acquired as construction of the observing instruments was completed at the University of Arizona farm site before the units were disassembled for shipment. The project also generated data products from the first GONG month (36 days) of network data that was acquired by the three-site mini-network of Teide, Learmonth, and the Big Bear station in Tucson. This included month time series, power spectra, and mode frequency information that were made available to the participants in the inversion workshop that was held in Boulder. With the onset of network operations, the data reduction activity in the DMAC has increased dramatically. For example, the number of site days calibrated per calendar month has increased from 25 in February to 142 in September. The upstream data reduction stages that perform site-dependent processing (VMICAL, GEOMPIPE, DNSPIPE, and AVER) have ramped-up more quickly than anticipated. Except for AVER, these reduction stages have been maintaining backlogs of about 50 site-days. A key technical hurdle for merging multi-site images was the precise determination of the orientation of the cameras in the observing stations. A procedure developed and implemented by the instrument team successfully satisfied this requirement. This information was integrated into the data reduction software that registers the velocity images into heliographic coordinates. The DSDS has moved the user interface for the catalog query and data product request functions from CURSES to HTML. The DSDS also recently conducted a survey of its users regarding various technical aspects of using the projects facility for accessing data products. The DSDS received 25 responses from the 64 GONG members to whom the survey was e-mailed. With the ramp-up of network operations, the frequency and volume of data requests has also increased. Eleven data distributions totaling 9.7 GB were made in July; six distributions totaling 19.2 GB were made in August; nine distributions totaling 423 MB were made in September. This compares to zero distributions in February and two distributions totaling 101 MB in March. The DMAC Users' Committee met at HAO in Boulder in August, and will meet in Tucson during December. In August, Tatia DeKeyser joined the project. Tatia operates the data reduction stage that refines the geometry of the solar images and extracts the MTF. Project Management Our annual budget struggle was resolved favorably in February with a $2.6M figure for FY 95. This has been adequate to allow us to deploy the field stations at a rate of about one every six weeks, since the first station arrived on Tenerife in January. It should be borne in mind that, since then, we have been simultaneously completing the integration and testing of the remaining stations, completing site land preparations, deploying stations, and operating them as they came on line. This has been an interesting challenge since the deployments have necessarily meant that we were carrying on these many tasks with half or more of our instrument staff thousands of miles away from home base. Nevertheless, the deployments were completed on schedule and under budget. As the project has completed its original development mission and has made the transition to a network of operational observatories, a sadder activity has been saying goodbye to some old friends. In the past few months, Mark Trueblood has joined the US Gemini Project as its Project Engineer, Bret Goodrich has also joined Gemini, Bob Hartlmeier has retired, Don Farris, Arden Petri, and Dee Stover have transferred back to ETS, Jerry Gonzales, Jeff Vernon, and Tom Bajerski have all left NOAO for other employment. Their contributions to the GONG program have been essential and we wish them every success. As usual, our FY 1996 budget is unclear. Our original request included funds to vigorously pursue a new development program to produce a 1024 x 1024 square-pixel camera that might be installed as a retrofit into the existing field stations to produce even higher quality data. However, this request was rejected in favor of a more modest one that included little more than the basic funding required to operate and maintain the network and data management facilities. Nevertheless, it amounts to some $2.1M, and if this figure survives the Federal process, and the network is blessed with relatively trouble-free operation, we may be able to at least begin a modest exploration of the new camera concept. All in all, we are looking forward to our first year of "normal" operations with great anticipation. John Leibacher and the GONG Project Team
Previous Article Next Article Table of Contents