Gemini Near-Infrared Spectrograph Restart Review
Committee Report

October 1999

Review Committee Members:

Roy Autry (Independent Consultant)
James Breckinridge (Chair; JPL)
Mark Hunten (IGPO)
Dave Montgomery (IGPO)
Adrian Russell (UK/ATC)
Alan Tokunaga (UH/IfA)

1.0 Executive Summary

The Review Committee was thoroughly impressed by the amount and quality of work done by NOAO personnel in redesigning the GNIRS instrument and in preparing for the July review. The GNIRS team (and the project manager especially) has a strong commitment to the schedule and budget. It is very encouraging to see the openness with which they are carrying out their work as well as their openness to external input. We now believe that NOAO is well on the way to developing a fully functional and viable system that will meet the design requirements. This represents a complete turn around on this project. Not only have they clearly integrated formal project management in the new program, but the lead mechanical engineers have done a great job in redesigning the instrument so quickly. The roles of customer and vendor have been recognized and the project is more open, actively seeking input from Gemini and the wider community. This is an enormous improvement over the situation faced by NOAO, IGPO and the Committee last fall. The Committee commends NOAO for the effective and highly competent manner in which they confronted the previous problems and created a new project that shows great promise in delivering a world-class scientific instrument.

Overall the new design is at the PDR level, though optics are clearly beyond that level while other design aspects (e.g., software) are not at the PDR level yet. With that one notable exception (software), NOAO's new instrument design seems to be sound, and it addresses each of the technical concerns listed in the December 1998 External Review Report. Software is singled out not because it is unacceptable, but because it was not included in the Restart Review, except for a very brief and general description by Richard Wolff of NOAO. Pending development of a software plan, the Committee has no reservations in recommending continuation of the GNIRS program, at least to the PreFab Review tentatively scheduled for next February or March. Finally, the Committee notes that the issues and concerns now being faced by the project personnel and management are those of a normally functioning technical development effort. There should continue to be informal reviews with IGPO involvement as key subsystems get further developed, including thermal analyses, finite element mechanical analyses, and software.

The Restart Review committee was given a Charge by the IGPO, which included key issues that need to be assessed as part of the review. These include:

1) What is NOAO's current cost estimate to complete GNIRS and on what schedule?

The budget presented was $3.88 million, including $444,000 to fund the effort during 1999 up to the July review, with system delivery in June 2002. Since the budget presented did not include a detailed software plan, this expenditure projection represents a lower limit on true future costs.

2) Is this estimate consistent with NOAO's planned resources over the duration of the GNIRS project?

The Committee is not in a position to comment on this, because the Committee is unfamiliar with NOAO's overall resources as well as NOAO's commitments to other projects. Failure to present a complete software plan also prevents the committee from answering this question.

3) Have the technical issues (AURA report attached for reference) identified by the AURA review committee been addressed adequately to permit continuation of GNIRS?

NOAO has addressed all of the technical concerns previously expressed by the Committee. Software development however was inadequately addressed during the review, and the Committee recommends that a software development plan be defined before full restart of the project. Among the issues which the Committee was most gratified to see addressed was the way in which NOAO personnel had worked with those developing the On Instrument Wave Front Sensor (OIWFS) and the Integral Field Unit (IFU). These subsystems are being developed with close coordination between NOAO and the developing agencies and under the rigor of well-documented interface control documents.

4) Does NOA0 now have in place an appropriate management structure to track, plan, and control resources to ensure that GNIRS will be delivered on time and budget?

The management control plan looks very good, and the NOAO Program Manager has taken care to include schedule and budgetary metrics by which he can objectively gauge progress and budgetary expenditures on a timely basis. The tracking and planning systems now in place should allow the timely discovery of any major problems with the schedule and budget. As mentioned above, the software effort has yet to be defined, and the schedule was not adjusted for labor leveling, nor was the critical path identified. These activities may lead to schedule growth, along with the potential need for additional full system cool-down tests. There was some concern expressed that the project has 3 project scientists. It was assumed, however, that Jay Elias would assume a lead role among the science team in the future.

5) Are there any approaches to designing and fabricating GNIRS that can significantly accelerate the planned delivery, e.g., through the injection of additional NOA0 resources, outsourcing the fabrication of components, etc.?

The Committee suggests several items for NOAO to consider under this charge. Outsourcing may be a viable or even necessary choice in some areas - notably in software development, parts machining and electronic circuit card development. The Committee requests that NOAO investigate and report on outsourcing as part of the Pre-Fab Review. In addition, having the OIWFS delivered as a working instrument will reduce the load on NOAO personnel during system integration. Also, the Committee recommends that NOAO investigate the possibility of buying, renting or contracting to use a coordinate measuring machine and a heavy duty welding rig for the system integration and test efforts. Finally, the committee suggests that penalty clauses be used in contracts for critical components to help ensure their timely delivery.

6.) Have all contractual matters involving out of scope work, definition of work, and interfacing requirements been settled with IGPO?

No, although much progress apparently has been made on the definition of work and interfacing requirements, and negotiations between IGPO and NOAO regarding out of scope work remained. These were to be undertaken in the days following the July meeting.

2.0 Schedule

A primary area of concern within the committee was schedule. NOAO presented a schedule that showed GNIRS delivery to the Gemini program in June 2002. This schedule had been recently prepared by the NOAO program staff, and it represented what they consider to be an aggressive plan without being overly optimistic. However there is no room for major slips for items on the critical path, which is not consistent with the committee's experience with other projects of this magnitude. Because the NOAO schedule had been recently completed, it had not been "scrubbed" prior to the review meeting. Neither the detailed critical path nor the amount of schedule contingency was known at that time.

To minimize the exposure to overruns on the schedule and budget, every effort must be made to expedite and simplify the design, fabrication, and integration work. For example the testing of cryogenic motors represents a task that can be eliminated now, by simply selecting the motor that is being used in NIRI to be done with the matter, even though it is a motor with higher torque than they need. Another example is that if there is some relaxation of the optical specification that would be acceptable to the Gemini Project, GNIRS should take advantage of this situation.

A key type of risk mitigation is tied to the number of cool downs required during the integration and test phase. In principle the 4 planned cool downs might be all that are needed. Experience indicates, however, that more cool downs will be needed. NIRSPEC required 11 cool downs before it was shipped to the Keck telescope and this was a result of their problems with light leaks. Likewise, IRCS will probably have 10 cool downs before it is ready for the telescope. The GNIRS team plans to test each subassembly separately, and this is an excellent plan and will no doubt reduce the number of cool downs. Nonetheless there is a distinct possibility that more cool downs than planned will be needed, and this area represents a major risk factor to the delivery schedule.

2.1 Other Strategies to Reduce Risk

3.0 Software

Software was viewed as a major risk area for GNIRS. While the committee agrees that the proposed approach, namely a "thin layer" of EPICS on top of C code, may be the most straightforward means of implementing code in the current Gemini software environment, we were nonetheless concerned not to see a more detailed software plan with manpower identified. We feel that this issue needs the urgent attention of NOAO. This is particularity important because the presented rough plan has only ~30 weeks of staff effort in it for all software activities, including documentation and support of lab commissioning. Other instrument programs, e.g: GMOS, suggest this estimate could easily be off by factors of three or more. Hopefully the new approach using a thin layer of EPICS will mitigate this risk. If it can be of any use in assessing the software requirements, the team should contact other instrument software engineers (e.g., Steven Beard at the ATC or Hubert Yamada at UH/IfA) who could provide copies of their software design and other documents, as well as advice on where mistakes were made.

A revised software plan that is consistent with the level of effort that has been injected into all the other phases of the revised GNIRS design should be generated. This plan should cover 4 basic areas, namely:

1. Instrument Operations and Control

2. Data Logging and Transmission

3. Instrument Housekeeping

4. GNIRS to Gemini Interfaces

The revised software plan should articulate the staffing, schedule, scope of work, functional requirements, cost, risk analysis, and documentation associated with each of these 4 elements. This plan will be circulated among the Restart Review committee for comment as soon as it is released to Gemini.

4.0 Electronics

The committee was impressed with the electronics design presented but remains concerned about the amount of manpower dedicated to electronics in the new GNIRS plan. A single electronics engineer and technician may not be adequate to completely wire and test an instrument as complex as GNIRS. NOAO should be prepared to involve more electronics manpower if this area falls behind in the overall integration effort. NOAO should also be aware of the NIRI experience with computer crashes leading to thermal instabilities in the instrument because the cryo-heads are being driven by the instrument computer. In practice, the downtime associated with these crashes is set by the thermal time constant of the instrument (perhaps ~1 hour), not by the reboot time of the computer. Needless to say, computer systems tend to crash often during the integration and test phase of an instrument's development, hence this can lead to potentially hours lost each day. NOAO is urged to adopt the same back-up stepper motor driver board that NIRI is using to eliminate this problem.

5.0 Budget

NOAO personnel estimated the budget for completing the GNIRS development effort to be $3.88 million, including approximately $444,000 for the redesign effort leading to the July review. Note that the dollar figures quoted here are direct labor and capital dollars and do not include the cost of the detector or array controller. As with schedule, the amount of budget contingency could be estimated only roughly. The NOAO budget was determined by multiplying total project labor hours by an average wage figure. However, some project personnel are professional staff members who are paid directly by NOAO. In addition, the program labor loading had not been scrubbed for "double booking" of personnel, nor had the appropriateness of the average wage figure been examined for the remaining personnel. According to NOAO estimates of these factors, a contingency of $400,000 to $600,000 may be built into the estimate. This remains to be verified, however. The issue of budget contingency should be clarified, and expenditures during the project should be watched closely, especially in view of the potential for additional full system cool-down tests prior to delivery. In any event, the overall proposed cost to complete GNIRS is commensurate with the complexity of the instrument. We believe the project manager has implemented the tracking mechanisms needed to adequately assess budget and schedule issues on a regular basis, but keeping within the budget and schedule will require very strong management discipline.

6.0 Systems Engineering

Another concern expressed by the Committee during the course of the meeting was the staffing in the program's systems engineering team. Under the management structure presented, the systems engineering task area is headed by three NOAO scientists working as a team. The Committee was concerned that the scientists' understandable concern about building an instrument to "do the best science possible" would interfere with the demand to build an instrument that was "good enough" to meet the system requirements. When this concern was expressed in the final Committee debriefing with NOAO Director Sidney Wolff and GNIRS Program Manager Neil Gaughan, Neil defended the scientists fulfilling the system engineering role and gave examples of their ability to meet the engineering capabilities required for that position. Neil's defense of his systems team, combined with the manner with which they jointly handled complex technical questions during the Restart Review, significantly alleviated the Committee's concerns, so no change in staff is recommended at this time. It is crucial that the team builds on this change and that a totally pragmatic attitude pervades in this team through the remaining phases of the project.

7.0 Flow Down of Requirements

An area in which the NOAO presentation seemed less than satisfactory in the opinion of some Committee members was the rigorous derivation of system requirements flowdown into design parameters. This is not to say that NOAO personnel did not perform such an analysis, but rather that it was not discussed in detail in the presentation. Indeed, several of the presenters referred review attendees to a wall chart detailing some aspects of the flow-down analysis. Given the excellent progress that NOAO personnel have made in the instrument redesign and the apparent quality of the effort and of the resulting system design, the Committee does not consider this an oversight serious enough to delay the project.

8.0 Thermal

A conceptual design level thermal analysis has been carried out and demonstrates that the new design is radically better than the original instrument, with near optimal placement of the coolers in the middle of the cold mass. The next step is to carry out a more thorough preliminary analysis with a model of the structure. Since this will validate the layout chosen, we advise that this analysis be carried out in the next couple of months (i.e., well before the pre-fabrication review). We suggest that the NOAO team together with David Montgomery of the IGPO review this informally, together with the preliminary flexure analysis.

This would be an excellent opportunity to evaluate some detailed issues including:

• Is it possible to use only two cryo-coolers, given that a liquid nitrogen pre-cool system will probably be used?

• What is the performance trade off between two floating radiation shields versus a light-tight active rad-shield vs. super-insulation, particularity in light of a more detailed understanding of the mass-flexure trades?

9.0 Mechanical

The overall design has been completely overhauled since CDR and we applaud the innovation and simplification the new GNIRS team has incorporated into the new design. The instrument has been transformed from an instrument that many people were dubious about being technically viable to one that everyone on the review committee feels can be made to work. Below are specific comments and suggestions for the mechanical design of GNIRS.

9.1 Modularity

The layout is not modular in the structural sense and it is appreciated that there are good reasons to adopt this design. However, we think the number of "modules" should be minimized and the mechanisms should be configured as self contained cartridge assemblies that can be extracted from the outside without disturbing the joints between the modules. This might be difficult (or impossible) in the case of the prism turret and could dictate a different axis orientation. Some mechanisms are mounted on a closure plate with a bolted flange that provides structural strength and a light seal. This arrangement does not lend itself to alignment adjustment and implies that the adjustment (thick shims) is between the sub-plate and the mechanism assembly. There is also an issue in handling the individual cast modules. It should be possible to remove the fasteners and have some secondary feature prevent it from falling off. Again, a labyrinth type sealing arrangement will exacerbate this problem.

9.2 Handling

The external layout is compatible with the Gemini facility handling equipment, although there a few details to resolve regarding the actual hoist interfaces. We congratulate the GNIRS team on accommodating the Gemini handling equipment so well in the new design. There needs to be some thought given to the safe placement and removal of modules with an overhead crane or hoist while the instrument is in the side looking orientation. As an example, keyhole slots and shoulder pins might be one-way.

9.3 Mechanisms

We believe the team has made the right decision to use cold stepper motor drives with open loop step counting position control, defined by micro-switch datums. We have some general comments and on each of the mechanisms:

9.3.1 Torque margin

We think it is important to estimate the minimum motor torque required to operate each mechanism and to set a safety factor for the required motor torque. This can account for uncertainties in the calculated friction properties. It might be possible to lower the safety margin after the results of the prototype testing are known. Of course, in a highly geared system such as worm mechanisms with anti-backlash screws, it is essential to consider the efficiency (or lack thereof) of transmission. Note that it may be necessary to provide friction damping of transmission elements in the stepper motor transmissions to prevent excitation of free body torsional vibration. Orientation will be an issue in mechanisms that are unbalanced (e.g., the slit slide) by design or configuration (e.g., the camera turret with one camera removed). We again recommend opening a dialog with the MICHELLE engineers who are now cold testing MICHELLE mechanisms in similar temperature ranges to those planned for GNIRS.

9.3.2 Balancing mechanisms

In general, all mechanisms should be balanced except perhaps for small masses. Failure to do so will tend to lead to abnormally high wear on Vespel parts, high flexure, and more torque required with associated dynamic problems with the stepper motors when they are driving mechanisms "down hill". This may be a challenge for the Decker mechanism (50 µm repeatability) and will almost certainly be a problem for the slit slide (1 µm repeatability). There is no balance mechanism shown, and we suspect that it will get more difficult to implement as the design evolves.

9.3.3 Slit/Decker module

We suspect that the cam follower rollers, which are machined at 45 degrees, could be replaced with "V" rollers machined at angles perpendicular and orthogonal to the reference surfaces. This should lead to reductions in cost and risk with this mechanism and is similar to how the GMOS linear slit exchange mechanism works.

9.3.4 Grating turret

The proposed grating turret appears to be much simpler and therefore lower risk than the previous design. However, we suggest using treated bearings mounted in separate bearing housings (housings having detents, brakes and drive train components) rather than making the races integral with the turret structure. The turret structure can then attach to this separate turret support and drive assembly. This approach has the advantage (beyond the use of commercially treated bearings) of being able to remove the turret assembly without disturbing the drive, thereby allowing the drive to be tested independently. Cabling and connectors should be worked out before the size and shape of the baseplate is defined permanently.

9.3.5 Home switch

Again, we endorse the use of a home switch and only suggest that the GNIRS team consider mounting the connector on the switch structure (e.g., a micro-miniature D connector). Alternately, mounting the micro-switch on an up-stand so that it remains fixed is an option that should be considered.

9.3.6 Camera turret

The use of an internal hub with the cameras attached at each end is clever. From the rendered images though, it appears that they are located on the hubs by means of a radius. We suggest using a "V" formed by two flats as it may be advantageous in terms of repeatability and adjustability through thick shims. Also, if the camera units are to be replaced individually, then we suggest that the "V" face upward in the exchange position. This allows the unit to be placed safely on the mounting feature, followed by the retaining component.

9.3.7 Prototyping mechanisms

We are in general concerned about the use of prototyping unless it is intended to measure something very specific, for example the friction of Vespel on Aluminum at 65K or the performance of actual Mechanisms. We prefer to allow time to test the actual mechanism and think through contingency to solve problems in areas where the design is less sound. Otherwise there is a tendency to hold everything up until the results of the prototyping are known. We suggest creating a prototyping plan that defines all of the prototyping tests expected and lists what information each test is intended to yield in order to "bound" the effort and maximize its return.

9.4 Castings

We encourage the proposed use of castings but suspect there is risk in using a casting for the center portion of the vacuum vessel. We suggest that the team have a dialog with Ian Bryson and Fraser Morrison at the UK/ATC as they considered casting part of the MICHELLE vacuum vessel but rejected it eventually in favor of a weldment. The problem of course is porosity which can lead to direct leaks and also difficulty in machining O-ring grooves and sealing surfaces due to grains of sand being exposed when surface milling cast modules. Also, the step in the center casting may result in a significant pressure induced tilt of the instrument's internal structure. The team should consider selecting a casting vendor that can also machine and inspect the modules. There are many advantages to ordering more than one casting of each type, hence consideration should be given to ordering more than one casting per module. This relates to the aforementioned modularity problem (9. 1), in that modules can be exchanged for "dummy" modules for various test purposes.

The interfaces to the casting modules present some problems in reducing the effective conductivity, compromising stiffness, complicating photon tightness and complicating assembly. These may not be great problems individually but it may be worth considering joining the mid, fore and aft castings as one casting. The mechanism modules could potentially be cartridge style sub-assemblies that plug in from the outside and have bolted flanges. Possibly, the mechanism cartridge can go all the way through and be supported off both casting walls or be internal features to avoid local flexing of the casting wall. The casting wall thickness should taper away from the mid section either continuously or discretely from casting to casting. As with the calculations that Jay Elias did on the radiation shield, it may also be beneficial to increase the casting thickness around the region where the heat is extracted.

9.5 Vacuum jacket

This layout provides great accessibility compared to the old design. Though we have reservations about using a casting as part of the vacuum system, this may be addressed by pushing the responsibility for leak tightness onto the vendor. As a suggestion, the team might consider using a square configuration for the center section with square end covers fabricated from aluminum plate. There are some real advantages to this approach including robustness, a wider choice of vendors, and we believe that the rear handling frame truss would not be required at all. The rear cover would easily be able to support the entire instrument weight with appropriate pads attached to the comers. We believe it would be easier to integrate the window area into the front cover also.

10.0 Flexure

The opto-mechanical layout should prove a lot more effective in meeting the flexural performance specifications than the old design. However, at this time it is unclear whether it does meet these requirements or, if it does, by what margin and at what cost in terms of design complexity. It is worth considering that the GMOS and HROS spectrograph teams decided early on in their instrument development that active flexure compensation was necessary given the mass constraints. It is admittedly premature to conclude that this control philosophy is necessary for GNIRS, but it may be less time consuming, less expensive, and less risky for the GNIRS instrument to take the same route. This issue impacts the design of the instrument in a number of fundamental ways, e.g., would adding flexure compensation allow the cold mass to be reduced significantly and, if so, what are the advantages for the thermal design/performance/schedule? Again, resolution of this basic question as soon as possible is advantageous for GNIRS and we encourage continued discussions with the IGPO as these design trades emerge with the evolution of the instrument.
 
 

NOAO Intranet Services

NOAO Copyright  Statement