GS TCS crash troubleshooting spreadsheet
December 2012 event
Below you will find the email thread of the 2012 TCS crash event.
Hola Arturo,
FYI.
The following email thread shows the type of problems we experienced
with the s/w versions. We had the very same behavior but with different
symptoms on the Feb run. On both the TCS version was the one
loaded/unloaded/loaded...after a huge amount on h/w bds.
swapping/config.changes/etc....
Saludos,
Ramón.
===========================================================================================================
From: Ramon Galvez
Sent: Wednesday, December 05, 2012 12:42 PM
To: Maxime Boccas
Cc: Gustavo Arriagada; Rolando Rogers; Andrew Serio; Cristian Urrutia;
Benoit Neichel; Rodrigo Carrasco; Javier Luhrs; Pablo Diaz; Gaston
Gausachs; Fabrice Vidal
Subject: Re: NGSWFS Status
Hola Maxime,
We did the swap of version on Monday and we had no issues so far with the new one. We have no clear understanding on what may have caused the problem we had this run. Since the only change we had between the previous run ( Oct 30 --> Nov 03 ) was the TCS s/w version we installed it and since that time we had no issues but as aforementioned going again to the new one ( Monday morning ) has not give problems either though it was tested w/o offlads and w/o closing loops so more test can be done in those conditions.
On the h/w side we have a complete pwr. & grndg. prev. maintenance scheduled for this subsystem to be carried out tomorrow morning.
It is very important to say that the NGSWFS probes have been used extensively in all the runs we have had since 2010 with the highly demanding Probe Mapping - I have seen demands of ~10nm ! - and all that time it has been successful. I have talked to Benoit about this and we think that the demand rate is unnecessarily high. We do have some minor timeouts - that are easily solved on the fly - that we are working on.
The weird behavior we saw on Friday and on Saturday was completely different, i.e. the OMS58 controller (8 axis controll.) was completely lost and this was highly intermittent making it very difficult to diagnose. We have had no more problems since Sunday and a lot of Probe Mapping was successfully done that night. Maybe Benoit wants to comment on this.
Saludos,
Ramón.
On 05-12-2012 12:08, Maxime Boccas wrote:
> I was a bit behind the news with my last email asking about this.... It seems confirmed there was a SW version issue then? Shall we review our change control protocol?
> Thanks
> M
>
> -----Original Message-----
> From: Gustavo Arriagada
> Sent: Monday, December 03, 2012 11:09 AM
> To: Ramon Galvez; Rolando Rogers
> Cc: Cristian Urrutia; Benoit Neichel; Rodrigo Carrasco; Javier Luhrs;
> Pablo Diaz; Maxime Boccas; Gaston Gausachs; Fabrice Vidal
> Subject: RE: NGSWFS Status
>
> These are very encouraging news, good job everyone!!!, hopefully today's tests will confirm your suspicions.
>
> Saludos
>
> Gustavo
>
>
> ________________________________________
> From: Ramon Galvez
> Sent: Monday, December 03, 2012 1:24 AM
> To: Gustavo Arriagada; Rolando Rogers
> Cc: Cristian Urrutia; Benoit Neichel; Rodrigo Carrasco; Javier Luhrs;
> Pablo Diaz
> Subject: NGSWFS Status
>
> HOla Rolando, Gustavo y Todos,
>
> Today we had to come up to work on the NGSWFS issues that were reported by Benoit. As I've mentioned to some of you I saw similar problems during h/w & s/w development, but that was about 4 years ago. From that time this is the 1st occasion that that type of highly intermittent issues showed up. Mainly while in Probe Tracking Mode. All this started on this particular run, nothing like this was seen on the run we had on Oct 30 --> Nov 03 or in any run since we started with the Commissioning Runs back in 2010.
>
> As reported, we have had the following issues :
>
> 1. Sudden stalled mechanism, i.e friday night P2Y stalled at -1.4[mm] while going from 0.0 --> 14[mm] .
> 2. Unable to do a very basic Index on some probes.
> 3. Spiral routine getting stalled by a Hardware Sensor Limit read by the s/w (not a hard Hardware Pwr. Shutdown Limit). Today we had P2Y reaching one of those limits with no explanation and we verified with Cristian that it was really reaching it physically. This was after having a demand from the TCS due to a CWFS2 following selection.
>
> To rule out the s/w TCS version in use I asked to have the version we used on the aforementioned previous run. Once Rodrigo was done with his GSAOI tasks it was remotely installed by Javier @ ~11pm. Then - the night time crew Benoit, Rodrigo, Drew - started to use the Tracking Mode for real science Probe Mapping on 3 different targets. No issues have showed up; this implies continued use of all 3 Probe Tracking for 2 hrs and it is actually on-going.
>
> We - Ramón and Cristián - want to do more similar tests during daytime so I entered an urgent TR for tomorrow to continue the tests prior to change to the new TCS version for further tests from the SOS Group.
>
> Saludos,
>
> Ramón, Cristián
>
Saludos,
Ramón.
--
Ramon L. Galvez
Senior Electronics Engineer
Gemini Observatory
www.gemini.edu
Phone : +56 51 205678
Recept. : +56 51 205600
Fax : +56 51 205655
February 2013 event
Below you will find the email thread of the 2012 TCS crash event.
Hola Arturo,
FYI.
The following email thread shows the type of problems we experienced
with the s/w versions. We had the very same behavior but with different
symptoms on the Feb run. On both the TCS version was the one
loaded/unloaded/loaded...after a huge amount on h/w bds.
swapping/config.changes/etc....
Saludos,
Ramón.
===========================================================================================================
From: Ramon Galvez
Sent: Wednesday, December 05, 2012 12:42 PM
To: Maxime Boccas
Cc: Gustavo Arriagada; Rolando Rogers; Andrew Serio; Cristian Urrutia;
Benoit Neichel; Rodrigo Carrasco; Javier Luhrs; Pablo Diaz; Gaston
Gausachs; Fabrice Vidal
Subject: Re: NGSWFS Status
Hola Maxime,
We did the swap of version on Monday and we had no issues so far with the new one. We have no clear understanding on what may have caused the problem we had this run. Since the only change we had between the previous run ( Oct 30 --> Nov 03 ) was the TCS s/w version we installed it and since that time we had no issues but as aforementioned going again to the new one ( Monday morning ) has not give problems either though it was tested w/o offlads and w/o closing loops so more test can be done in those conditions.
On the h/w side we have a complete pwr. & grndg. prev. maintenance scheduled for this subsystem to be carried out tomorrow morning.
It is very important to say that the NGSWFS probes have been used extensively in all the runs we have had since 2010 with the highly demanding Probe Mapping - I have seen demands of ~10nm ! - and all that time it has been successful. I have talked to Benoit about this and we think that the demand rate is unnecessarily high. We do have some minor timeouts - that are easily solved on the fly - that we are working on.
The weird behavior we saw on Friday and on Saturday was completely different, i.e. the OMS58 controller (8 axis controll.) was completely lost and this was highly intermittent making it very difficult to diagnose. We have had no more problems since Sunday and a lot of Probe Mapping was successfully done that night. Maybe Benoit wants to comment on this.
Saludos,
Ramón.
On 05-12-2012 12:08, Maxime Boccas wrote:
> I was a bit behind the news with my last email asking about this.... It seems confirmed there was a SW version issue then? Shall we review our change control protocol?
> Thanks
> M
>
> -----Original Message-----
> From: Gustavo Arriagada
> Sent: Monday, December 03, 2012 11:09 AM
> To: Ramon Galvez; Rolando Rogers
> Cc: Cristian Urrutia; Benoit Neichel; Rodrigo Carrasco; Javier Luhrs;
> Pablo Diaz; Maxime Boccas; Gaston Gausachs; Fabrice Vidal
> Subject: RE: NGSWFS Status
>
> These are very encouraging news, good job everyone!!!, hopefully today's tests will confirm your suspicions.
>
> Saludos
>
> Gustavo
>
>
> ________________________________________
> From: Ramon Galvez
> Sent: Monday, December 03, 2012 1:24 AM
> To: Gustavo Arriagada; Rolando Rogers
> Cc: Cristian Urrutia; Benoit Neichel; Rodrigo Carrasco; Javier Luhrs;
> Pablo Diaz
> Subject: NGSWFS Status
>
> HOla Rolando, Gustavo y Todos,
>
> Today we had to come up to work on the NGSWFS issues that were reported by Benoit. As I've mentioned to some of you I saw similar problems during h/w & s/w development, but that was about 4 years ago. From that time this is the 1st occasion that that type of highly intermittent issues showed up. Mainly while in Probe Tracking Mode. All this started on this particular run, nothing like this was seen on the run we had on Oct 30 --> Nov 03 or in any run since we started with the Commissioning Runs back in 2010.
>
> As reported, we have had the following issues :
>
> 1. Sudden stalled mechanism, i.e friday night P2Y stalled at -1.4[mm] while going from 0.0 --> 14[mm] .
> 2. Unable to do a very basic Index on some probes.
> 3. Spiral routine getting stalled by a Hardware Sensor Limit read by the s/w (not a hard Hardware Pwr. Shutdown Limit). Today we had P2Y reaching one of those limits with no explanation and we verified with Cristian that it was really reaching it physically. This was after having a demand from the TCS due to a CWFS2 following selection.
>
> To rule out the s/w TCS version in use I asked to have the version we used on the aforementioned previous run. Once Rodrigo was done with his GSAOI tasks it was remotely installed by Javier @ ~11pm. Then - the night time crew Benoit, Rodrigo, Drew - started to use the Tracking Mode for real science Probe Mapping on 3 different targets. No issues have showed up; this implies continued use of all 3 Probe Tracking for 2 hrs and it is actually on-going.
>
> We - Ramón and Cristián - want to do more similar tests during daytime so I entered an urgent TR for tomorrow to continue the tests prior to change to the new TCS version for further tests from the SOS Group.
>
> Saludos,
>
> Ramón, Cristián
>
Saludos,
Ramón.
--
Ramon L. Galvez
Senior Electronics Engineer
Gemini Observatory
www.gemini.edu
Phone : +56 51 205678
Recept. : +56 51 205600
Fax : +56 51 205655
February 2013 event
The team:
Ramon Galvez, Vanessa Montes, John White, Pedro Gigoux, Roberto Rojas, Cristian Urrutia, Chris Morrison, Cristian Silva, Jose Varas, Benoit Neichel, Ariel Lopez, Pedro Ojeda, Herman Diaz
May 12-13 event
SW and IS tools compilation document
In the link above you will find all the tools SW G and IS G have identified to be used during this troubleshooting process.
May 12-13 TCS crash troubleshooting plans
In the link above you will find the troubleshooting plans that each group is proposing.
May 12-13 event
The team:
Ramon Galvez, Roberto Rojas, Cristian Urrutia, Chris Morrison, Cristian Silva, Jose Varas, Arturo Nunez, William Rambold, Javier Luhrs, Gustavo Arriagada
SW and IS tools compilation document
In the link above you will find all the tools SW G and IS G have identified to be used during this troubleshooting process.
May 12-13 TCS crash troubleshooting plans
In the link above you will find the troubleshooting plans that each group is proposing.
