Fregat/Meteor loss caused by Vostochny being too vostochny, a computer outsmarting itself, and gimbal lock

[DDE]

Quote

The complex failure scenario of the second Soyuz rocket launch from Vostochny continued emerging in the days following the accident. Although the culprit had quickly been pinned down by flight control specialists, even seasoned space engineers, who were not directly involved in the intricacies of guidance systems, struggled to fully comprehend the bizarre nature of the accident.

In the Soyuz/Fregat launch vehicle, the first three booster stages of the rocket and the Fregat upper stage have their two separate guidance systems controlled by their own gyroscopic platforms. The guidance reference axis used by the gyroscopes on the Soyuz and on the Fregat had a 10-degree difference. The geographical azimuth of previous Soyuz/Fregat launcher from Baikonur, Plesetsk and Kourou normally laid within a range from positive 140 to negative 140 degrees. To bring the gyroscopic guidance system into operational readiness, its main platform has to be rotated into a zero-degree position via a shortest possible route. The azimuth of the ill-fated Vostochny launch was 174 degrees, and with an additional 10 degrees for the Fregat's reference axis, it meant that its gyro platform had to turn 184 degrees in order to reach the required "zero" position. 

In the Soyuz rocket, the gyro platform normally rotated from 174 degrees back to a zero position, providing the correct guidance. However on the Fregat, the shortest path for its platform to a zero-degree position was to increase its angle from 184 to 360 degrees. Essentially, the platform came to the same position, but this is not how the software in the main flight control computer on the Fregat interpreted the situation. Instead, the computer decided that the spacecraft had been 360 degrees off target and dutifully commanded its thrusters to fire to turn it around to the required zero-degree position. After a roughly 60-degree turn, the gyroscope system on the Fregat stalled, essentially leaving the vehicle without any ability to orient itself in space.

http://russianspaceweb.com/meteor-m2-1.html

Edited December 1, 2017 by DDE

[PB666]

22 hours ago, DDE said:

http://russianspaceweb.com/meteor-m2-1.html

"

Instead, the computer decided that the spacecraft had been 360 degrees off target and dutifully commanded its thrusters to fire to turn it around to the required zero-degree position.
"

lol, do I need a job, I just finished programming that exact issue. If query is between 0 and π/2 and subject is between 3π/2 then add 2π to query and calculate else just calculate.

=IF(AND(RC[-15]<=0, RC[-15]<=(PI()/2),RC[-1]>=(3*PI()/2)), ABS(RC[-15]+2*PI()-RC[-1]),ABS(RC[-1]-RC[-15])) [This is in Excel cause my visual studio license has expired]

That particular equation was to determine if the space craft was traveling prograde by looking at dx, dy per dt (dz had to be projected to equator). First it had to determine the radial direction of velocity vector, then compare it with an ideal prograde vector for that position, if it was closer to the ideal prograde vector, then it was considered to be prograde (and cartesian coordinate mapping then used prograde assumptions), otherwise is was considered to be retrograde. From that point on the circular velocity (in x,y,z coordinate system) and radial velocity can be extracted.

A smarter and simpler solution is just to make a course change at Northern geographic pole, since they were spitting distance from it, and then make a correction turn at the equator after they were nearly in orbit. Making a course change at the equator 'greatly' simplifies the math and course change.

EDIT: I should point this out for the general audience.

If you trigger a flight navigation system with no other information other than the desired vectors, then presumably that flight system gathers information from satellites and other spacecraft systems, and then goes through an initiation by which it can then use Markov Chain Ramer–Douglas–Peucker  chain analysis or the like. But the problem in initiating the system lies in a couple of facts. (for simplicities sake an initiation process does not need a Markov chain simply because the velocity and acceleration vectors can provide enough confinement in the first pass that testing vectors are not neccesary).

That traveling close to a zero point you either need to convert to 2π from zero or zero from 2π but for velocities and accelerations its much more complicated. The vectors for prograde and retrograde are +/- π/2. For instance if your position is radius Px  = 4525 km, y = 4525km from earths center and the prograde direction vector is such that  Arcsin(pY) = ArcCos(pX) [There are ussually two solutions for arcsin and arccos but only one solution that they match] is +/- π/2. The angle for positiion direction is π/4 in this exemplary coordinate system, and the ideal prograde circular velocity has a direction vector of 7π/4. Lets say that you add a radial or normal vector that bring a rotation of more than π/4 to the velocity vector and want to know the angular distance, the proper |distance| is between π/2 and π/4, but the answer might appear to be 5π/4 to 6π/4. This occurs because sine and cosine functions are a waveform that stretch from negative infinity to positive infinity and that wave form -2π = 0 = 2π = 4π = 6π. But when you are trying to set a course you want to use the set of values that give the lowest difference. So in the case above I can raise the ideal prograde direction vector or lower the observed vector by 2π, but not both, and recalculate. But if you add the correction and it is not needed you will over correct. So in the above code snippet I first test to see if the ideal vector is in a 'trouble' spot and see if the actual vector is a reciprocal 'trouble' spot, If all these conditions are met,  then perform the alteration of the ideal vector and calculate, otherwise just calculate the distance. 

This is not a general correction one wants to make, unfortunately, it needs to be done in the confines of each calculation. This is a known issue so I am not describing something new.  So if you miss making this correction in a single calculation you may end up making at >370' turn when you only wanted to make a 10' turn. A similar situation occurs in Aircraft, if you are flying along a course and you are told make a right turn [of say 270]. If you throw the heading into the autopilot quickly it will turn left by 90 degrees and some ATC is going to be upset as hell at you. If your flight computer had ears it would know to make a course correction of 135 degrees and another one of 135 before the first solution is achieved.

One problem I should note is that different programming languages use different internal function calculations, and this is a platform problem more than a software problem, so if a computer is capable of producing negative direction vectors from basic trignometry functions that will affect how the answer is provided. Therefore the platform needs to be tested in a wide variety of circumstances to see how it performs.

Edited December 2, 2017 by PB666

[NSEP]

20 hours ago, PB666 said:

"

Instead, the computer decided that the spacecraft had been 360 degrees off target and dutifully commanded its thrusters to fire to turn it around to the required zero-degree position.
"

lol, do I need a job, I just finished programming that exact issue. If query is between 0 and π/2 and subject is between 3π/2 then add 2π to query and calculate else just calculate.

=IF(AND(RC[-15]<=0, RC[-15]<=(PI()/2),RC[-1]>=(3*PI()/2)), ABS(RC[-15]+2*PI()-RC[-1]),ABS(RC[-1]-RC[-15])) [This is in Excel cause my visual studio license has expired]

That particular equation was to determine if the space craft was traveling prograde by looking at dx, dy per dt (dz had to be projected to equator). First it had to determine the radial direction of velocity vector, then compare it with an ideal prograde vector for that position, if it was closer to the ideal prograde vector, then it was considered to be prograde (and cartesian coordinate mapping then used prograde assumptions), otherwise is was considered to be retrograde. From that point on the circular velocity (in x,y,z coordinate system) and radial velocity can be extracted.

A smarter and simpler solution is just to make a course change at Northern geographic pole, since they were spitting distance from it, and then make a correction turn at the equator after they were nearly in orbit. Making a course change at the equator 'greatly' simplifies the math and course change.

EDIT: I should point this out for the general audience.

If you trigger a flight navigation system with no other information other than the desired vectors, then presumably that flight system gathers information from satellites and other spacecraft systems, and then goes through an initiation by which it can then use Markov Chain analysis or the like. But the problem lies in a couple of facts.

That traveling close to a zero point you either need to convert to 2π from zero or zero from 2π but for velocities and accelerations its much more complicated. The vectors for prograde and retrograde are +/- π/2. For instance if your position is radius Px  = 4525 km, y = 4525km from earths center and the prograde direction vector is such that  Arcsin(pY) = ArcCos(pX) [There are ussually two solutions for arcsin and arccos but only one solution that they match] is +/- π/2. The angle for positiion direction is π/4 in this exemplary coordinate system, and the ideal prograde circular velocity has a direction vector of 7π/4. Lets say that you add a radial or normal vector that bring a rotation of more than π/4 to the velocity vector and want to know the angular distance, the proper |distance| is between π/2 and π/4, but the answer might appear to be 5π/4 to 6π/4. This occurs because sine and cosine functions are a waveform that stretch from negative infinity to positive infinity and that wave form -2π = 0 = 2π = 4π = 6π. But when you are trying to set a course you want to use the set of values that give the lowest difference. So in the case above I can raise the ideal prograde direction vector or lower the observed vector by 2π, but not both, and recalculate. But if you add the correction and it is not needed you will over correct. So in the above code snippet I first test to see if the ideal vector is in a 'trouble' spot and see if the actual vector is a reciprocal 'trouble' spot, If all these conditions are met,  then perform the alteration of the ideal vector and calculate, otherwise just calculate the distance. 

This is not a general correction one wants to make, unfortunately, it needs to be done in the confines of each calculation. This is a known issue so I am not describing something new.  So if you miss making this correction in a single calculation you may end up making at >370' turn when you only wanted to make a 10' turn. A similar situation occurs in Aircraft, if you are flying along a course and you are told make a right turn [of say 270]. If you throw the heading into the autopilot quickly it will turn left by 90 degrees and some ATC is going to be liquided as hell at you. If your flight computer had ears it would know to make a course correction of 135 degrees and another one of 135 before the first solution is achieved.

One problem I should note is that different programming languages use different internal function calculations, and this is a platform problem more than a software problem, so if a computer is capable of producing negative direction vectors from basic trignometry functions that will affect how the answer is provided. Therefore the platform needs to be tested in a wide variety of circumstances to see how it performs.

You're hired.

[YNM]

I guess whoever handles the launches is really sloppy then.

 

Do they have a really bad budget cut or what ?

[wumpus]

4 hours ago, YNM said:

I guess whoever handles the launches is really sloppy then.

Do they have a really bad budget cut or what ?

I'm guessing the bug was always there and you don't touch code that works.  This becomes a bit of a problem when you change other bits but never check your "working" code (Ariane 5 is the classic example).

[kerbiloid]

Frigate was tacking.

[monophonic]

This actually does sound like a plausible root cause. Still wicked, though. 

https://arstechnica.com/science/2017/11/this-weeks-failed-russian-rocket-had-a-pretty-bad-programming-error/

Gorram thing now won't load the preview when posting from browser! In short: Programmed for Baikonur, launched from Vostochny.

[DerekL1963]

2 hours ago, wumpus said:

I'm guessing the bug was always there and you don't touch code that works.  This becomes a bit of a problem when you change other bits but never check your "working" code (Ariane 5 is the classic example).

Not even close - the loss wasn't caused by a bug.  The loss was caused by a configuration error.

[PB666]

1 hour ago, monophonic said:

This actually does sound like a plausible root cause. Still wicked, though. 

https://arstechnica.com/science/2017/11/this-weeks-failed-russian-rocket-had-a-pretty-bad-programming-error/

Gorram thing now won't load the preview when posting from browser! In short: Programmed for Baikonur, launched from Vostochny.

It should not be the case, polar correction means that everything is changed to V_{tan or} V_rad . No matter where you are at  or what system you used you will always need to roughly gather your V_tan and V_rad .

If Earth is the origin of the coordinate system  simple calculation. r_t0 = SQRT(x + y + z), r_t1 = SQRT(x + y + z) then r_to,t1' =   r_t1 - r_t0 / (t₁-t₀)

Lets say your Z origin is the equator, X origin is GMT, and Y is at 90' rotation. The latitude component is distance from the plane XY, knowing the radius at that latitude is fairly easy to calculate Z. The longitude in degrees is fairly easily converted to X and Y. X = cosine (latitude) and Y is sine(latitude). GPS gives information in x,y,z but I assume the Russians used their own satellite or ground based navigation systems.

Once you have r' then calculating V_tan , (w*r, the circumgeode component of velocity) = SQRT((x_t1-x_t0)^2 + (y_t1-y_t0)^2 +(z_t1-z_t0)^2- r_to,t1'^2). A tenth grader can do this geometry. Its basically application of the Pythagorean theorem. If the second stage was seeded it would have been noticing a problem with vectors all along, if it was a initiation problem then it was trying to discover its desired vectors.

The next is more complex. But it goes like this. Given mu = 4.98 x 10¹⁴, then  if you know r and Speed(SQRT((x_t1-x_t0)^2 + (y_t1-y_t0)^2 +(z_t1-z_t0)^2) then you can calculate a=SMA (vis visa equation) and also A'=V_tan*r/2. If you know A' and a you can calculate the period(P), if you know the P and A' then A = P*A'. If you know A and a then you know also b. b = A/(π*a). If you know b and a then you know e = b²/a. If you know e then you know  Pe and Apo, l and d. If you know these you can calculate the inclination node between the orbit you are in and the orbit you want. From that you take the mean anomoly of the distance and calculated the true anomoly. Once you have this you simple calculated the time to burn divide by two and burn t/2 before reaching the true anomoly of the inclination node. 

I think the news article is trying to simplify things. Assuming that the PL-stage were pointing prograde then its likely it quickly determined Vrad.

This problem is all about system initialization, if your apriori's are immediate then any corrections that need to be made are relatively simple. For example if you cross the a system boundary going down degrees then you know to convert almost zero to almost 2π or vice versa.

The computer program tried to initialize itself base on the apriori's that it had, then it tried to determine the course that it wanted to intercept. So what can happen, this is where I guess, the computer in the stage starts gathering information about its positions and velocities, then it realizes its more off course than its suppose to be and immediately initiates a burn, unfortunately the all-possibilities calculation is unfit for the error-level. Next the gimble on the rocket is too slow, so basically instead of turning say left, it tries to turn right as it is turning is it burning most of the time retrograde. Burning retrograde cause the centrepal acceleration to fall, which means that apparent gravity is higher and the space craft radial velocity begins to decrease and go negative. So by the time it has reached its destination it has now posed to enter the atmosphere and because its speed is much higher than when it left the atmosphere, potentially it is also turned sideways against the direction of motion, parts of the spacecraft disintegrate. Failure ensues. 

I should also point out that as it turns in the wrong direction, the amount that it needs to turn is increasing from the original prograde so that by the time it reaches where it should have been its now burning at an angle so increased from prograde that the amount of thrust required to stay above safe altitude is probably too low to save the orbit. 

 

 

Edited December 4, 2017 by PB666

[DDE]

1 hour ago, DerekL1963 said:

Not even close - the loss wasn't caused by a bug.  The loss was caused by a configuration error.

Right now we have both versions up and running. Anatoly Zak claims bug, Russian media claims configuration error. Commission's to work 'till Dec 15. What's worse, Dmitry sloped shoulders Medvedev is on a high horse and demands someone to assign blame to, which isn't doing a good service to objectivity.

[YNM]

6 hours ago, wumpus said:

I'm guessing the bug was always there and you don't touch code that works.  This becomes a bit of a problem when you change other bits but never check your "working" code (Ariane 5 is the classic example).

Well, surely you should always check things before running them differently ?

[PB666]

45 minutes ago, YNM said:

Well, surely you should always check things before running them differently ?

The first bug was actually a bug. I think that the correct statement is that its error checking subroutines were not up to the task.

[IncongruousGoat]

And another one bytes the dust...

One of these days, I need to compile a list of rocket/spacecraft failures that occurred as a result of the flight computer freaking out.

[cratercracker]

On 04.12.2017 at 2:04 PM, YNM said:

I guess whoever handles the launches is really sloppy then.

 

Do they have a really bad budget cut or what ?

Russia itself is a really big budget cut.

 

[DDE]

Official conclusions and designated scapegoats to be announced today.

https://www.rbc.ru/society/11/12/2017/5a2ea1639a79477a54146240

[DDE]

https://ria.ru/science/20171212/1510707313.html?referrer_block=index_main_3

Quote

Мы столкнулись даже не с тем, что перевернули датчики угловых скоростей, и не с тем, что перепутали припой при сварке, и даже не халатностью и отсутствием дисциплины на производстве, а с особенностями алгоритма программного обеспечения, который составлялся порядка 20 лет назад и успешно применялся 62 раза при работе разгонного блока "Фрегат" с других космодромов.

Quote

We encountered not just an inverted accelerometer or incorrect weld flux material, or poor worker discipline, but the peculiarities of an algorithm programmed roughly 20 tears ago and successfully used in 62 Fregat launches from other spaceports.

 

[PB666]

5 hours ago, DDE said:

 

Whoopsie. Algorhythms are like fine wine, you never truly know how well (or badly) it has aged until you finished the bottle. Inverted accelerometer seems like a bad thing, no .. . . is that just a cya excuse the fact that no-one wanted to put up rubels for another programmer on staff?

i.e. you will know the next day.

Edited December 12, 2017 by PB666

[DDE]

58 minutes ago, PB666 said:

Inverted accelerometer seems like a bad thing

Very.

58 minutes ago, PB666 said:

is that just a cya excuse the fact that no-one wanted to put up rubels for another programmer on staff?

I don’t think so. It’s a testament to the excessive “if it ain’t broke, don’t fix it” attitude in attitude control and the program in general.

The extra programmer wouldn’t have touched that code either.