Ticket Discussion

« Main Hub


Report #: MNA-427127853428
Created: 2112/06/02 14:03:27
Access Restrictions: (none)
System FQN: scp.simulacrum.vhh.cons3.0918f5a3
Monitor #: MN-563884
Monitor Type: Threshold Value (≥)
Correlated Reports: (+) 3137 items hidden
Comments:

Description automatically generated.
Please report inaccuracies by using <MON DESCRIBE ERR>!

Monitor for dropped BR_WAVEFORM3 packets has exceeded threshold.
Precondition: ACTIVE_USER > 10000
Monitor: BR_WAVEFORM3[drop] > ACTIVE_USER_BR_TRAFFICNX / 700
Current value: 4,304,059
Acceptable value: 3,120,000
Alarm began: 2112/06/02 13:57:02
Alarm end: N/A
2112/06/02 14:03:41 CORRELATION.aic
2,350 similar monitor alarms have been identified. Monitor alarm may represent large-scale temporary outage. Impact unknown.

2112/06/02 14:39:10 CORRELATION.aic
491 similar monitor alarms have been identified, and 81 existing correlated monitor alarms have quieted. Issue seems to be not resolving. Monitor alarm may represent outage. Provisional impact report developed, see 14 linked items.

2112/06/02 14:57:32 COMMAND.aic
Created RAISA ticket 4TYHB2M. This and 1,986 monitor alarm reports have been attached. Projected severity is MODERATE with flag INFRA_CONCERN.

2112/06/02 17:39:58 CORRELATION.aic
117 similar monitor alarms have been identified, and 2,650 monitor alarms have quieted. Issue seems to be resolving.

Ticket #: 4TYHB2M
Creator: COMMAND.aic
Owner: cmeade
Assignee: smehari
System: SIMULACRUM Verge Networking

Description: (+) collapsed

Comments:

2112/06/02 15:48:10 COMMAND.aic
Issue created. Correlated monitor alarms have raised concerns about an outage. Please fill out severity and impact fields as the incident progresses. If there are correlation issues, run <MON CORR CHECK 4TYHB2M> for cleanup.

2112/06/02 15:48:28 RAISA.sys
Rerouting to Site-119 engineering.

2112/06/02 16:22:35 jjones
The failures seem to be correlated along the eastern Indian Ocean pipe. Potentially a network failure, though this doesn't explain BR_WAVELAPSE successes.
Paging @smehari, could your change to reencoding be responsible?

2112/06/02 16:39:02 smehari
Looking.

2112/06/02 16:51:40 smehari
Issue looks like jxpardo edge case with the newer brainscan hardware. Easy fix, rolling out change. Progress: <DEPLOY 77ae54c8c1>

2112/06/02 17:00:05 cmeade
We can avoid haptic issues in the future by marking WZ_HOURFL and WZ_EXCL packets as M_DISCARD. I have a proposed test case to avoid this case in the future, even when they add new stuff.

2112/06/02 17:15:54 smehari
I don't think that's the right solution, afaik there's mental poisoning ramifications using those. We need to consult with sim.psych anyways, but FYI I've made ticket VZ8H9U4 for discussing improvements to jxpardo's flow system. Your tests look fine though.

2112/06/02 17:40:16 COMMAND.aic
CORRELATION.aic reports monitor alarms are resolving in coordination with <DEPLOY 77ae54c8c1>. Changing automated task status.
@smehari, Please attach associated fixes to ticket for <CODERISK 4TYHB2M> analysis.

2112/06/02 16:39:02 smehari
Done.

Unless otherwise stated, the content of this page is licensed under Creative Commons Attribution-ShareAlike 3.0 License