Ok, so T marketing admitted a misconfigured update broke things, plausible but that's not detailed enough, something of this magnitude has to have more to the story...
25 replies (most recent on top)
When upper management is forced to provide details then we’ll know what happened. Remember when the FAA NOTAM outage caused flight cancellations, it took a couple weeks before the contractors identity was revealed to the flying public.
Sandeep didn’t watch the mandatory cartoons where gremlins explain about procedures, testing, stakeholders, MOPs, etc. He just did the needful cuz Pradnathasamabathir told him to push updates, don’t bother testing or checking.
Connection changes everything. Let's see if DISconnection changes anything. My guess is "no". AT&T execs are too complacent wallowing in the money trough to do anything substantial. They may fire some poor schmuck at the bottom of the hierarchy but don't expect more.
Opus crashed and when they tried to bring it back up, salesforce took over and that's why the spinning wheel of death held up the network. Hold on while we attempt to pull our heads out of.................
" Our network is a part of our national communications infrastructure.
As such, the government should restrict ATT from hiring anyone outside of the US. "
Parts of the network are. You cannot have access to or work on certain internal T networks unless you are a US Citizen living in the US.
Some old guy unplugged the internet. Didn't understand what the hoopla was about.
I heard, Stankey was installing a new sprinkler system and accidently cut a major trans-Atlantic fiber optic line that goes through his house. He called the IQ Bar to come and fix it but they were in India and it took them 11 hours to fly to US to fix it. Sad but true.
Keep using age discrimination to get rid of experienced talented employees and replace them with cheaper non-caring scrubs and voila!!!
I'm all for blaming 3rd world hiring at a fraction of the expense, and forced relicates/RTO, but have we also considered AI writing code that no one checked?
"Prague?
These people do not care, and the latter are former Soviets."
Not Prague, but Brno. 90% of T people there are not local (even the Czech were never Soviets) - many of them are US, UK, west EU etc. expats. It is funny to think that 'pure' US stuff is better as anybody else. Very funny.
Our network is a part of our national communications infrastructure.
As such, the government should restrict ATT from hiring anyone outside of the US.
India? Prague?
These people do not care, and the latter are former Soviets.
Are you F’ing kidding me? Our government is completely clueless!
“Go read the GTOC.”
The id--ts that post on this board can’t even spell GTOC.
FBI doesn't believe the company's story. They are investigating.
Because First Net went down, there will be investigations and the whole story will come out. Company likes to say procedures weren't followed but the truth is probably more in line with that they've cut back the work force so much, that the people who know how to do this stuff don't have time to check every box anymore and still get things done.
Go read the GTOC.
It was Jeremy Legg in the study with the wrench.
T employees tried to reclaim the wrong legacy copper wire.
RTO - I guess does not work... making people unhappy and unsatisfied being forced to move to some State far away from their family, Most are leaving the company, no one cares, way for these employees to get back at "T" for not caring for employee... more of this will occur... "T" will go down in flames becasue of these poor decisions, after losing good quality, caring employees
I was told Solar Flares
We are doing the needful and working the outage.
We will get back with you at the soonest.
- Sandeep “Joe” Gupta
Forced RTO/moves causing experts to leave creating a knowledge vacuum. Offshore operations that don't have a clue WTF they're doing. What should have been a simple update becomes a sloppy code push. Oops, oh well.
Fiber & 5G was supposed to be the crème de la crème for AT&T going forward, but instead they failed at that simple connectivity. Not to mention the consequences of AT&T's FirstNet being impacted by this debacle which is supposed to be a critical fail-safe.
Huge embarrassment for AT&T. Regardless of how they spin this, the meme for 2024's history books will be the big outage.
As usual - incompetent L3+ people 'approve' operation plans which they understand no word. Business customers experience thousands of outages every week as T kept just people able parroting shallow phrases without a basic clue of technologies. We all can provide tons of examples how this company damages own reputation and customers budgets.
Outsourcing to India
They have to bring the network down to install 6G radios before everyone else
There isn’t more to the story. It was a botched maintenance window change. Happens all the time but thankfully not on that scale.
FBI doesn't believe the company's story. They are investigating.