Thread regarding AT&T layoffs

Outage: I'm not Carl, but could be

As someone with extensive IT experience, I can almost guarantee the AT&T outage yesterday was over some certificate expiring somewhere and nobody knowing how to regenerate it because Carl got laid off seven years ago and the only machine with the keys decommissioned in 2019. –@sisterinferior

Absolutely the most likely scenario. I am not Carl but could be. In my case there were reports considered critical—prompting executive escalation if not delivered—that landed in my lap because the team that created and maintained them were let go.

Then I myself was let go, and I left behind the code for the reports to consultants halfway around the world. Good luck with that, fellas.

Apparently these reports would no longer be valid if kicked off from a home office. This would now require driving three hours into a Dallas office. I chose not to live that designation or "follow that work." (But thanks to being outside a 50 mile radius, the rules said it was "involuntary separation" and I got me some severance.)

After I left I'm sure there were some veepees running around with hair on fire because The Report had not been put before them on time. Womp womp.

For the Big Outage I can just hear the Teams call: "Carl used to handle that" followed by silence. Womp³. And I'm nearly certain it was on a bridge; the key people needed to fix this were unlikely to standing around the same watercooler.

by
| 2216 views | | 17 replies (last ) | Reply
Post ID: @OP+1rekvS6y

17 replies (most recent on top)

Bottom line : empty suits and clowns run this company. Treat everyone like a cog in a machine and few care about the machine. Only a matter of time before some cogs don’t work or are missing.

by
| | Reply
Post ID: @6jpd+1rekvS6y

"We don't need no stinkin' test box."

I'm old enough to remember when we had test box/ test servers. I don't know that we do anymore, or at least my group doesn't. My favorite is when someone drops an update and then wants to argue with you that it didn't break a bunch of sh-t when it so obviously did.

by
| | Reply
Post ID: @2fml+1rekvS6y

@2wfm+1rekvS6y - I think you hit the nail on the head with your comment. I'm almost 60 y/o...32 years at SWBT/SBC/ATT. So us new "old" folks might have not liked the OPs written by the generations before us but they did serve a purpose to help prevent a recurrence of an outage/situation. Yeah I liked to throw things into production "fixing a problem" - thankfully never caused a larger one...but easily could have since "I know what I'm doing". The cuts at T continue to erode the knowledge of systems and how A connects to B connects to "n" systems. It matters. It's not about young vs. old...it's about experience and some of that experience comes from OPs, learning from those older that you and then you passing on your lessons learned to newer employees.

by
| | Reply
Post ID: @2pxa+1rekvS6y

" Telco lesson 1: don't skimp on your Labs investment to vet procedures/MOPs."

Tell me you're 60 + years old without saying it.

Those days are long gone (as are you...I'm guessing)...

And I am not saying it's a good thing, just saying what it is.

by
| | Reply
Post ID: @2wfm+1rekvS6y

Moral of the story: if you cannot afford to break it, don't drop the ball.

The "let things break" approach only works when it doesn't cause the entire business to grind to a halt. It also doesn't work when the entire team gets laid off.

by
| | Reply
Post ID: @2zoc+1rekvS6y

if you aint breaking sh-t then you aint working.

by
| | Reply
Post ID: @2ins+1rekvS6y

“ We always hear that everything is so easy that a monkey can do it but here we are!”

T has breakage and outage in the outsource math, even with it, they are millions ahead after paying high overhead badged employees. It’s days after the event and no one in the US cares.

by
| | Reply
Post ID: @2oue+1rekvS6y

my guess was uam

by
| | Reply
Post ID: @1kol+1rekvS6y

Telco lesson 1: don't skimp on your Labs investment to vet procedures/MOPs.

Telco lesson 2: if the maintenance has the potential of an FCC reportable event, see lesson 1.

by
| | Reply
Post ID: @1hke+1rekvS6y

If I had to guess, an attempt to cut in a new 5GC service, possibly UDM, but went poorly. So the network was told that all devices did not have service - except for what is mandated whether a subscriber has an active account or not (emergency calling)

by
| | Reply
Post ID: @1ifa+1rekvS6y

The outage was due to the accounting guy responsible for paying the electricity bill got laid off. Couldn't find anyone who knew where the bill was.

by
| | Reply
Post ID: @1krv+1rekvS6y

@vsk+1rekvS6y

We always hear that everything is so easy that a monkey can do it but here we are! The monkeys must of taken a banana break during the maintenance window. It’s just started, lose your knowledge base and see how that works out for you.

by
| | Reply
Post ID: @vwy+1rekvS6y

I laugh because everyone thinks they’re so valuable and irreplaceable. lol. Trained monkey can do it.

by
| | Reply
Post ID: @vsk+1rekvS6y

OP here. No bubble here, I saw this kind of thing firsthand across different parts of the business, over and over.

A failed "maintenance window activity" is different than a missing report that affects quarterly revenue, but both are bad.

For business critical tasks, RTO has pi---d away institutional knowledge in favor of improved collaboration and culture preservation of a vast stake in commercial real estate values and tax incentives.

The point is not that a certificate really expired, but rather that same kind of mistake is more likely in today's T. The company was unprepared for the maintenance activity. We all make mistakes, and vendor-supplied products go into production with defects. Maybe this was going to happen anyway, or just maybe the right person for the place and time was the wrong person hit with RTO.

by
| | Reply
Post ID: @iwz+1rekvS6y

Retired from ATT and yes I had a job that people had no clue what I actually did. Half of my Team got laid off a few years before my retirement. "We don't need those people because the upper management didn't even have a clue what they did" LOL

by
| | Reply
Post ID: @acc+1rekvS6y

Sorry to burst your bubble but I was deeply involved and assure you, you are wrong.

The company was surprisingly transparent with what took place and their updates to the public. They know the FCC and FBI are investigating and it would be a PR nightmare they don’t need right now to fabricate some fake story.

Too many people were directly involved to hide what took place. It was due to maintenance window activity. Service began restore when they backed out the change. Restoring 100+ million customers back onto the network is why is took so long.

by
| | Reply
Post ID: @pmy+1rekvS6y

https://x.com/sisterinferior/status/1761040517129703729?s=20

  1. 1M views
by
| | Reply
Post ID: @kib+1rekvS6y

Post a reply

: