System update? Are you serious? This an update that took down the debit card system, Sears card payments, store coupons, layaway system, all returns even with a receipt, inventory look ups, online ordering, and the SYW points system. System updates are usually done overnight, not during the business day. I just watched people leave carts of merchandise at a Sears store over cash only at the register...and several had a $10 offer that could not be used because of the system update (management was instructed not to make good for any coupons or offers that were unaccessible due to the "update.") No one could pay their layaways either. This system update is resulting in millions in lost sales. I guess this is what happens when a company does not have control over their own IT offshore departments that are actually wholly-owned subsidiaries located in India and Israel.
55 replies (most recent on top)
Guys, I am sorry I am confusing everyone with the IT questions. Thanks for your patience.
Thankyou for clearing up what I thought.
FYI based on the article from 1998, the data center is 18+ years old. I think lack of maintence of maintence at the troy data center is taking a toll on the equipment since it has not been properly maintained for 11+ years.
I also think that IT has to keep the data center running though a patchwork of hardware and software upgrades. I bet the hardware in the data center is obsolete and since the IPS was not maintained properly since 2005+ and the hardware is overloaded with new software on older equipment.
I can only hope that you guys in corporate IT can continue the patchwork of software and outdated hardware to get the company to the point that filing Chapter 11, restructing or chapter 7 total liquidation of the company.
At this point from what I have read is filing a Chapter 7 for an orderly liquidation of the entire company. I feel sorry for you guys at the stores honestly and at corporate IT having to do patchwork to keep the systems online.
I hope this crash does not happen again gearing up for Christmas. If a major crash happens hopefully the stores will be up fast, I feel sorry for everyone from corporate IT to the local levels getting screwed and having to play IT tech at store level.
Oh, did the crash effect pharmacy also? Were they able to fill prescriptions or were they taken offline also? As a result of the crash.
(Oh, anyone who works in corporate IT that is dealing with this mess. Is their a way to exchange email address with you guys, I am probably going to go back to a Kmart and want to know the h e l l I will be dealing with IT wise.)
Is the data center still in Troy? I drive by the old Kmart HQ every day and it's empty and forlorn, been that way for 10 years (thanks, Eddie, we appreciate the crumbling eyesore). Unless the data center is in one of the other office buildings in the area, I don't see any evidence of life near the old HQ
Just for the LULZ. http://www.prnewswire.com/news-releases/kmart-announces-development-of-new-data-center-77877857.html
TROY, Mich., May 12, 1998 /PRNewswire/ -- Kmart Corporation (NYSE: KM) today
announced the development of a new, state-of-the-art data center which will
house the retailer's computer systems when it opens in June 2000. The 86,400
square-foot, two-story facility represents a $103 million investment,
including the building, equipment, training and employees. The new data
center will be located behind Kmart's existing headquarters facility in Troy,
Mich.
When completed, the Kmart Data Center will improve the reliability and
productivity of the company's merchandise and operations systems while
accommodating future growth. The Kmart Data Center will be designed to
withstand severe problems such as power blackouts or extreme weather without
interrupting the proper functioning of critical systems and will be connected
by fiber optic cables to the main headquarters.
Registers that freeze or shut down when credit, debit and gift card have been swiped already will have processed those transactions -- taking money from members -- despite coming back online with a void transaction receipt. Be very careful with this.
So registers are still freezing since 8/18 requiring reboot and backing up lines. And doing weird things, and chip readers are being difficult. Reliability is 0.
and guess what? us old-timers that learned how to add, subtract, find percent off and count change back are the ones that have been kicked to the curb. I could do a register job without a computer in my sleep. But, I think I'll just cruise like the teenagers running this place till I can retire next year. Keep your keys in your pocket and batteries in a close flashlight. Be ready to go because it's getting close. Out the door and never look back. We gave it our best. NEVER look back! Just be ready to go.
Offline: cash only, overrides, returns with receipt at the store of purchase (means this data is stored locally), no checks because the check verification system would also be offline.
Receipt returns on 8/18 could only be done for purchases made with cash. If a member paid with credit, debit, gift card or SYW points, the return transaction failed. On prior outages, the data would be transmitted as soon as we came back up for credit and SYW points so those types of returns would go through.
Please tell me if what you can do in offline mode is correct?
Thank you the "half dead register" comment made me laugh. Unfortunately in addition to my posts on here what I know about the crumbling IT equipment and the sad patchwork of hardware and software to keep the store servers and main office running until SHC permentaly goes out of business. It is only a matter of time before the systems fail permentaly and unless Fast Eddie wants to patch the systems enough to do a going out of business sale for the remaining locations.
If someone from corporate IT can chime in please on this.
From what I remember before the chip cards came out if a register was offline from the main corporate you could still process credit cards through the register, unfortunaly from what I have been informed that is not allowed. Due to the issuance of chip cards.
It is sad that when a store is offline that you can either do only cash transactions or a manual check transaction?
If I remember, please excuse me I remember this from 2014 before the chip card upgrade please correct me if I am wrong.
You can process in offline mode the following:
Credit cards(no longer)
Cash
Checks
Overrides
Offline returns
I think that is about it.
You cannot process:
Credit card Apps
Online returns
SYWR
Product lookup
Rain checks
Layaway
Ect.
I need to go fix supper but this is getting so funny to me that I can't quit! Reason: I was thinking about a man and a song and posted a link to it for everyone to enjoy. It seemed relevant. This site won't let me post the man's name because it contains the letters g a y. Let's see if that goes through. The man's first name was Marvin. The song was 'What's going on' Oh, well. It was meant as a break for us all and we'll go back tomorrow and do it all again.
This thread is a fascinating read into Sears Holdings IT problems that boil down to a lack of investment in infrastructure.
When the e-commerce sites are down for hours or all day, and cannot take orders, the excuse is a "system update."
When our bread and butter brick and mortar stores are offline and cannot process sales at a half-dead register, the excuse is a "system update."
JUST GOT THE BIGGEST LAUGH OF MY DAY! Ain't It great? Marvin Is Laughing somewhere beside Our Father. The site just censored the spelling of the name G--e. Ha Ha Ha! Forrest Gump was told that stupid is as stupid does. Eddie L. wasn't as lucky with a mama in the wisdom category as Forrest. s---ing greedy was his god that he worships.
RE; post ' Is it to the point if SHC has another crash the main system is fried since there is no redundant data center to kick in if this one fails?' Pray that it will be tomorrow and puts us out of our misery. In the old days you would at least shoot your horse between the eyes and let it be done. Thanks for all the posts that are over my head but I still get what's going on. (link to anybody who listened to Marvin G--e ...What's Going On) Watch and forget our grief at the K for a few....YouTube Here....AAhh, that's Nice! https://www.youtube.com/watch?v=H-kA3UtBj4M
Does everyone agree? What do you think?
If I am confusing anyone with the IT questions please bear with me. I am finally starting to understand what happened on 8/18/16, and it all makes sense now.
What it boils down to is this. When Lampert fired IBM that ran the pre SHC data centers before the merger with Sears in 2005 the systems were 100% online.
After the merger with Kmart to form SHC, Lampert moved everything to the old Kmart Troy data center that ran Kmart systems. I am just speculating at this point but I think lamperts severe cost cutting that started in 2005/2006 has caused systems to fail very slowly, I think this hardware crash is due to two things making sure the IPS is holding a charge which he did not do, and general failure to do proper hardware maintence.
If SHC properly maintained the IPS and the back office servers,mainframes,ect.
This crash could have been prevented.
Ok. I understand exactly what happened now. When Lampert fired IBM right after he bought Sears in 2005.
I am wondering if the IPS have gotten to the EOL stage and were no longer holding a charge? Due to lack of maintence because of lamoerts severe cost cutting?
I am beginning the crash to be a combination of things, hardware failure, software issues,lack of maintence, ect.
I am thinking from what this poster is explaining is that this could have been prevented if the IPS was maintained properly?
I know your in the software end of SHC at corporate. I am just guessing and tell me if I am wrong, I know this crash was severe from a hardware standpoint, but can SHC keep the data center,mainframes,ect.
Running until chapter 11 is filed next year?
I know your doing a lot of IT patchwork hardware and software wise.
What I want to know if you can find out is the hardware at corporate at the EOL stage? Is it to the point if SHC has another crash the main system is fried since there is no redundant data center to kick in if this one fails?
All I know i that we were told that Troy had a 'power failure' and that it affected SAN disk. However, it did affect Mainframe as well as distributed systems all over the company. Building B floor 6 (VP's) here at corporate were asking for status after status from us asking how fast we could get everything back up and running. I worked all day Friday until midnight and the weekend to get things going. There were dozens and dozens of systems on the list of affected systems. I cannot say if this was due to a lack of maintenance becasue I don't work in hardware support. But, I can tell you a 'power failure' should never happen because most large IT shops should have an 'IPS' backup that kicks in so that if power is cut off you can switch over for at least 3 or 4 hours and if power doesn't come back then you have time to bring systems down gracefully... in this case it wasn't graceful and we had corruped data and lost files. When IBM ran our data centers back in the 90's and early 2000's they had large backup generators for this purpose but then we fired IBM to save money and switched everything to Troy after Kmart bought us out in 2005-06. Somebody has to answer why the IPS did not work.
The way things have been going, they would have better luck hosting everything on godaddy. There are short outages all the time. Some last a half minute, others several minutes. They put the blame on registers or stores, but we know better. And let's not forget the Black Friday outages and all those five hour outages here and there that stores believed were local only to them, while every store had the same outage.
Sorry guys I was thinking out loud in my post and wondering if anyone knew?
I wonder if the servers in Michigan are still using "floppy disc" to archive data? Or I wonder if the servers are older than 15+ years. I do not think they upgraded the main server when they upgraded the store servers.
My estimate unless someone from corporate IT can verify this, the systems are at least 15 to 20years old? For the corporate systems, I know the store systems are from 2001 at the earliest.
Does anyone know how old the centeral servers are in Michigan that power the store systems? Were they upgraded in 2001 like the store systems?
Or are they very old like when Kmart was running multiple POS systems before the upgrade in 2001?
What I am trying to get at is unless the corporate servers have "Emergency" maintence done to keep them running till SHC can declare chapter 11 and close stores to take the overload off the system another one of these crashes will happen.
My best suggestion at store level, is keep your registers clean and vacuum the back panel where the cables plug into it, and with the Server in the Cash office or where ever the POS server is is to make sure to keep the area around it clean and the cables organized. also vacuum the back of the server where the cables plug in to avoid overheating.
Guys I am making these suggestions to try to prolong the systems for now, since they refuse to send a tech out to fix crashed registers or a crashed back office system.
If ANYONE in the store is tech savvy and comptent enough to be able to safely make sure these registers will function till at least the bankruptcy is filed.
You guys will be in good shape.
PROPER MAINTENCE
Your right "system upgrade" sounds much nicer than " F----I-NG D-A-M-M-I-T" the servers crashed and we think our employees are morons.So we just make an excuse as to where we are doing an "upgrade" and our idiot employees will believe that.
So what I am saying is this crash, has pissed off employees and customers, because customers were lied to along with the employees about what really happened.
FYI: if I use $$$ or --- since I want to curse like say words like A$$. That is the only way to be able to say how you really feel without the site filter deleting it.
SHC IT Helpdesk insisted it was a system update when I called them at 2 p.m. CT about our store being offline.
Ultimately, it appears corporate wanted to keep this snafu hush-hush while it was happening.
Most customers, which is what our members are, would have been understanding had we been able to tell them the entire company is having an IT outage.
It is concerning, from a information security and redundancy standpoint, that everything -- multiple databases and "systems" is on servers in one place in Troy, Michigan. It is not 1992 anymore.
The person who made the comment about the "disc failure" how severe was the crash?
Are the back office servers, NOT the store systems but the systems at corporate on their last legs like the registers and hardware are at store level?
What I am asking is, at the corporate level is the lack of proper IT maintence at the main servers causing them to fail?
I know the store systems are on their last legs, is the corporate system on it last legs also?
Crap, sorry I forgot about that. I feel like an idiot. For even suggesting that idea.
With the introduction of chip-reading card terminals, credit cards can no longer be processed manually.
Ok, I thought that is what happened a "major" system crash caused everything, to go offline, what I do not get even if registers go "offline" as in disconnect from the corporate server, why credit cards could not be run? Usually you can run a credit card when the system is offline, you have to call for manual Authorization. I am puzzled you could not even do a manual credit card?
this was not an 'update' ... it was a disk failure that brought everything down... a power failure at the troy data center.... apparrently the IPS backup system did not kick in so all systems crashed hard.
So anyone who is complaining about the failing registers and RMUs read this article, it explains things.
Guys please read the link in the post bellow, the equipment IT wise Kmart is running is from 2001 if not older.
Here is a link explaining the IT equipment Kmart rolled out in 2001, everything the stores have IT wise is from 2001/2002.
http://www.computerworld.com/article/2591284/mobile-wireless/kmart-to-spend--270-million-on-new-technology-for-its-stores.html
I hope this link provides helpful info and explains the IT equipment that stores are running is at least 15years old if not older.
Very funny.
Look out Amazon, here we come !!
Those systems hardware wise at Kmart are on their last legs they are between 14 & 15 years old. They were rolled out pre bankruptcy and before bankruptcy was filed in 2002.
FYI: The RMU's are older than the registers. So if you wonder why the RMU's are failing along with the meridian phone system. None of the IT equipment at Kmart has ever been maintained properly.
SHC is now having problems, maybe I am just guessing, the central servers that control everything, including, SYWR, RMU, ect. Crashed possibly and that caused both Sears and Kmart systems to go offline.
Again, that is only a logical guess, Since the store controller at each store was not compromised the registers we're able to run in "offline" mode.
Usually if the registers are offline you can take a credit card but it will prompt for manual authoration.
Something very serious must have happened in Illinois to bring the entire system down.
Oh God, I am a former employee looking from the outside and I worked at a Kmart. Do not jinx the rest of the stores " with the Black Friday outage"
Why bother upgrading the systems when bankruptcy is imminent and just around the corner?
you think this outage was bad wait for "Systems Down Black Friday!"
I work in a small Kmart in Ohio. We had the blackout everyone else did on Thursday. As of the end of my shift at 5 p.m., there was no communication from corporate as to what the problem was or when it would be resolved. When we called Shop Your Way, they were in the dark also with no access to anything. When we called corporate in Hoffman Estates all we could get was a fast busy signal.
It is beyond absurd that such a large organization can essentially be taken down for an entire day with no explanation or communication to the stores. They could have called each store. Instead we were left to fend for ourselves.
An email was sent to the store managers late this morning with nothing more than a vague explanation, that made it seem like it was not a big deal. Our store receipts were less than a third of what they should have been -- members could not use or get points on purchases and were forced to pay cash.
I wonder how much the stock will tank when the 2nd quarter results come out.
Well here's another likely 1% drop in this quarter's sales between the effects of today and the sales repercussions from disgruntled customers down the road...
My Sears store closes in an hour and they just confirmed everything has been functioning for them for the past half hour.