· Föderation EN Fr 19.07.2024 12:52:15 I know I'm supposed to feel sorry for Crowdstrike rank and file. But I mostly feel sorry for their victims/customers. I suppose I can also feel sorry for anyone who didn't understand what kind of company they went to work for, but they should be trying to get a job elsewhere by now. |
Föderation EN Fr 19.07.2024 13:06:37 At some point we need to start investigating the body count and ruined lives linked to these outages. 911 service is down in some places in America. CrowdStrike is killing people today, but they will not have to answer for it. There's not criminal negligence if you're a technology company, and there needs to be. |
Föderation EN Fr 19.07.2024 13:16:06 @quinn There is certainly something to be said about neglicence in technology, but I would not exclude the infrastructure organisations that created knowingly a single point of failure and handed the trigger to an external entity. |
Föderation EN Fr 19.07.2024 15:47:05 @gmassen i think you're assuming a level of transparency that isn't there, and is in fact constrained by licensing. (if i'm understanding your argument here) |
Föderation EN Fr 19.07.2024 16:14:29 @quinn Not entirely sure if we get each other. My point was that infrastructures (~services with broad impact) that allow an external provider to break their services are as negligent as that provider could be. Not knowing that it can happen is not an excuse - as infrastructure you have to understand your dependencies (to a certain depth). |
Föderation EN Fr 19.07.2024 17:53:35 @gmassen @quinn we know that power can go out, so we have fallback to UPS. |
Föderation EN Fr 19.07.2024 17:59:30 @gmassen the current legal regime in most countries simply does not allow for this. you can't dig into what your providers are doing without breaking contracts or laws. |
Föderation EN Fr 19.07.2024 20:06:21 @quinn Understood. I’d argue that infrastructure should not depend on external systems or unchecked updates that can break them. Of course that disqualifies a lot of shiny products from glossy brochures, and that might be a hard choice. The further away you get from core infra, the more tolerable such failure modes become. |
Föderation EN Fr 19.07.2024 21:08:33 @gmassen @quinn Where I work, we endure an audit every year, for some certification I don't remember the name/number of. But basically, we have to prove that we have documented procedures in place, & that we have safeguards in place to force us to follow said procedures. What steps need to be followed to get code from Dev to QA to Prod. Our customers don't see our audits, but they know what our certification means. |
Föderation EN Sa 20.07.2024 16:10:15 @gmassen the thing is those disqualified shiny projects are legally and contractually required for compliance regulations, while not being legally open to customers. |
Föderation EN Sa 20.07.2024 06:08:31 |
Föderation EN Sa 20.07.2024 16:12:10 @ocdtrekkie @gmassen this is a tough call, because on the whole, outsourcing compliance has probably helped make many industries without access to adequate tech staff safer than they would have been. it's truly a mess out here and there's no fixes. like, not no easy fixes, but for a lot of industries, no fixes. just ask school districts, for an example. |
Föderation EN Sa 20.07.2024 18:53:42 @quinn @gmassen I definitely understand why it happened, IT staff have been minimized, disrespected, and ignored for years, prevented from budgeting critical needs, etc. Turning it all into a subscription means someone else raises the prices, someone else staffs the problems, etc. But centralizing it on a bunch of companies heavily motivated to raise their own profits instead of protecting yours is going to lead to a lot of ruin. |
Föderation EN So 21.07.2024 08:07:25 @quinn @ocdtrekkie In quite a few cases I’d suspect the outsourcing has become way more expensive than decent IT staff. The way back has become near impossible. And compliance is part of it. But I still hold infrastructure to a different standard than a school district, in terms of understanding and being in control of the core systems. It is also the only way I can see to prevent large-scale failures: limiting impact of concentration. |
Föderation EN Fr 19.07.2024 13:33:52 @quinn The negligence is on the part of whoever thought using cloud products for this was a good idea. Anyone can offer any shitty product, but you're not forced to purchase and use it for critical services, are you? |
Föderation EN Fr 19.07.2024 13:49:01 @raucao @quinn About the "not forced" thing, they kind of are. 'HAE-NCF-1: There should be a unified endpoint management (UEM) [48] solution in place to Not any specific brand or solution, but to be honest is there a big worst case scenario difference between Crowdstrike and its competitors? I'm oversimplifying but all of them require more or less a centralized management console with full admin access to endpoints. |
Föderation EN Fr 19.07.2024 16:44:09 |
Föderation EN Fr 19.07.2024 22:17:04 @raucao @makdaam @quinn but this bug would have occurred if you self hosted the same platform and did your due diligence with keeping up to date The failure isn’t that there was a bug. The failure is that there seems to be no circuit breaker mechanism for if a bug occurs and it took them 90 minutes to manually remediate the problem (meaning they probably manually detected the problem) |
Föderation EN Fr 19.07.2024 15:44:28 @raucao this is a bad argument in a lot of ways -- they are often forced by regulation to pick something from a set of bad and obfuscated choices. the cloud products conversation is done; the cloud won, and regulations around the world reflect this. hell, i think most services should be protocols! but i lost, for now. and there isn't the talent to run that world right now if i'm being real about it. |
Föderation EN Fr 19.07.2024 16:43:12 @quinn Can you link some actual regulations to back up those claims? I don't know of any, and I'm sure most jurisdictions don't require specific products at all. |
Föderation EN Fr 19.07.2024 19:49:00 @quinn I was just musing to myself this morning about this very thing. It's sick |
Föderation EN Fr 19.07.2024 22:49:43 criminal liability belongs also to the checkbox compliance that led to 911 services being impacted, tbh. |
Föderation EN Fr 19.07.2024 23:40:30 @quinn I posted a little while ago about the old adage that no one ever got fired for buying Microsoft. There needs to be a serious review about IT monocultures and enterprise reliance on certain vendors, and the "conventional wisdom" that has led to this state. |