Main character

I want to clarify something that you hinted at in your post but I’ve seen in other posts too. This isn’t a cloud failure or remotely related to it, but a facet of a company’s security software suite causing crippling issues.

I apologize ahead of time, when I started typing this I didn’t think it would be this long. This is pretty important to me and I feel like this can help clarify a lot of misinformation about how IT and software works in an enterprise.

Crowdstrike is an EDR, or Endpoint Detection and Response software. Basically a fancy antivirus that isn’t file signature based but action monitoring based. Like all AVs, it receives regular definition updates around once an hour to anticipate possible threat actors using zero-day exploits. This is the part that failed, the hourly update channel pushed a bad update. Some computers escaped unscathed because they checked in either right before the bad update was pushed or right after it was pulled.
Another facet of AVs is how they work depends on monitoring every part of a computer. This requires specific drivers to integrate into the core OS, which were updated to accompany the definition update. Anything that integrates that closely can cause issues if it isn’t made right.

Before this incident, Crowdstrike was regarded as the best in its class of EDR software. This isn’t something companies would swap to willy nilly just because they feel like it. The scale of implementing a new security software for all systems in an org is a huge undertaking, one that I’ve been a part of several times. It sucks to not only rip out the old software but also integrate the new software and make sure it doesn’t mess up other parts of the server. Basically companies wouldn’t use CS unless they are too lazy to change away, or they think it’s really that good.
EDR software plays a huge role in securing a company’s systems. Companies need this tech for security but also because they risk failing critical audits or can’t qualify for cybersecurity insurance. Any similar software could have issues - Cylance, Palo Alto Cortex XDR, Trend Micro are all very strong players in the field too and are just as prone to having issues.
And it’s not just the EDR software that could cause issues, but lots of other tech. Anything that does regular definition or software updating can’t or shouldn’t be monitored because of the frequency or urgency of each update would be impractical to filter by an enterprise. Firewalls come to mind, but there could be a lot of systems at risk of failing due to a bad update. Of course, it should fall on the enterprise to provide the manpower to do this, but this is highly unlikely when most IT teams are already skeleton crews and subject to heavy budget cuts.

So with all that, you might ask “how is this mitigated?” It’s a very good question. The most obvious solution “don’t use one software on all systems” is more complicated and expensive than you think. Imagine bug testing your software for two separate web servers - one uses Crowdstrike, Tenable, Apache, Python, and Node.js, and the other uses TrendMicro, Qualys, nginx, PHP, and Rust. The amount of time wasted on replicating behavior would be astronomical, not to mention unlikely to have feature parity. At what point do you define the line of “having redundant tech stacks” to be too burdensome? That’s the risk a lot of companies take on when choosing a vendor.
On a more relatable scale, imagine you work at a company and desktop email clients are the most important part of your job. One half of the team uses Microsoft Office and the other half uses Mozilla Thunderbird. Neither software has feature parity with the other, and one will naturally be superior over the other. But because the org is afraid of everyone getting locked out of emails, you happen to be using “the bad” software. Not a very good experience for your team, even if it is overall more reliable.

A better solution is improved BCDR (business continuity disaster recovery) processes, most notably backup and restore testing. For my personal role in this incident, I only have a handful of servers affected by this crisis for which I am very grateful. I was able to recover 6 out of 7 affected servers, but the last is proving to be a little trickier. The best solution would be to restore this server to a former state and continue on, but in my haste to set up the env, I neglected to configure snapshotting and other backup processes. It won’t be the end of the world to recreate this server, but this could be even worse if this server had any critical software on it. I do plan on using this event to review all systems I have a hand in to assess redundancy in each facet - cloud, region, network, instance, and software level.
Laptops are trickier to fix because of how distributed they are by nature. However, they can still be improved by having regular backups taken of a user’s files and testing that Bitlocker is properly configured and curated.

All that said, I’m far from an expert on this, just an IT admin trying to do what I can with company resources. Here’s hoping Crowdstrike and other companies greatly improve their QA testing, and IT departments finally get the tooling approved to improve their backup and recovery strategies.

If it’s any consolation, this is the first issue of its kind in the multiple years we’ve been using CS. Still unacceptable, but historically the program has been stable and effective for us. Hopefully this reminds higher ups the importance of proper testing before releases

This occurred overnight around 5am UTC/1am EDT. CS checks in once an hour, so some machines escaped the bad update. If your machines were totally off overnight, consider yourself lucky

The subtitle of the article says it’s not available in the US -

PC Manager app is only available in some regions, but could come to the US eventually

WHAT
When did it die? That’s so sad

Cam cam with the cancan song took me by surprise

Mine is already talking about this news in a negative light. Makes my life easier to bring in opentofu

It does need other iPhones nearby that have internet connection. We got a handful to test for family during our trips last November even though we both use Android. They didn’t report in when we were away from other people, but kept location decently when in crowded places like the airport. Android has ways to detect when they are following you, but don’t participate in reporting metrics to the source (maybe that’ll change with upcoming Find My Device features in Android 15)

Most formal cleaning companies will do a deep cleaning to form a clean baseline, or let you add on extras per session to address when you need. Usually it’s hourly based with a minimum time requirement. If you don’t need them to do something, it might work out to be cheaper if doing it would have put you over the minimum time.
If you find a self employed cleaning person, they are more likely to be more flexible and able to handle the odd tasks like tidying up or handling laundry. Maybe even help organize or do other routine tasks that you don’t have the bandwidth for.

I have been addicted to making ramen eggs (ajitama) for lunches lately. I eat one or two eggs over rice with some furikake or toasted seaweed and that’s all I need to power through the day. You could pair the dish with more veggies or a miso soup if you’re feeling fancy. The nice part is making half a dozen eggs squares me away for the week, so I hardly have to think about what to do.

Another dish I like is Korean steamed eggs (gyeranjjim). It takes not even 10 mins to cook on the stove. Making rice takes longer, and you can make a lot of rice to reheat later in the week. I would cook the eggs fresh each day though , I’m not sure how reheating them would go. The broth that goes with the eggs keeps me fuller than I ever expect.

Baba ghanoush is so tasty when you make it yourself. This requires more effort up front to roast the eggplant, but the dip is good all week. I eat it with carrots, cucumber, cauliflower, and some pita chips.

Regular tuna salad or this chickpea “tuna” salad is always easy to whip up. I always have celery, pickles, and bread on hand so if I’m feeling up for it, I crack open a can of tuna or chickpeas for an easy lunch.

I saw Windows subsystem and panicked thinking it was the Linux version. Did not know an Android variant existed, that’s a bummer

If there’s an option on the AP to not permit link local routing within a vlan/ssid, that will force all traffic up to the firewall. Then you can block intrazone traffic at the firewall level for that vlan.
I’ve seen this in Meraki hardware where it’s referred to as “client isolation”. Ubiquiti might be able to do this too.

deleted by creator

In shorter terms to what the other comment said, your website won’t work in networks that use DNS served by your DC. The website is fine on the Internet, but less so at home or at an office/on a VPN if you’re an enterprise.
“I can’t go to example.com on the VPN!” was a semi common ticket at my last company 🙃

Oh yes, I’m very glad that he did this video and attempted to make singles himself. My first sentence was a jab at how ungracefully he cooked the cheese haha

What happens when a chemist tries to cook.
Sodium citrate is pretty easy to find online and can go in all kinds of cheese sauces. You can even use it to make a stovetop mac and cheese with your favorite cheeses that won’t get goopy or oily. The other chemical Nile used isn’t really needed, it’s probably more so it holds together into the sheet shape for mimicking singles.
Another alternative to getting sodium citrate is to add American cheese to your cheese blends. Not singles, but deli counter/block cheese. It’s enough to smooth out sauces made from other cheeses.

Right, and the data in Texas contains your address and name attached to it. It’s all public, you can’t avoid it. I don’t know about other states but I assume it’s the same

If you’re paying property taxes, you’re going to be in a database. At least here in Texas, all addresses (home, business, empty land) are in the county’s appraisal and tax database that’s publicly searchable.

Firefox Mobile can open PDFs fine - https://lemmy.blahaj.zone/pictrs/image/1c51814a-c761-4325-baa1-346e8ae2e7fb.png