Sysadmin
- How do you track the "mess" you are doing with your system?
Programs with custom services, virtual environments, config files in different locations, programs creating datas in different location...
I know today a lot of stuff runs in docker, but how does a sysadmin remember what has done on its system? Is it all about documenting and keeping your docs updated? Is there any other way?
(Eg. For installing calibre-web I had to create a python venv, the venv is owned by root in
/opt
, but the service starting calibre web in/etc/systemd/system
needs to be executed with theUser=<user>
specifier because calibre web wants to write in a user home directory, at the same time the database folder needs to be owned by www-data because I want to r/w it from nextcloud... So calibreweb is installed as a custom root(?) program, running in a virtual env, can access a folder owned by someone else, but still needs to be executed by another user to store its data there... )Despite my current confusion in understanding if all of this is right in terms of security, syntax and ownership, No fucking way I will remember all this stuff in a week from now.. So... What do you use to do, if you do something? Do you use flowcharts? Simple text documents? Both?
Essentially, how do you keep track?
- How to wget/curl OCI files (docker, github package)
This article will describe how to download an image from a (docker) container registry.
| [!Manual Download of Container Images with wget and curl](https://tech.michaelaltfield.net/2024/09/03/container-download-curl-wget) | |:--:| | Manual Download of Container Images with wget and curl |
Intro
Remember the good `'ol days when you could just download software by visiting a website and click "download"?
Even
apt
andyum
repositories were just simple HTTP servers that you could justcurl
(orwget
) from. Using the package manager was, of course, more secure and convenient -- but you could always just download packages manually, if you wanted.But have you ever tried to
curl
an image from a container registry, such as docker? Well friends, I have tried. And I have the scars to prove it.It was a remarkably complex process that took me weeks to figure-out. Lucky you, this article will break it down.
Examples
Specifically, we'll look at how to download files from two OCI registries.
Terms
First, here's some terminology used by OCI
- OCI - Open Container Initiative
- blob - A "blob" in the OCI spec just means a file
- manifest - A "manifest" in the OCI spec means a list of files
Prerequisites
This guide was written in 2024, and it uses the following software and versions:
- debian 12 (bookworm)
- curl 7.88.1
- OCI Distribution Spec v1.1.0 (which, unintuitively, uses the '/v2/' endpoint)
Of course, you'll need '
curl
' installed. And, to parse json, 'jq
' too.sudo apt-get install curl jq
What is OCI?
OCI stands for Open Container Initiative.
OCI was originally formed in June 2015 for Docker and CoreOS. Today it's a wider, general-purpose (and annoyingly complex) way that many projects host files (that are extremely non-trivial to download).
One does not simply download a file from an OCI-complianet container registry. You must:
- Generate an authentication token for the API
- Make an API call to the registry, requesting to download a JSON "Manifest"
- Parse the JSON Manifest to figure out the hash of the file that you want
- Determine the download URL from the hash
- Download the file (which might actually be many distinct file "layers")
| [!One does not simply download from a container registry](https://tech.michaelaltfield.net/2024/09/03/container-download-curl-wget) | |:--:| | One does not simply download from a container registry |
In order to figure out how to make an API call to the registry, you must first read (and understand) the OCI specs here.
- <https://opencontainers.org/release-notices/overview/>
OCI APIs
OCI maintains three distinct specifications:
- image spec
- runtime spec
- distribution spec
OCI "Distribution Spec" API
To figure out how to download a file from a container registry, we're interested in the "distribution spec". At the time of writing, the latest "distribution spec" can be downloaded here:
- <https://github.com/opencontainers/distribution-spec/releases/tag/v1.1.0>
- <https://github.com/opencontainers/distribution-spec/releases/download/v1.1.0/oci-distribution-spec-v1.1.0.pdf>
The above PDF file defines a set of API endpoints that we can use to query, parse, and then figure out how to download a file from a container registry. The table from the above PDF is copied below:
| ID | Method | API Endpoint | Success | Failure | |------|----------|------------------------------------|--------|-----------| | end-1 |
GET
|/v2/
|200
|404
/401
| | end-2 |GET
/HEAD
|/v2/<name>/blobs/<digest>
|200
|404
| | end-3 |GET
/HEAD
|/v2/<name>/manifests/<reference>
|200
|404
| | end-4a |POST
|/v2/<name>/blobs/uploads/
|202
|404
| | end-4b |POST
|/v2/<name>/blobs/uploads/?digest=<digest>
|201
/202
|404
/400
| | end-5 |PATCH
|/v2/<name>/blobs/uploads/<reference>
|202
|404
/416
| | end-6 |PUT
|/v2/<name>/blobs/uploads/<reference>?digest=<digest>
|201
|404
/400
| | end-7 |PUT
|/v2/<name>/manifests/<reference>
|201
|404
| | end-8a |GET
|/v2/<name>/tags/list
|200
|404
| | end-8b |GET
|/v2/<name>/tags/list?n=<integer>&last=<integer>
|200
|404
| | end-9 |DELETE
|/v2/<name>/manifests/<reference>
|202
|404
/400
/405
| | end-10 |DELETE
|/v2/<name>/blobs/<digest>
|202
|404
/405
| | end-11 |POST
|/v2/<name>/blobs/uploads/?mount=<digest>&from=<other_name>
|201
|404
| | end-12a |GET
|/v2/<name>/referrers/<digest>
|200
|404
/400
| | end-12b |GET
|/v2/<name>/referrers/<digest>?artifactType=<artifactType>
|200
|404
/400
| | end-13 |GET
|/v2/<name>/blobs/uploads/<reference>
|204
|404
|In OCI, files are (cryptically) called "
blobs
". In order to figure out the file that we want to download, we must first reference the list of files (called a "manifest
").The above table shows us how we can download a list of files (manifest) and then download the actual file (blob).
Examples
Let's look at how to download files from a couple different OCI registries:
Docker Hub
To see the full example of downloading images from docker hub, click here
GitHub Packages
To see the full example of downloading files from GitHub Packages, click here.
Why?
I wrote this article because many, many folks have inquired about how to manually download files from OCI registries on the Internet, but their simple queries are usually returned with a barrage of useless counter-questions: why the heck would you want to do that!?!
The answer is varied.
Some people need to get files onto a restricted environment. Either their org doesn't grant them permission to install software on the machine, or the system has firewall-restricted internet access -- or doesn't have internet access at all.
3TOFU
Personally, the reason that I wanted to be able to download files from an OCI registry was for 3TOFU.
| [!Verifying Unsigned Releases with 3TOFU](https://tech.michaelaltfield.net/2024/09/03/container-download-curl-wget) | |:--:| | Verifying Unsigned Releases with 3TOFU |
Unfortunaetly, most apps using OCI registries are extremely insecure. Docker, for example, will happily download malicious images. By default, it doesn't do any authenticity verifications on the payloads it downloaded. Even if you manually enable DCT, there's loads of pending issues with it.
Likewise, the macOS package manager brew has this same problem: it will happily download and install malicious code, because it doesn't use cryptography to verify the authenticity of anything that it downloads. This introduces watering hole vulnerabilities when developers use brew to install dependencies in their CI pipelines.
My solution to this? 3TOFU. And that requires me to be able to download the file (for verification) on three distinct linux VMs using curl or wget.
> ⚠ NOTE: 3TOFU is an approach to harm reduction. > > It is not wise to download and run binaries or code whose authenticity you cannot verify using a cryptographic signature from a key stored offline. However, sometimes we cannot avoid it. If you're going to proceed with running untrusted code, then following a 3TOFU procedure may reduce your risk, but it's better to avoid running unauthenticated code if at all possible.
Registry (ab)use
Container registries were created in 2013 to provide a clever & complex solution to a problem: how to package and serve multiple versions of simplified sources to various consumers spanning multiple operating systems and architectures -- while also packaging them into small, discrete "layers".
However, if your project is just serving simple files, then the only thing gained by uploading them to a complex system like a container registry is headaches. Why do developers do this?
In the case of brew, their free hosing provider (JFrog's Bintray) shutdown in 2021. Brew was already hosting their code on GitHub, so I guess someone looked at "GitHub Packages" and figured it was a good (read: free) replacement.
Many developers using Container Registries don't need the complexity, but -- well -- they're just using it as a free place for their FOSS project to store some files, man.
- Secure Boot is completely broken on 200+ models from 5 big device makersarstechnica.com Secure Boot is completely broken on 200+ models from 5 big device makers
Keys were labeled "DO NOT TRUST." Nearly 500 device models use them anyway.
TLDR: An AMI testkey was used in production by a bunch of manufacturers. The key has now been leaked.
- Global IT issues caused by Crowdstrike update causes BSOD on Windowswww.timesnownews.com Latest Crowdstrike Update Causes Blue Screen Of Death On Microsoft Windows, Multiple Users Affected
Latest Crowdstrike Update Issue: The issue seems widespread, affecting machines running various CrowdStrike sensor versions. CrowdStrike has acknowledged the problem and is currently investigating the cause., Technology & Science News - Times Now
cross-posted from: https://sh.itjust.works/post/22460079
> Today I'm grateful I'm using Linux - Global IT issues caused by Crowdstrike update causes BSOD on Windows > > This isn't a gloat post. In fact, I was completely oblivious to this massive outage until I tried to check my bank balance and it wouldn't log in. > > Apparently Visa Paywave, banks, some TV networks, EFTPOS, etc. have gone down. Flights have had to be cancelled as some airlines systems have also gone down. Gas stations and public transport systems inoperable. As well as numerous Windows systems and Microsoft services affected. (At least according to one of my local MSMs.) > > Seems insane to me that one company's messed up update could cause so much global disruption and so many systems gone down :/ This is exactly why centralisation of services and large corporations gobbling up smaller companies and becoming behemoth services is so dangerous.
- Major IT outage affecting banks, airlines, media outlets across the worldwww.abc.net.au Live: Global IT outage affecting banks, airlines, media outlets across the world
There are reports of IT outages affecting major institutions in Australia and internationally.
cross-posted from: https://lemmy.ml/post/18154572
> All our servers and company laptops went down at pretty much the same time. Laptops have been bootlooping to blue screen of death. It's all very exciting, personally, as someone not responsible for fixing it. > > Apparently caused by a bad CrowdStrike update.
- Slack alternatives?
Hello! I am looking for suggestions for Slack alternatives that meet the following (likely impossible) criteria:
- Modern UI
- Self-hosted FOSS
- Actively developed, or at least stable and maintained
- Comprehensive API for integrations
- Non-shit strategy for determining which device to send notifications to
Regarding UI, I am hoping to find something with a more streamlined implementation of threaded conversations - this is my primary complaint with Slack.
I know there are tons of articles on Slack alternatives, but I'm hoping for a more technical perspective. Are there any Matrix-based options that are refined enough for a small team to rely on as primary method of communication?
Thank you!
- MS Exchange Relay
I am working part time for a small company, they have about 40 employees that use the email everyday for work and recently they have acquired a MS account for 10 employees that use it mainly for teams with customers but also sharepoint, etc.
To buy an MS account for each of the 40 would be too expensive and necessary because the other 30 only really use email in the day to day work.
So what I did initially was to follow this Microsoft doc: https://learn.microsoft.com/en-us/exchange/mail-flow-best-practices/how-to-set-up-a-multifunction-device-or-application-to-send-email-using-microsoft-365-or-office-365
So our
MX
register point to Exchange server and exchange relay it to the secondary email server where all those 30 accounts exists.It was working fine until I we started to get this "Not delivered message" email returning with this error:
Error: 550 5.7.367 Remote server returned not permitted to relay -> 554 5.7.1 : Relay access denied
I talked to the support of this secondary email server and they told me they do not support this operation.
So I am looking for help in finding some server that would allow me to work like this. Do you happen to know some company you could recommend?
- Best practice for using a disk partition inside a VM?
Hi all, I want to setup a fileserver as a KVM which will access a 2TB disk partition to store its data. In order to do this I saw 5 options:
-
Attach the whole disk to the VM and access the partition as you do in the host machine. -> contraindicated by the RHEL documentation for security reasons.
-
Attach only the partition to the VM. Inside the VM, the partition appears as a drive which needs a new partition table. This seems good to me (for reasons I'll explain later), but I don't know how the partition-table-inside-a-partition thing works and what implications it comes with.
-
Create a sparse max-2TB qcow2 image, store it in the physical partition and attach it to the VM. -> rejected by me because the partition inside the qcow2 image needs constant resizing as your storage needs grow.
-
Create a fully initialized 2TB qcow2 image. -> current way of doing it, no resizes, no security concerns (I guess). The only drawback I perceive is the time required to initialize a 2TB image (~2.5hours in an HDD).
-
Use the physical partition as NFS. I haven't really investigated this solution -nor am I experienced with NFS- but to me it seems like it will require some configuration in the host too, which is something I want to avoid because I don't want to redeploy the host in case shit hits the fan.
So, why 2 seems good to me? Neither resizes as in 3 nor long setup times (image initializing) as in 4.
Is there any other solution that I have missed? If not, out of these, which should I choose?
Sorry for the long, I tried to be as detailed as possible.
-
- Quad9 and Comcast
Not sure if this is the right place.
The last few days I've been experiencing a few issues resolving DNS on my home network. Strangely, rebooting the router seemed to fix it for a time. After running into the issue again I decided to investigate further. I'm using a Mikrotik router with my PC wired in with ethernet cable. The router is using DoH to Quad9 (https://dns.quad9.net/dns-query as per their documentation). I've also imported root certificates for validation.
As of right now, my desktop cannot resolve dns against 9.9.9.9, however it can resolve dns against 1.1.1.1 and 8.8.8.8.
$ dig @9.9.9.9 reddit.com
;; communications error to 9.9.9.9#53: timed out
Interestingly also cannot curl the DoH URL (also a timeout). I thought maybe Quad9 is having issues so I jumped over to my EC2 instance, and I can dig/curl just fine.
I also turned on debug logging on the router, the logs indicate the same issue my desktop is having (timeout errors, sometimes and SSL handshake error).
My question to you all is, have I missed something in my testing/setup, or is Comcast blocking Quad9?
Additional info:
The mikrotik is the latest firmware (6.49.10). I can switch to CloudFlare DoH on the router and it works fine. I can remove the DoH setting entirely and it works. I've got 8.8.8.8 as a static DNS server and the 2 comcast dns servers are dynamic (75.75.75.75 and 75.75.76.76). NTP is setup and the router has the correct date/time/timezone.
As of this writing rebooting the router is no longer temporarily fixing the problem.
Edit:
Thanks u/melmi@lemmy.blahaj.zone !
Per their post the status page shows issues in my area: https://uptime.quad9.net/
- recommendations for personal and family password management?
Not sure if this is the right place to ask, but recommendations for personal and family password management?
I finally switched to Firefox on my phone, because Chrome "privacy". And then when trying to find out how enable password storage, I accidentally set up Microsoft Authenticator as password management phone-wide. Realizing this meant cross-app password management, I finally accepted that my old approach of politely ignoring the problem and manually memorizing algorithmic passwords is no longer tenable. I honestly would prefer the anti-privacy approach where every service just uses oAuth and only one provider has my password, but we're not there today, so time to learn the new tech.
So basically, what's the current OSS best-practice for a one-stop-shop password management software? I know "OSS" and "big safe cloud storage provider" are kind of oxymoronic, but imho encrypted-cloud-storage is the best tradeoff between security and convenience.
And, ideally, something I could get my kids onto as well and manage some shared family-PWs as well, since I assume their password management strategies are either "reset every time" or "just use the same PW everywhere and it's a ticking time-bomb".
- btrfs raid1 with two disks behaviour.
Hey all,
I want to start using btrfs on my san/nas and use that as a backend for my nextCloud. Before I had read up on btrfs I was thinking about using RAID1. I thought RAID1 would fulfill my two requirements:
- It would allow me to just pull out a disk and put in a usb dock and read its contents. (disaster recovery, or for my SO to just power down the server and get her data off if something happens to me).
- It would simply Mirror the data so a single drive can fail and everything is fine.
Now I read things on the documentation of btrfs and in some other places that the RAID1 implementation of btrfs is non-standard, in that is also has some striping functionality.
The image included is from the btrfs docs and it seems it also stripes, not just mirrors, when using 4 disks.
Now my question is: What is its behaviour when using 2 disks? Will this fullfill my two requirements? If not do you have any other recommendations? (I mean i could use zfs...)
A penny for your thoughs :-).
- btrfs partition recognized as swap
Hey all,
Edit: changed confusing wording based on dack's comment.
I have a problem. I'm building a SAN and I'm playing around with btrfs to learn more about how to use it.
I run into the problem where my sdd1 partition is recognized as
swap
filesystem. I don't understand whats going on here. I formatted all these drives through my usb-dock via my desktop. All the others are fine, so why is this one giving me problems? I tried removing it with parted amd recreating it as btrfs or ext4 doesnt seem to help.Does anyone have any insight of why this is happening?
root@server :~# lsblk --fs .... sdc └─sdc1 ext4 1.0 e3e8849d-a25e-4235-8ebf-ca84a7637f64 sdd └─sdd1 swap 1 445ae89e-05ef-4fd0-98e3-b592fb2a8a9c sde └─sde1 btrfs bc864736-2bf6-4379-aa57-46f1c0f3a95d
- how to deal with the planned vs adhoc work.
Hey all.
I need some advice on how to deal with the adhoc vs planned work. There are emails, tickets and verbal interruptions that need my attention. Additionally there is an incrr sing amount of meetings I need to attend. At the same time I want to focus on the development of the infrastructure for the planned work. I notice that all the interruptions are detrimental to both the planned and the adhoc work.
The fact that I have to switch my attention all the time and can't just focus starts to frustrate me. It also has to do with my adhd. I cant utilize my hyperfocus to finish the planned work, instead it stimulates the attention switching side of my adhd and cant get into the problem. I just notice I am not as effective as I was before I got this workload.
Do you people recognize this struggle? How do you deal with this?
- How are you identifying least privileged access in 365?
I’ve started at a medium-sized org (~1500 users) that has over a dozen global admins in 365, plus another 80 users with various 365 admin access. Does anyone have any tips for how to identify what access the users actually need?
I tried punching up a questionnaire with all of the available options, but my test group reported that it was too convoluted. I’m not sure how I can better identify their needs without interviewing them one-on-one, or just ripping away access and seeing who screams.
- Windows feature that resets system clocks based on random data is wreaking havocarstechnica.com Windows feature that resets system clocks based on random data is wreaking havoc
Windows Secure Time Seeding resets clocks months or years off the correct time.
- Azure ChatGPT: Private & secure ChatGPT for internal enterprise usegithub.com GitHub - microsoft/azurechatgpt: 🤖 Azure ChatGPT: Private & secure ChatGPT for internal enterprise use 💼
🤖 Azure ChatGPT: Private & secure ChatGPT for internal enterprise use 💼 - GitHub - microsoft/azurechatgpt: 🤖 Azure ChatGPT: Private & secure ChatGPT for internal enterprise use 💼
- best way to study for RHCSA / RHCSE
Hey all,
I would like to get the above certifications. What resources did you use to study? I can't afford the official training and my employer doesn't want to pay for it.
Any and all help, and all tales of your experience is aplriciated.
- BlueJeans, Verizon’s Google Meet competitor you’ve never heard of, is shutting down9to5google.com BlueJeans, Verizon's Google Meet competitor you've never heard of, is shutting down
Did you know Verizon acquired a video conferencing app back in 2020? It’s fine if you haven’t because Verizon has...
Hopefully not a lot of you had their business use BlueJeans as their core videoconferencing software. Because if you do, you’ll want to plan a migration soon.
- PostgreSQL vs SQLite date management
As a sysadmin mostly used to the nice and powerful way Postgres manages dates, every time i’ve had to do stuff on SQLite i find myself missing that. Feels like they offloaded that into whatever code connects to the database instead of handling it at DB level.
Is there a way to give SQLite the powerful and reliable date management Postgres has, or at least something similar? Hopefully something as devoid of dependency hell as SQLite itself is
- Software management for Windows Server
I have 15 VM's running for clients and I'm looking for a way to keep the tools up to date without having to connect to each server and do it manually. A few examples are WinDirStat, Firefox, SSMS, Filelocator, etc.
We have expanded recently and I'm at the limits of doing this manually. These servers are not domain joined and are in separate virtual networks.
- Recovery of a VM DC
So, I have a VM DC that I had to restore from a month ago. I had other DCs that were physical and up. My understanding that if sub 60 days "off" it is fine to basically "power back on" the snapshot. However, now the "restored" DC has disabled replication in both directions. Should I manually enable inbound replication first and then after a while enable outbound replication?
Or a better fix method?
- Happy System Administrator Appreciation Day!
Thanks guys, gals and everybody in between.
- Reconstruct Docker containers config from a running machine?
Hi! I've inherited a machine installed by somebody else who's no longer in the company or the country. The machine is running just fine, but i see no Dockerfiles or docker-compose.yml, and this looks like something that came from a Compose file with a few linked containers.
Is it possible to reconstruct that info from the running containers? I'm still a raw Docker newbie at this point so i don't know if this is even possible, would be helpful not to have to try and contact the person who set it up.
- Compromised Microsoft Key: More Impactful Than We Thought | Wiz Blogwww.wiz.io Compromised Microsoft Key: More Impactful Than We Thought | Wiz Blog
Our investigation of the security incident disclosed by Microsoft and CISA and attributed to Chinese threat actor Storm-0558, found that this incident seems to have a broader scope than originally assumed. Organizations using Microsoft and Azure services should take steps to assess potential impact.
This really doesn't make me love cloud identity management. It's exactly the scenario (kind of nightmare one) where you attack the cloud infrastructure and get access to many different customers and apps... potentially in a way completely undetectable by you. At least with local identity providers they have to compromise you, and you might have logs.
- Do you use VMWare ROBO Licensing? Price changes cominglemmy.world Do you use VMWare ROBO Licensing? Price changes coming - Lemmy.world
We recently had a meeting with our new (as in 4th in a year) rep. They let us know ROBO licensing is moving away from the VM Pack method it is now to per socket licensing. Minimum of 16 core per socket purchase, and you can’t stretch a license across multiple cores. We about blew a gasket when we we...
Cross posting from lemmy.world.
We recently had a meeting with our new (as in 4th in a year) rep. They let us know ROBO licensing is moving away from the VM Pack method it is now to per socket licensing. Minimum of 16 core per socket purchase, and you can’t stretch a license across multiple cores.
We about blew a gasket when we were told this. It is going to make our ROBO license jump from about $2K up to $30K PER YEAR. We were told changes to Ent+ are coming too, but details were not known. We are in the process of looking at how moving to another option would look like. Either Hyper-V or Nutanix AHV.
I guess we can see how Broadcom is making their money back. By screwing over their customers.
- Need recommendation of SSH client with host management
I’ve been working on a project that I need constant access (and executing commands) among at least 3 hosts in work. I’ve been using SSH’s Host function to manage which host I’m connecting to. However, I find it increasingly annoying that I can’t see which host I’m connected to via the tab on my terminal emulator (I’m using Windows Terminal on Windows and Konsole on Linux).
Is there a good SSH GUI client that can show which host a session is connected to? I’ve tried Termius. But $10 per month is too expensive to me for what I’m doing (and I don’t need most of the paid feature).
- Oracle and SuSE responds to IBM
Kind of finally. SuSE https://www.suse.com/news/SUSE-Preserves-Choice-in-Enterprise-Linux/
So... I think this is kind of the worst case scenario re SuSE - an actual fork. But Oracle kind of hints at that, and Amazon already dropped a RHEL compatible AWS Linux for sort of a Fedora Server?
Obviously none of this is great, but would anyone really want Oracle leading a RHEL "close as possible" rebuild? I don't know anyone is going to downstream them.
SuSE is even weirder, as I understand it, SLE/OpenSuSE is a fork from decades ago, or at least also uses RPM? I can't imagine they get any value from trying to make a RHEL fork really... Why not push SLE? All very confusing, that's for sure.
- Patch Tuesday Megathread
Hello c/sysadmin, and welcome to the Patch Megathread! I'm editing this post and leaving it up as a single catch-all sticky post for patch days for the time being, since we're not seeing enough activity to warrant new threads IMO. If someone wants to help moderate / curate content and actively create new patch day posts, please let me know and I'll add you to the mod team.
This is the place to talk about the latest patches, updates, and releases. We put this thread into place to help gather all the information about this month's updates: What is fixed, what broke, what got released and should have been caught in QA, etc. We do this both to keep clutter out of the community, and provide a singular resource to read.
While this thread is timed to coincide with Microsoft's Patch Tuesday, feel free to discuss any patches, updates, and releases, regardless of the company or product.
Remember the rules of safe patching:
- Deploy to a test/dev environment before prod.
- Deploy to a pilot/test group before the whole org.
- Have a plan to roll back if something doesn't work.
- Test, test, and test!
- Serious question, how does your company view the First-Line Support?
Hey all, in my company we've been having a lot of trouble with our first-line support team and I wanted to get some ideas how it works in other companies.
To give some context, I work in a Customer Team (L2-L3 Support) for a MSP, previously I belonged to the Internal Operations Team and they had a very negative view on the first-line team, with opinions like:
- we don't need them
- they lack knowledge
- management can't create a good first-line team because they don't want to invest
But I didn't interact a lot with them before, but now, I have to interact with them on a daily basis, and I see some things that have started to make me worried about the team:
- They ignore KB's
- They say that they don't have access to certain servers, or that they don't find the correct credentials and just pass the ticket for us to solve
- They have people that lack knowledge in some basic support, I have had tickets passed on with notes like "I don't know how to use Linux"
From my point of view and the team I belong now, we all think that management didn't really verify the required knowledge for some members of that team, but they really have a few that are trying really hard to improve their skills.
We have started to try to help them, so that our job can also become easier:
- Improve the language in legacy KB's
- Simplify the process in the monitoring platform with more directions
- Automating some processes so that the first-line can execute fixes without having the required knowledge on the backend
- Picking the best members of their team and promoting them to our team
That team also has some problems that I fully recognize:
- Shit pay
- Bad leadership, that team has had 6 different Team Leaders in a short time (I have been here for only 2 years)
- Lacking interview and requirements for the position
Sorry for the long text, would love to have some feedback from your sides, or is this normal in a lot of companies?
- "Among other things, Suncor employees have also been told in recent days not to use social media on company devices or let people tailgate behind them into an elevator."
Suncor is replacing employee computers after a cybersecurity incident last week shut down debit and credit processing at Petro-Canada gas stations across the country, among a series of other security measures at the Calgary-based company.
"Normally you wouldn't expect hardware to be compromised so fully that you need to replace everything,"
- Handy rsync tricks
What are you favourite/useful
rsync
tricks these days?Mine is
rsync -r --chown=AUSER:AGROUP SRC DST
to copy the files and change the ownership on the fly. - FYI - my Samsung 970EVO Pro died after 4 months of use. Spare capacity at 0
I got my new PC for about 3 or 4 months. Today, I was using my PC as usual and suddenly everything stopped reacting. Rebooting just boots be into the UEFI interface. Which is very concerning.
Then I got a liveusb to look into what's happening. Upon using smartctl. It shows that my SSD have 0% spare capacity despite only writing 15TB to it.
So far, I knew that Samsung's EVO 980 and 990 SSDs have a firmware bug that can cause this. But this is the 1st time I know of 970 Pros having this issue.
I know there's a lot of servers using consumer drives for their system. Be careful and check if you are using a 970. If so, check the spare capacity RIGHT NOW and decided if to upgrade the firmware or RMA the product.
- Happy 50th birthday, Ethernet | APNIC Blogblog.apnic.net Happy 50th birthday, Ethernet | APNIC Blog
Why has Ethernet endured as a ubiquitous data framing protocol over all these years?
- TIL in 1997, a Reply All storm took down all of Microsoft's internal Exchange system
It started off with an employee sending an email to a distribution list called "Bedlam DL3" asking to be taken off the list. With 13,000 recipients and everyone replying all with, "Me too!" and other messages, it was estimated that over 15 million messages were sent through the system in an hour. This crashed the MTA service due to a recipient limit. Each time the MTA service recovered, it would attempt to resend the message again which lead to a crash loop.
As a result of the incident, the Exchange team introduced message recipient limits and distribution list restrictions to Exchange, which is something we all use today!
More on the story here: https://techcommunity.microsoft.com/t5/exchange-team-blog/me-too/ba-p/610643
cross-posted from: https://techy.news/post/2224
- OWA outage this morning
Looks like OWA is down for some users, the rest of the O365 apps appear to be fine.
- Monitoring is a Pain - And we're all doing it wrong (including me)matduggan.com Monitoring is a Pain
And we're all doing it wrong (including me) I have a confession. Despite having been hired multiple times in part due to my experience with monitoring platforms, I have come to hate monitoring. Monitoring and observability tools commit the cardinal sin of tricking people into thinking this is an ea...
Monitoring and observability tools commit the cardinal sin of tricking people into thinking monitoring is an easy problem. It is very simple to monitor a small application or service. Almost none of those approaches scale.
- Outage and vulnerability notifications.
Hey like the title says I'm looking for a way to keep up with outages and vulnerabilities. I mostly used r/sysadmin to alert me to things, but given everything I don't want to go to reddit if I don't have to.
If I can have my preference I just want one point to check at least until the numbers build up here and it becomes the go to place for that info. Anybody know of anything?
- New vCenter Security Vulnerability
VMSA-2023-0014 - VMware vCenter Server updates address multiple memory corruption vulnerabilities (CVE-2023-20892, CVE-2023-20893, CVE-2023-20894, CVE-2023-20895, CVE-2023-20896) Please see the advisory here: https://www.vmware.com/security/advisories/VMSA-2023-0014.html Impacted Products: • VMware vCenter Server (vCenter Server) • VMware Cloud Foundation (Cloud Foundation)
- Testing Service Accounts in `Kubernetes`
cross-posted from: https://lemmy.run/post/10475
> ## Testing Service Accounts in Kubernetes > > Service accounts in Kubernetes are used to provide a secure way for applications and services to authenticate and interact with the Kubernetes API. Testing service accounts ensures their functionality and security. In this guide, we will explore different methods to test service accounts in Kubernetes. > > ### 1. Verifying Service Account Existence > > To start testing service accounts, you first need to ensure they exist in your Kubernetes cluster. You can use the following command to list all the available service accounts: > >
bash > kubectl get serviceaccounts >
> > Verify that the service account you want to test is present in the output. If it's missing, you may need to create it using a YAML manifest or thekubectl create serviceaccount
command. > > ### 2. Checking Service Account Permissions > > After confirming the existence of the service account, the next step is to verify its permissions. Service accounts in Kubernetes are associated with roles or cluster roles, which define what resources and actions they can access. > > To check the permissions of a service account, you can use thekubectl auth can-i
command. For example, to check if a service account can create pods, run: > >bash > kubectl auth can-i create pods --as=system:serviceaccount:<namespace>:<service-account> >
> > Replace<namespace>
with the desired namespace and<service-account>
with the name of the service account. > > ### 3. Testing Service Account Authentication > > Service accounts authenticate with the Kubernetes API using bearer tokens. To test service account authentication, you can manually retrieve the token associated with the service account and use it to authenticate requests. > > To get the token for a service account, run: > >bash > kubectl get secret <service-account-token-secret> -o jsonpath="{.data.token}" | base64 --decode >
> > Replace<service-account-token-secret>
with the actual name of the secret associated with the service account. This command decodes and outputs the service account token. > > You can then use the obtained token to authenticate requests to the Kubernetes API, for example, by including it in theAuthorization
header using tools likecurl
or writing a simple program. > > ### 4. Testing Service Account RBAC Policies > > Role-Based Access Control (RBAC) policies govern the access permissions for service accounts. It's crucial to test these policies to ensure service accounts have the appropriate level of access. > > One way to test RBAC policies is by creating a Pod that uses the service account you want to test and attempting to perform actions that the service account should or shouldn't be allowed to do. Observe the behavior and verify if the access is granted or denied as expected. > > ### 5. Automated Testing > > To streamline the testing process, you can create automated tests using testing frameworks and tools specific to Kubernetes. For example, the Kubernetes Test Framework (KTF) provides a set of libraries and utilities for writing tests for Kubernetes components, including service accounts. > > Using such frameworks allows you to write comprehensive test cases to validate service account behavior, permissions, and RBAC policies automatically. > > ### Conclusion > > Testing service accounts in Kubernetes ensures their proper functioning and adherence to security policies. By verifying service account existence, checking permissions, testing authentication, and validating RBAC policies, you can confidently use and rely on service accounts in your Kubernetes deployments. > > Remember, service accounts are a critical security component, so it's important to regularly test and review their configuration to prevent unauthorized access and potential security breaches.