Okay so my question might not make a lot of sense but how does an OS or I'd guess a kernel knows what's the CPU utilization? I'm pretty sure it's reported by the CPU, but how does the CPU measure that? From my understanding, CPUs have multiple "units" that can specialize in specific operations. Is that taken into account, does it just measure the utilization of it's "general use" units, and if so how does the CPU keep track of it's own usage? I'd google it but I honestly don't know what to look for past the OS/kernel layer
The OS is in charge of telling the CPU what to run. Literally the context switching code sets saves its old registers and reads in the registers of the program that's about to run (including the program counter that says what address to start at). So at that time it can save information about how the current time and long the previous code ran.
Low-level hardware block usage generally isn't accounted for that way, but CPUs can keep track of performance counter events tracking how many of each operation are performed and how many cycles each subunit is active. The OS can periodically read that data, but it's only used for low level metrics about how efficient the code is rather than for calculating overall CPU utilization.
There are separate measurements about GPU and hard disk utilization, which are either determined by when driver queues work on the device and receives a notification that the device is done, or by the device keeping track of what work it's switching between and reporting it periodically.
