Yep, one thing that helps in this context is to only look into threads in R or D state (and ignore the hundreds or sometimes thousands of kernel threads that are just sleeping or waiting for more work to do in idle (I) state).
But yeah, even that doesn't clearly tell you what amount of active threads is normal for your workload/configuration. Or you can just save the thread count info (grouped by thread state) and save/graph it over time, to have a better idea of what's normal. The difference of this approach vs just graphing Linux-reported load is that you'll get a breakdown of how much of it was CPU demand vs sync I/O demand. Or you could even break the the thread states down further, by syscall and WCHAN, for example. This will give you a much more detailed idea of why the load is high, not just that it is high.
(Like I mentioned in the article), I even have a tool for sampling /proc at regular intervals (1 Hz by default) and saving it into hourly CSV files, so you can easily go "back in time" and zoom into short spikes and see what was going on:
$ xcapture
0xTools xcapture v1.0 by Tanel Poder [https://0x.tools]
Sampling /proc...
DATE TIME PID TID USERNAME ST COMMAND SYSCALL WCHAN
2020-10-17 12:01:50.583 6404 7524 mysql R (mysqld) fsync wait_on_page_bit
2020-10-17 12:01:50.583 6404 8944 mysql D (mysqld) fsync wait_on_page_bit
2020-10-17 12:01:50.583 6404 8946 mysql D (mysqld) fsync wait_on_page_bit
2020-10-17 12:01:50.583 6404 76046 mysql D (mysqld) fsync wait_on_page_bit
2020-10-17 12:01:50.583 6404 76811 mysql D (mysqld) fdatasync xfs_log_force_lsn
2020-10-17 12:01:50.583 6404 76815 mysql D (mysqld) fsync blkdev_issue_flush
2020-10-17 12:01:50.583 8803 8803 root R (md10_resync) [running] 0
DATE TIME PID TID USERNAME ST COMMAND SYSCALL WCHAN
2020-10-17 12:01:51.623 6404 7521 mysql D (mysqld) pwrite64 xfs_file_buffered_aio_write
2020-10-17 12:01:51.623 6404 7524 mysql D (mysqld) fsync xfs_log_force_lsn
...
But yeah, even that doesn't clearly tell you what amount of active threads is normal for your workload/configuration. Or you can just save the thread count info (grouped by thread state) and save/graph it over time, to have a better idea of what's normal. The difference of this approach vs just graphing Linux-reported load is that you'll get a breakdown of how much of it was CPU demand vs sync I/O demand. Or you could even break the the thread states down further, by syscall and WCHAN, for example. This will give you a much more detailed idea of why the load is high, not just that it is high.
(Like I mentioned in the article), I even have a tool for sampling /proc at regular intervals (1 Hz by default) and saving it into hourly CSV files, so you can easily go "back in time" and zoom into short spikes and see what was going on: