I've got a Debian server running with a ssd and 2 hdds (SDA & SDB). SDA is EXT4 and SDB is XFS. Both are Toshiba MG07ACA14TE (512e), 14TB.
The situation is as follows:
The hdds are mostly in standby and only awaken when needed. Every night I run a Snapraid script and some other scripts for backup. Around every two weeks, the hdds show some strange behavior. Once SDA awakens for something, it stays active for around 2 days and then goes to standby mode again. It's just active spinning, and as far as I could check, nothing gets written or read. I did:
- ps -ef: no process that links to this behavior.
- iotop -a: nothing for SDA
- fatrace -c: nothing for SDA
- dstat --nocolor -dD sda: nothing for SDA
While normally if I check hdparm -C /dev/sda, the state of the drive is standby, idle or active. But now it shows active/idle. When I check again after two days, it show standy, idle or active again.
Another interesting thing I noticed, is that before I formatted SDB to XFS, it was also EXT4 and it showed the same behavior as SDA. The only difference was that the awakening was a day later, because at some point I disconnected the disc from the server for a day for testing. Yet everytime SDA starts spinning for 2 days, a day later SDB followed.
From a topic on a different forum, I got some info that maybe this behavior is caused by Ext4-rsv-conver. I checked through PS and I see this process, but now I doubt that this is causing the issue, since it shows after the disk already spinning and while the disk is normally in standby, I also see this process running from time to time.
Does anybody have an idea what is causing the hdds spinning for 2 days and go into normal mode again? Or give me some advice on what else I can try for troubleshooting?