Monitoring file access in Linux

Question

Is there a way to gather statistics about blocks being accessed on a disk?

I have a scenario where a task is both memory and I/O intensive and I need to find a good balance as to how much of the available RAM I can assign to the process and how much I should leave for the system for building its I/O cache for the block device being used.

I suspect that most of the I/O that is currently happening is accessing a rather small subset of files (or parts of large files) and that performance could be optimized by increasing the RAM that is available for I/O buffering.

Ideally, I would be able to create something like a "heat-map" that shows me which parts of the files are accessed most of the time.

Setup currently is based on CentOS 5 on AWS/EC2 m1.large instance. Disk setups are either ephemeral block devices in a RAID0 setup (LVM) or alternatively a single (500GB) EBS

Update: Originally, this question was talking about disk blocks, which was misleading as I am actually interested in the logical blocks being accessed and I don't care where they are on the physical devices. I changed this to make clear that it is parts of files I'm interested in. I apologize for the confusion.

Answer 1

I'm not sure you fully understand how modern buffer caches work -- you're about half right in that you want to limit how much RAM your process uses (so there's "enough" available for the buffer cache, but you're thinking about it in the wrong way.

What you're asking for is not really useful for tuning the buffer cache -- it MIGHT be useful if you have a single contiguous disk (or an array that presents as one and behaves as one) and are looking at optimizing on-disk layout, but that's getting into Deep Filesystem Magic.
You can read McKusick's papers on filesystem design (or spend 42 minutes and watch this great video) to get a basic concept of how the filesystem already tries to optimize that for you - Filesystems are pretty good at getting the on-disklayout right.

---

In terms of buffer cache optimization, you want to look at the number of cache hits vs. cache misses (and specifically what's causing the misses). The physical location on the disk doesn't matter - what matters is how many times you have to hit the disk to get what you want, and whether your cache is big enough that it's not constantly churning (essentially negating the cache efficiency).

Tuning that is a bit more trial-and-error than anything else -- a grossly inefficient rule of thumb is to leave 2x the size of your biggest file/chunk of data for the buffer cache, but you're almost always better off starting hugely skewed to either the app or the cache and adjusting toward peak performance.

Answer 2

If you're talking about a server-class system, there are other variables to consider. I understand what you're asking for, but on modern systems, these things have been abstracted by multiple levels of cache and the optimizations from intelligent RAID controllers.

For write-biased activity, much of your random write workload should be written to battery or flash-backed non-volatile cache (in order to provide quick acknowledgement of writes), coalesced and flushed sequentially to your disks. If you're not employing the use of something like this, you're leaving performance on the table.

For read activity, the OS does a reasonable job of cachine. Having additional controller cache helps. And beyond that, you can employ the use of a few tricks to help control your virtual memory subsystem. (see: Virtual Memory Toucher)

Also see: Clear / Flush cached memory

But again, we need details of your setup to help understand how to help.

Answer 3

I will put my vote in for DSTAT (http://dag.wieers.com/home-made/dstat/). Take a look at some of the switches like top-io, top-latency, top-mem, etc. It is not going to do a heat map for you or which parts of the disk are being accessed but it may help point you in the right direction

Answer 4

Use iotop. That is exactly what you need.