I was wondering what is the "proper" or "most significant" way in which to measure the boot time of a client machine:
etc...
I generally use the third, as this measures the time before the system becomes usable and useful. What do you think?
I would measure the time from the point that POST ends to the time that an automated login completes and the user is able to use their most frequently used application (ie: Firefox,etc). Waiting for the hard drive light to stop is mostly meaningless with background indexing and read ahead related tasks, your hard drive may not stop activity until several minutes after you're actually able to begin using your computer effectively.
If you're doing this on Linux you can try Boot Chart, it will help you identify what's taking so long to boot.
The main thing is to be consistent in your measuring methodology so that if you tweak anything you're comparing apples to apples.
I don't know if there's a standard for this, but personally I consider my system to have 'booted' when I am able to open the start menu (or dock) and start an application. Before that point I don't really consider the system usable.
That said, I imagine any manufacturers that are quoting boot time are quoting the shortest possible time, which will be to login prompt, or desktop appearing I would imagine.
I think boot time is generally measured as when you get your login prompt and can start interacting with the operating system, as the OS has background services running, you can interact with it, and if you're launching applications to measure "usefulness" you're introducing variables that the OS manufacturer has no control over (they didn't make firefox, or other applications you commonly use, so it's not their fault if the libraries/etc. are slow to load).
If you're benchmarking the OS's boot performance, you measure what the OS is taking up in time before it's telling you "Hello, world!"...the login prompt's time to begin taking input.
I usually measure and record all these as they will, over time, all vary with system problems, start-up processes, changes to applications, changes to the machine specs, change of models of PC purchased and so on. Have a record of just what these times were at original build can be quite helpful.
On when the final test should finish (3 apps open) there can be value in not just having the app open but having performed a simple first use (e.g. a browser will load a home page and this may suffice; a word processor could load a test document (same for every test: already loaded at least once previously if on a network resource)).
In a few cases a more detailed analysis is worthwhile. Being network-centric I would usually look at timings from power-on such as:
Typically these would be done by noting the time on power-up and then looking through a tcpdump or wireshark trace subsequently.
Again, a record of this, perhaps with indications of numbers of frames and total bytes, is helpful as software and systems changes and as a reference in the event of problems.
On Windows I generally measure until the Start menu can open or you can get a response by right-clicking on the desktop. Another metric can be obtained by keeping taskmgr.exe in your Startup folder; once CPU usage drops to a usable state the machine can be considered "booted". A lot of software can load post logon (AV, various agents or clients, etc) and there seems to me to be no point whatsoever in getting you to the desktop as fast as possible if it's completely unusable for a few minutes after that.
The official Windows definition of boot time, starts when NTLDR or WINLOAD finishes and the kernel starts by noting the system time and initializing its drivers and ends at the first logon, when the boot is considered "good".
(In Windows, the WMI object for this is Win32_OperatingSystem/LastBootUpTime.)
For any system, Windows or not, you might want to get a stopwatch and see how long the POST goes. Stop when you see the OS screen. It can be very long if you have one or more RAID or SCSI controllers in the machine. Then take another measurement from that point until the machine is usable/you log in/all your daemons start.
