SSD (Intel 530) read/write speed very slow with RAID 10

Question

Explanation:

We have a Server:

Model: HP ProLiant DL160 G6
4 x 240GB SSD (RAID-10)
72GB DDR3 RAM
2 x L5639
HP P410 RAID Controller (256MB, V6.40, Rom version: 8.40.41.00)

SSD drives are 4 brand new 2.5" Intel 530 with 540MB/s read speed and 490MB/s write speed

CentOS 6
File systems are ext4

but this is the test result for read speed on raid 10:

hdparm -t /dev/sda
/dev/sda:
 Timing buffered disk reads:  824 MB in  3.00 seconds = 274.50 MB/sec
[root@localhost ~]# hdparm -t /dev/mapper/vg_localhost-lv_root
/dev/mapper/vg_localhost-lv_root:
 Timing buffered disk reads:  800 MB in  3.01 seconds = 266.19 MB/sec

and this is for write speed:

dd bs=1M count=512 if=/dev/zero of=test conv=fdatasync
512+0 records in
512+0 records out
536870912 bytes (537 MB) copied, 4.91077 s, 109 MB/s

we were hoping for 1GB read speed with raid 10 but 270MB isn't even the speed of a single disk!

Questions:

Why is it so slow?
Is it because of the RAID Controller?

Update 1 - Same Read/Write Speed:

After changing some settings as mentioned in answers i have the result below:

(Any one knows why it shows 4GB instead of 400MB as read speed?!)

EDIT: looks like the command was wrong and we should've used -s144g for this amount of ram, thats why it shows 4GB (as suggested in comments by ewwhite)

[root@192 ~]# iozone -t1 -i0 -i1 -i2 -r1m -s56g
        Iozone: Performance Test of File I/O
                Version $Revision: 3.408 $
                Compiled for 64 bit mode.
                Build: linux
    Record Size 1024 KB
    File size set to 58720256 KB
    Command line used: iozone -t1 -i0 -i1 -i2 -r1m -s56g
    Output is in Kbytes/sec
    Each process writes a 58720256 Kbyte file in 1024 Kbyte records

    Children see throughput for  1 initial writers  =  135331.80 KB/sec
    Children see throughput for  1 rewriters        =  124085.66 KB/sec
    Children see throughput for  1 readers          = 4732046.50 KB/sec
    Children see throughput for 1 re-readers        = 4741508.00 KB/sec
    Children see throughput for 1 random readers    = 4590884.50 KB/sec
    Children see throughput for 1 random writers    =  124082.41 KB/sec

but the old hdparm -t /dev/sda command still shows:

Timing buffered disk reads: 810 MB in 3.00 seconds = 269.85 MB/sec

Update 2 (tuned-utils pack) - Read Speed is now 600MB/s:

Finally some hope, we had disabled cache from raid controller and did some other things earlier with no luck, but because we reloaded the server and installed the OS again, we forgot to install "tuned-utils" as suggested in ewwhite's answer (Thank you ewwhite for this awesome package you suggested)

After installing tuned-utils and choosing enterprise-storage profile the read speed is now ~600MB/s+ but the write speed is still very slow (~160MB) (:

Here is the result for iozone -t1 -i0 -i1 -i2 -r1m -s144g command:

    Children see throughput for  1 initial writers  =  165331.80 KB/sec
    Children see throughput for  1 rewriters        =  115734.91 KB/sec
    Children see throughput for  1 readers          =  719323.81 KB/sec
    Children see throughput for 1 re-readers        =  732008.56 KB/sec
    Children see throughput for 1 random readers    =  549284.69 KB/sec
    Children see throughput for 1 random writers    =  116389.76 KB/sec

Even with hdparm -t /dev/sda command we have:

Timing buffered disk reads: 1802 MB in 3.00 seconds = 600.37 MB/sec

Any suggestions for the very slow write speed?

Update 3 - Some information requested in comments:

Write speed is still very low (~150MB/s which isn't even 1/3 of a single disk)

Output for df -h and fdisk -l:

[root@192 ~]# df -h
Filesystem      Size  Used Avail Use% Mounted on
/dev/sda1       441G  3.2G  415G   1% /
tmpfs            36G     0   36G   0% /dev/shm
[root@192 ~]# fdisk -l
Disk /dev/sda: 480.0 GB, 480047620096 bytes
255 heads, 63 sectors/track, 58362 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk identifier: 0x00040c3c
Device Boot      Start         End      Blocks   Id  System
/dev/sda1   *           1       58363   468795392   83  Linux

score 16 · Answer 1 · edited Apr 13 '17 at 12:14

While the other answer here beings up some points, your specific issues are due to platform limitations and OS configuration:

You're throughput-limited by the use of consumer SATA SSDs on an HP Smart Array P410 RAID controller. SATA disks run at 3.0Gbps (3G) on these controllers rather than 6.0Gbps (6G). So that's a barrier that impacts the read speeds of your Intel SSDs; 300MB/s or less per drive.
The Smart Array P410 controller has specific requirements and best-practices when used with SSDs. In short, the controller is capable of 50,000 IOPS, disable the array accelerator for your SSD volume and performance tops out at ~6 drives.
Disk performance is not always about sequential read/write speed. Try benchmarking with a proper tool like iozone or bonnie++. You still get the random I/O benefits of your multiple drives.
At the operating system level, install the tuned-utils package and set the profile to enterprise-performance to remove write barriers from your filesystems and set the right I/O elevator for your setup. This is covered in other questions here, too.
It looks like you're using LVM. That can have an impact as well...

Here's an iozone report for a G7 ProLiant running with four consumer 6G SATA SSDs (downshifted to 3G speeds) on the same HP Smart Array P410 RAID controller.

You should be seeing ~470MB/s writes and 650MB/s+ reads.

[root@mdmarra /data/tmp]# iozone -t1 -i0 -i1 -i2 -r1m -s56g
        Iozone: Performance Test of File I/O
                Version $Revision: 3.394 $
                Compiled for 64 bit mode.
                Build: linux 

        Record Size 1024 KB
        File size set to 58720256 KB
        Command line used: iozone -t1 -i0 -i1 -i2 -r1m -s56g
        Output is in Kbytes/sec
        Each process writes a 58720256 Kbyte file in 1024 Kbyte records

        Children see throughput for  1 initial writers  =  478209.81 KB/sec
        Children see throughput for  1 rewriters        =  478200.84 KB/sec
        Children see throughput for  1 readers          =  677397.69 KB/sec
        Children see throughput for 1 re-readers        =  679523.88 KB/sec
        Children see throughput for 1 random readers    =  437344.78 KB/sec
        Children see throughput for 1 random writers    =  486254.41 KB/sec

score 12 · Answer 2 · answered Nov 26 '13 at 12:47

Oh dear, where to begin?

There is so much involved, and you need a good understanding of everything. Just throwing a bunch of disks against a RAID controller won't yield the results you're looking for.

This can't be easily answered. But at least, here is a list of stuff you have to look at:

Does the Controller even have the throughput needed? (-> Datasheets)
Does the Controller even have enough bandwidth to the Host (it does, even at v1.0, since it's x8)
Does the system's chipset have enough throughput (CPU-Controller)? (unknown)
What write strategy have you instructed the Controller to employ? (this is what most likely has bitten you)
Is everything aligned (Partition starts, LVs, PVs)?
Are the block sizes harmonized? (RAID stripe size, block size, FS blocks,...)
Is the filesystem optimized to the RAID setup? (Level and Block size)

Since your throughput against the whole RAID (disregarding FS) is significantly lower than a single disk, it's likely you have set up your write strategy wrongly; the Controller is probably waiting for all disks to confirm the write (and unless you've got the RAM on the controller battery backed, this might be in your best interest).