Sheepdog is a distributed object storage system designed to provide block storage for virtual machines, known as no single point, zero configuration, linear expansion (not more merit description). This article focuses on the performance of how, for the current test version of the latest stable version 0.7.4. Test Environment node number: six disks: each node is equipped with 7200 RPM SATA hard Nike Free 3.0 V5 drive, model WDC WD10EZEX-22RKKA0, capacity of 1TB, additional testing nodes (ie, used to start the virtual client host) more a SSD hard drive configuration, model INTEL SSDSA2CW300G3, capacity of 300GB network: test node with PCI-E Dual Gigabit Ethernet, by bonding for load balancing file system: ext4fs, mount options: rw, noatime, barrier = 0, user_xattr, data = writebacksheepdog cluster space available: 6 nodes each assignment 100GB, 600GB total number of copies of objects: 3, client actual maximum available space for 200GB, the capacity of the object cache near the project planning: Using SSD drives, distribution 256GB, can be fully cached client file system, with the test projects need to mount or unmount the virtual client resource: 4GB Mens Nike Free Run 3 Shoes Blue 3 of memory, four logical CPU start using qemu 1.7, support for KVM acceleration, virtio two test items IOPS: mainly on the ability of small block random read and write data, test tools fio (sudo apt-get install fio), 150GB throughput test files: mainly on the capacity of larger data block sequential read or write, test tools iozone (sudo apt-get install iozone3), test file from increments of 64MB to 4GB , unless otherwise specified, the test results from the operation of 512MB of data files Unless otherwise stated, qemu start the virtual client command as follows: qemu-system-x86_64 --enable-kvm -m 4096 -smp 4 -drive file = sheepdog: mint , if = virtio, index = 0 \\ Nike Free 3.0 V4 -drive file = sheepdog: data, if = virtio, index = 1 -net nic, model = virtio -net user -vnc: 1 -daemonizeIOPS Test Test conditions About SATA hard disk IOPS test 7200 ordinary SATA hard drive, the official announcement of its average seek time of 10.5 milliseconds, according to the following formula, the theoretical maximum IOPS value of only 65. Found a result, in the case of single-threaded synchronous IO (bottom blue line), Nike KD 7 Womens also 65, in the multi-tasking (purple line) or using asynchronous IO (yellow line), the operating system since the IO scheduler, IOPS can reach 80 between 100. If the file read and write operations is far smaller than the SATA disk size, reduce the storage unit addressing range, reducing the whole disk seek time, but also improve IOPS, for example, test file size accounted for 1/8 of the disk size when, IOPS highest in about 90 (light blue line). IOPS SSD hard drive test on the SSD hard drive, in the case of single-threaded synchronous IO, IOPS can reach up to about 9000 (bottom blue line), at multi-tasking (purple line) or asynchronous IO (yellow line) in the case of , can reach up to between 40000-50000. Unlike traditional disk due to the construction principle, reduce the size of the test file did not significantly improve IOPS. About literacy ratio because most business scenarios both read operations, but also writes, and generally read much more than write operation, therefore, the reader literacy 579765-700 Sonic Yellow / Sail - Cool Grey - Tour Yellow Nike Air Max LeBron 10 Low Sonic Yellow Online test project mixing ratio is set to 4: 1. Few general business situation completely random write, so do not write only test. Elevator algorithm on virtual guest operating system IO scheduling algorithm uses noop, online information can improve IOPS, but the actual effect is not obvious, so the final report did not put it singled out. IOPS Test 1: Do not use the object cache, read-only single-threaded synchronous IO test case (bottom blue line), sheepdog of IOPS almost reached 100, higher than the single-node single SATA hard drive, the reason: the client test file as 150GB, there are three copies, which together accounted for 450GB, the cluster has six nodes, each node average 75GB, and each node where the disk capacity of 1TB, only its 1/13, therefore, no other IO task case next, IOPS will be higher than the full disk random operation. Reducing the size of the client test file to the original 1/8, namely 19GB, IOPS can reach 130-140 validated without other IO tasks, the smaller the test file, IOPS higher (light blue line). Recover the client test file to 150GB, in multitasking (the number of threads 10, purple line) or asynchronous NIKE KD 6 NSW IO (queue depth of 10, yellow line) case, IOPS up whether the sheepdog cluster around 230-250.250 Limit? Other combinations of further change numjob and iodepth tested, the answer is yes, the test results in about 250, the following is the number of threads 8, asynchronous IO queue depth Air Jordan Outlet test results were 1,4,16,64,256 and thread the number increased to 16, the test data did not improve. IOPS Test 2: Use the Air Max 2011 Womens Grey Red Black object cache, read-only test single-threaded synchronous IO, using the object cache and the cache hit rate of 100% of the cases (below the blue line), sheepdog of IOPS up to 6000, compared to the same conditions SSD drive The test results (up Nike Air Max to 9000), or some loss. By multitasking (purple line) or asynchronous IO (yellow line) way to increase the number of concurrent IO, in the cache hit rate of 100% of the cases, IOPS can be increased to 40000-50000, roughly equal with the test results under the same conditions SSD drive ʱ?? Note Pictured on a logarithmic scale, and there is no data value, difficult to compare the numerical size, the following figure omitted half of the data block, but marked with a data value. The emphasis on the Air Max 2011 Womens Grey Purple cache hit rate, is incomplete because when the cache hit, IOPS decrease very much, less than 80%, it may not be as no object cache. The figure is the sheepdog and the object cache hit ratio IOPS relationship diagram, wherein the data block size test multiple increments from 512B to 512KB, the cache hit rate is in the testing process, depending on the number of cache directory sheepdog existing object file estimate The average value, IOPS value is the average of the data block read operation at the estimated cache hit rate measurement conditions. 50000 whether the sheepdog cluster limit? Follow the above method for testing, the answer is yes, the test results are less than 50,000, and the following is the number of threads 8, asynchronous IO queue depth test results 1,4,16,64,256 respectively, the number of threads to 16 , the test data did not improve. IOPS Test 2015 Nike Free 5.0 3: do not use the object cache, read and write mix testing hybrid test write IOPS than read-only test results generally lower, and undulating, you need to calculate the average of several tests. IO threaded synchronous case (bottom blue line), can reach 80-100. Reducing the size of the client test file to the original 1/8, namely 19GB, IOPS can reach 100-120 (light blue line). Recover the client test file to 150GB, in multitasking (the number of threads 10, purple line) or asynchronous IO (queue depth of 10, yellow line) case, IOPS mostly between 180-250, but only sometimes to about 160. IOPS Test 4: Using object cache, read and write single-threaded synchronous hybrid test IO, using the object cache and the cache hit ratio under 100 percent of the cases, IOPS almost reached 4,000. Increase the number of concurrent IO IO through multi-tasking or asynchronous manner, at the cache hit rate of 100% of the cases, IOPS up between 10000-20000. Note Pictured on a logarithmic scale, and there is no data values, the following figure omitted half of the 579827-300 Sport Turquoise Nike LeBron X PS Elite Online data block, but marked with a data value. You can see, in the case of large data blocks read mixed, multi-tasking or asynchronous IO of IOPS may not be as single-task synchronization IO throughput test test conditions About SATA hard disk throughput ordinary SATA hard drive throughput of 7200 data, compared to look at the back of the SSD throughput data, indicating the ability to read and write traditional disk order is still quite good. About SSD hard disk throughput of large data blocks (512KB or more) sequential read throughput of up to 250MB / s-270MB / s, sequential write throughput of up to 200MB / s-210MB / s. Small blocks of data (16KB) sequential read throughput is also 140MB / s, sequential write about 120MB / s-130MB / s. Under no optimization sheepdog throughput optimized software and hardware without any conditions (not binding dual card, do not use virtio, with default parameters to start sheepdog), sheepdog data throughput is relatively low: the order of write block size of 16KB, Throughput 13-14MB / s data block size between 256KB-4MB, throughput 30-35MB / s sequential read data block size of 16KB, the throughput of 25MB / s or so, if the data file size exceeds the available memory, reduced to 20MB / s data block size between 512KB-4MB, throughput 125-140MB / s, if the data file size exceeds the available memory, reduced to 80MB / s or less following two figures, the picture shows the test data file 512MB of data, Nike Lunarglide+ 3 the figure below 4GB data files test data Note, do not use the virtio way to start sheepdog command line parameters: qemu-system-x86_64 --enable-kvm -m 4096 -smp 4 -drive file = sheepdog: mint, index = 0 \\ - drive file = sheepdog: data, index = 1 -vnc: 1 -daemonize Throughput Test 1: Do not use the object cache from the test results, using virtio way to start the client the pros and cons: increase the throughput of small data blocks read and write operations When the amount of data block size at between 16KB-512KB, sequential read throughput increase 10MB / s-30MB / s range at the time when the data block size between 16KB-256KB sequential write throughput increase 5MB / s- 8MB / s range greatly reduces large data block read throughput virtio as to why in this case will reduce the throughput test, it is not clear when the data block size reaches 2MB or 4MB, sequential read throughput reduce 80MB / s-90MB NIKE KD 6 NSW / s The following two figures, the picture shows the test data at 512MB, the next picture shows this case, with no optimization sheepdog write difference between throughput of 512MB data file. Throughput Test 2: Use an object cache uses the object cache, read and write throughput with sequential data file size is not directly related, even if the data file size exceeds the available memory. Also use the virtio, and not to read large Nike KD 7 blocks of data thus reducing throughput. Because of the characteristics of SSD drives, the data block, the higher the throughput, in the case of a data block size of 4MB, sequential read throughput of up to 240MB / s, sequential write also have 130MB / s; if the data block size of only 16KB sequential read and sequential write throughput were only 55MB / s and 35MB / s. Summary sheepdog performance object cache IOPS throughput (MB / s) without a read-only 250, read and write the mixed (4: 1) 200 block 512KB, sequential reads 150, 35 have read only 45,000 sequential write, read and write the mixed (4: 1 ) 16,000 blocks 512KB, sequential reads 175, sequential write 105; case of the data block 4MB, sequential reads 240, sequential write 130 or more for the sheepdog cluster nodes is 6, and each node only mount a SATA hard disk measured The result, if the increase in the cluster nodes, and each node are covered with hard drives, IOPS should be higher, and throughput depends on the cache and network bandwidth should not have changed significantly.Sheepdog distributed object storage system performance test