Help Center > Object Storage Service > Developer Guide > Performance Optimization

Performance Optimization

Suggestions on Single Bucket Optimization

OBS updates the same metadata when processing concurrent PUT Object or DELETE Object operations on a bucket. These concurrent operations may conflict and compromise system performance. System performance, however, is not affected by concurrent GET Object operations. For concurrent PUT Object or DELETE Object operations, the Transactions per Second (TPS) limit of a bucket in OBS is 70, and you are advised to keep the TPS less than 70. When the TPS has exceeded this value, error 503 is returned. However, the performance of a bucket is subject to the bucket's TPS only when the object size is small. When the object size is relatively large, bucket performance has little to do with bucket's TPS because PUT operations in this case mainly process object data and seldom update metadata. In general, bucket performance is inversely proportional to the object size in PUT operations. According to the test result, when the average size of the objects is larger than 10 MB, the impact of concurrent metadata updates on the bucket performance can be ignored. For concurrent DELETE operations on a bucket, the performance of the bucket is subject to the bucket's TPS regardless of object sizes.

NOTE:

The preceding PUT, DELETE, and GET operations refer to those performed on different objects in a bucket. If concurrent operations are performed on the same object, bucket performance is determined by processing capabilities of one or several specific disks.

Performance Tuning for Scenarios of Intensive PUT or DELETE Operations

In typical OBS application scenarios such as archiving and backup, there are a large number of concurrent PUT operations. To tune bucket performance, particularly in scenarios where the object size is small, the following methods are recommended:

1. Balancing loads among multiple buckets

Upload backup data to different buckets in OBS to reduce data hotspots caused by concurrent PUT operations on the same bucket. Buckets are independent logical entities in OBS and have their own metadata. Therefore, performance of a single bucket is not affected when concurrent operations are performed on different buckets. In addition, data uploaded to different buckets can be logically separated, which improves data listing and query performance.

2. Controlling the number of concurrent operations on a bucket

Solely increasing the number of concurrent operations on a bucket cannot improve the performance of the bucket, but may lead to operation conflicts that cause performance deterioration. In OBS, you are advised to limit the number of concurrent operations on a bucket to 70. This value has been proved by tests of uploading small objects concurrently to a bucket. If the object size is large, you can increase this value to a larger one based on bucket's TPS to achieve optimal bucket performance.

3. Uploading small objects after packaging them

In scenarios where there are a large number of small objects, to improve bucket performance and storage space utilization, upload small objects after packaging them.

4. Using random object names

You can use random object names to reduce impact on a single node caused by conflicts of concurrent operations. For example, in a typical scenario of log archiving, the names of objects to be uploaded are as follows:

yourbucket/obslog/20140610-01.log.tar.gz

yourbucket/obslog/20140610-02.log.tar.gz

yourbucket/obslog/20140610-03.log.tar.gz

yourbucket/obslog/20140610-04.log.tar.gz

...

yourbucket/obslog/20140611-01.log.tar.gz

yourbucket/obslog/20140611-02.log.tar.gz

yourbucket/obslog/20140611-03.log.tar.gz

yourbucket/obslog/20140611-04.log.tar.gz

The names of these objects have the same prefix. When being uploaded concurrently, these objects are very likely to be routed to the same node. To reduce data hotspots and improve bucket performance, add random prefixes to object names. For example, a 6-digit hexadecimal string is added to each of the following object names:

yourbucket/6ac3e2-obslog/20140610-01.log.tar.gz

yourbucket/b425da-obslog/20140610-02.log.tar.gz

yourbucket/17fe36-obslog/20140610-03.log.tar.gz

yourbucket/ac96b4-obslog/20140610-04.log.tar.gz

...

yourbucket/95d36e-obslog/20140611-01.log.tar.gz

yourbucket/4a5ca2-obslog/20140611-02.log.tar.gz

yourbucket/ea2cfe-obslog/20140611-03.log.tar.gz

yourbucket/ba36c2-obslog/20140611-04.log.tar.gz

NOTE:

This method is applicable only when a bucket contains sufficient objects.

5. Managing bucket properties

OBS delivers high performance and reliability based on distributed architecture. To achieve the high performance, OBS uses cache widely, and bucket attributes are part of the cache content. Therefore, modifications to bucket attributes cannot take effect immediately, which will be effective in approximately 10 seconds at most. Be aware of this point when developing applications.

Registration