分级存储技术及其在水利大数据处理中的应用

Application of hierarchical storage technology in big data of water conservancy

摘要:

随着水利信息化的快速发展，水利大数据应用日益广泛，通常的存储技术已经无法满足需要，存储容量及数据读写速率已成为大数据处理发展过程中的瓶颈。与全国山洪灾害防治管理平台中的分级存储数据集群建设实践相结合，根据信息生命周期理论，应用分级存储技术与方法，再结合水利数据特性制定迁移策略，搭建分级存储集群，有效地满足对海量数据存储和处理速率的需求，打破存储对大数据处理的制约。最终，该分级存储集群能够在节省存储成本的同时有效增加存储容量并提高对数据的读写速率，从而使集群的综合性能趋于平衡。

Abstract:

With the fast development of water conservancy information technology, the application of big data technology in the water conservancy area is increasingly extensive. However, the storage requirement can no longer bemet by the conventional technology and the storage and the data write-read rate have become the bottleneck of the bigdata processing. It combines the theory of the hierarchical storage data cluster and the implementation of the National Mountain Flood Prevention and Management Platform,according to the information life cycle theory, the hierarchical storage technology and method, combining with the characteristics of water conservancy data to develop a migration strategy to build the hierarchical storage data cluster, to effectively meet the demand for massive data storage and data processing, and break the constraints of storage on big data processing. Finally, the hierarchical storage data cluster effectively cuts storage cost, increases storage capacity and improves data write-read rate, thus balances the performance of the cluster.