面向大数据的北京水务数据融合技术研究

Research on big data oriented data fusion technology of Beijing water affair

摘要:

为解决北京水务数据资源面临多源异构数据难以分析及数据集成困难等难题，在分析北京水务数据资源的基础上，采用大数据和云计算等相关技术进行北京水务数据资源的有效融合。针对北京水务的结构化及非结构化数据，研究相应的数据抽取、转换及存储的技术，提出基于水务数据资源融合的技术架构，其中结构化数据抽取使用 D2RQ 工具，非结构化数据的抽取使用 jieba 分词工具及 tf-idf 权重算法，并进行实验验证，证明该套技术方案的可行性与可信度，在数据存储模块采用基于云计算的分布式数据存储技术，用于存储融合后的海量数据。 研究的数据资源融合技术方案能够帮助提高北京水务数据资源的融合效率与应用能力。

Abstract:

In order to solve the problem of diffificult analysis and data integration of multi-source heterogeneous data in Beijing water resources, based on the analysis of Beijing water resources, big data and cloud computing technologies are adopted to effectively integrate Beijing water resources. In view of the structured and unstructured data in Beijing, the corresponding data extraction, transformation and storage technology, the technical architecture of data based on water resources integration, including structured data extraction using D2RQ tools, unstructured data extraction using jieba segmentation tools and tf- idf weight algorithm are researched, and experimental verifification is made, to prove the feasibility of this set of technical solutions and credibility. Furthermore, the data storage module uses the distributed data storage based on cloud computing technology, after fusion is used to store huge amounts of data. The technical scheme of data resource fusion can help improve the effificiency of data resource fusion and the application ability of data resource.