Abstract:In the current intelligent coal mine scenario, the time-series data generated by a large number of coal mine microseismic sensors is growing explosively, which in turn has higher requirements for the existing storage system and performance. There are already instances where HBase, a distributed column family database, can store industrial time-series big data, the existing strategies are still unable to better meet the needs of coal mine microseismic waveform time-series data because they do not consider the feature association of data in specific business scenarios. specific storage requirements. In view of the above problems, based on the distributed storage system HBase, using the characteristics of coal mine microseismic waveform time series data, a coal mine microseismic time series big data storage performance optimization (CM2TS-HBase) based on HBase and Netty is proposed, which is divided into four parts, namely data acquisition layer, data preprocessing layer, data transfer layer, and data storage layer. Among them, the data acquisition layer is divided into offline part and real-time part. The offline part is the historical microseismic time series data files stored in the hard disk of the data center, and the real-time part is the multiple microseismic waveform sensors deployed in a coal mine. The layer performs data buffering; the data preprocessing layer implements alignment, parsing, and serialization operations on files of waveform time series data. According to the characteristics of coal mine microseismic waveform time series data, the HBase data table structure, pre-partition strategy and primary key optimization strategy suitable for microseismic waveform time series data are proposed, which effectively solves the data hot spot problem and data dispersion problem in the data storage process; data transfer layer A data forwarding middleware platform based on Netty and Redis is proposed to provide an asynchronous processing mechanism for the entire storage system, which better solves the problem of high concurrent storage; the data storage layer is based on the distributed database HBase as the underlying storage medium of the storage system. Finally, according to the time-consuming storage of real data sets, it is proved that compared with the native storage method (HBase API) and financial time-series data storage optimization (FT -HBase), CM2TS -HBase has significantly improved the storage performance of coal mine microseismic time-series data.
