一种中尺度数值预报模式数据并行传输应用研究
Parallel Transmission of Data from a Mesoscale Numerical Prediction Model: An Application Study
中文摘要：
以重庆中尺度数值预报模式为例，设计开发一种针对中尺度数值预报模式海量数据的并行传输程序，提高数据传输的效率。重庆中尺度数值预报模式每天输出的大量数据需传输到数据管理服务器处理和备份。目前模式的数据传输任务是以scp单核串行程序为主，数据传输时间长，不能充分利用处理器核和网络带宽，未有效满足重庆天气预报业务对模式数据及时使用的需求；设计开发的并行传输程序以进程池为主要架构，采用Rsync代替scp提高模式数据传输速率；同时，基于输出的模式数据的特征实现多种数据分治策略，对比选择最佳的数据分治策略；结果表明基于最佳数据分治策略的并行传输程序能够极大减少数据传输时间，极大利用空闲的处理器核和网络带宽资源。
中文关键词：
数值预报模式，并行传输，数据分治策略，Rsync
KeyWords：
numerical prediction model, parallel transmission, data partition strategy, Rsync
Abstract：
Taking the Chongqing Mesoscale Numerical Prediction model as an example, the authors designed and developed a parallel transmission program for large data of the mesoscale numerical prediction model to improve transmission efficiency. The large dataset the Chongqing mesoscale model produced has been transferred to the data management server for every day processing and analysis. The one-core serial program with scp was currently used to complete the task. Because it took more time and could not make full use of cpu cores and network bandwidth, this program could not meet the demand of timely use of Chongqing Weather Forecasting. The parallel transmission program takes process pool as main architecture, and adopts Rsync instead of scp to improve the data transmission speed. Meanwhile the model data is divided on the basis of data features, and the best data partition strategy would be chosen by comparison. The results showed that the parallel transmission program with the best data partition strategy can greatly reduce the data transmission time, and effectively improve the availability of cpu cores and network bandwidth.