基于Alluxio的新闻文本存储优化方法
An optimization method of news text storage based on Alluxio
投稿时间: 2023/12/20 0:00:00
DOI:
中文关键词: 分布式系统;新闻数据;小文件存储;缓存;Alluxio
英文关键词: distributed system; news text; small files storage; cache; Alluxio
基金项目:
姓名 单位
桂静 中国传媒大学媒体融合与传播国家重点实验室
王永滨 中国传媒大学媒体融合与传播国家重点实验室
点击数:217 下载数:153
中文摘要:

新闻数据的指数级增长对新闻的高效存储和管理提出了重大挑战。为了解决存储与计算分离模式下新闻文本的存储与读访问瓶颈问题,提出了一种基于数据编排平台Alluxio的新闻文本存储优化方法。首先,利用Alluxio对不同存储系统中的新闻文本进行缓存并提供统一命名空间,加速了计算应用的数据访问。其次,为了解决新闻文本存储在远程场景下所面临的读访问性能瓶颈问题,对新闻文本进行了基于目录聚合的合并,并利用最小完美哈希算法对新闻文本元数据构建索引,实现了新闻文本的快速检索。

英文摘要:

The exponential growth of news data has presented a significant challenge to efficiently storing and managing news. To address the bottleneck issues of news text storage and access in the separation mode of storage and computation, we proposed an optimization method for news text storage based on Alluxio, a data orchestration platform. Firstly, Alluxio was utilized to cache news texts across various storage systems and provided a unified namespace, enhancing data access for computational applications. Secondly, to overcome performance limitations in reading and accessing remotely stored news text, we incorporated directory aggregation for merging the news text and emploied the Minimum Perfect Hash algorithm to index metadata associated with the news text. This enables rapid retrieval of relevant information from the stored news texts.

参考文献: