The JingQi Hadoop® platform was developed based on open-source Apache Hadoop and two softwares were put forward for reliable, scalable, distributed computing.
The JingQi Hadoop software library open two products: (1) Data Mining for Petroleum Exploration and Development; (2) Big Data Analysis for Co-evolving Data Streams. Both software allows for the distributed processing of large data sets across clusters of computers using simple programming models. It is designed to scale up from single server to thousands of machines, each offering local computation and storage. Rather than rely on hardware to deliver high-availability, the two products itself is designed to detect and handle failures at the application layer, so delivering a highly-available service on top of a cluster of computers, each of which may be prone to failures. The two productions support the following modules: Hadoop Common: The common utilities that support the other Hadoop modules. Hadoop Distributed File System (HDFS™): A distributed file system that provides high-throughput access to application data. Hadoop YARN: A framework for job scheduling and cluster resource management. Hadoop MapReduce: A YARN-based system for parallel processing of large data sets.
News 18 November, 2015: release 5.0 JingQi Hadoop DM available JingQi Hadoop for data mining project with petroleum exploration and development contains a number of significant enhancements such as:
·Hadoop Common oKey management server (beta) oCredential provider (beta) ·Hadoop HDFS oHeterogeneous Storage Tiers - Phase 2
·Application APIs for heterogeneous storage
·SSD storage tier
·Memory as a storage tier (beta)
oSupport for Archival Storage
oTransparent data at rest encryption (beta)
oOperating secure DataNode without requiring root access
oHot swap drive: support add/remove data node volumes without restarting data node (beta)
oAES support for faster wire encryption
·Hadoop YARN
oSupport for long running services in YARN
·Service Registry for applications
oSupport for rolling upgrades
·Work-preserving restarts of ResourceManager
·Container-preserving restart of NodeManager
oSupport node labels during scheduling
oSupport for time-based resource reservations in Capacity Scheduler(beta)
oGlobal, shared cache for application artifacts (beta)
oSupport running of applications natively in Docker containers(alpha)
May 2014 - Avro and HBase were integrated into Big Data Analysis project.
Big Data Analysis project for abnormality detection with co-evolving data streams was developed based on Hadoop's Avro and HBase. It has graduated to become top-level JingQi Hadoop projects.