Fetcher Nodes
Last updated
Last updated
Fetcher Nodes are essential components of the Agentic Data Coordination Service (ADCS), designed to intelligently retrieve, process, and aggregate data from diverse external sources. By leveraging a structured data adapter framework, Fetcher Nodes ensures that requestors have access to standardized and high-quality data structures. These nodes serve as pivotal data provision points for oracles within the ADCS system, guaranteeing the supply of reliable information necessary for efficiently fulfilling consumer requests.
The Fetcher is responsible for intelligently fetching data from designated sources based on predefined configurations. It ensures efficient and accurate data retrieval by adhering to the specifications set within the data adapter, optimizing the process for both speed and reliability.
The Data Adapter framework ensures compatibility and standardization across various data sources. Each adapter is identified by a unique adapter hash, which guarantees that the data retrieved by the Fetcher is correctly structured and compatible for aggregation. This standardization is critical for maintaining data integrity throughout the ADCS ecosystem.
The Aggregator consolidates data retrieved from multiple Fetchers, providing the final aggregated data. Each Fetcher Node maintains both a local aggregate and a global aggregate. The Aggregator processes data using methods such as median or majority voting to ensure accuracy and relevance. The Raft consensus mechanism is employed to achieve agreement among all nodes on the global aggregate, maintaining consistency and reliability across the network.
To ensure that all nodes in the network agree on the global aggregate data, ADCS utilizes the Raft consensus mechanism. This process involves several critical steps:
A leader node is elected among the Fetcher Nodes. The leader manages the log replication process and ensures synchronization across all follower nodes.
The leader proposes the global aggregate data by creating a new log entry containing the aggregated data. This log entry is then sent to all follower nodes for replication.
Each follower node receives the proposed log entry from the leader and appends it to its own log. Followers send acknowledgments back to the leader to confirm receipt and storage of the log entry.
Once the leader receives acknowledgments from a majority of the nodes, it considers the log entry committed. The leader updates the state machine and finalizes the global aggregate, ensuring that all nodes reflect the agreed-upon data.
