You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Copy file name to clipboardExpand all lines: DESIGN.md
+6-7Lines changed: 6 additions & 7 deletions
Original file line number
Diff line number
Diff line change
@@ -53,12 +53,12 @@ The VC-Ethereum Data Service architecture streamlines the retrieval, processing,
53
53
54
54
**Bootstrapper**
55
55
-**Role**: Retrieves the most recent 50 block data from the Ethereum Node using HTTPS RPC.
56
-
-**Flow**: Initiates an HTTPS request to fetch the latest 50 blocks for initial synchronization.
56
+
-**Flow**: Initiates an HTTPS request to fetch the latest 50 blocks for initial synchronization, loads the data to Redis and then exits.
57
57
-**Documentation**: For more details, please refer to the [bootstrapper README](https://github.com/srinathln7/ethereum-data-service/tree/main/internal/services/bootstrapper).
58
58
59
59
**Block Notification**
60
60
-**Role**: Listens for new blocks in real-time via WebSocket subscription from the Ethereum Node.
61
-
-**Flow**: Establishes a bi-directional WebSocket connection to receive immediate updates on new blocks.
61
+
-**Flow**: Establishes a bi-directional WebSocket connection to receive live updates on new blocks.
62
62
-**Documentation**: For more details, please refer to the [pub README](https://github.com/srinathln7/ethereum-data-service/tree/main/internal/services/pub).
63
63
64
64
**Redis Channel**
@@ -71,13 +71,13 @@ The VC-Ethereum Data Service architecture streamlines the retrieval, processing,
71
71
-**Documentation**: For more details, please refer to the [sub README](https://github.com/srinathln7/ethereum-data-service/tree/main/internal/services/sub).
72
72
73
73
**Data Formatter**
74
-
-**Role**: Formats raw block data into a structured format using `model.Data` struct definitions.
74
+
-**Role**: Formats raw block data into a structured format as per `model.Data` struct definition.
75
75
-**Integration**: Modular integration into Bootstrapper and Block Subscriber ensures consistent data formatting before storage.
76
76
-**Documentation**: For more details, please refer to the [model README](https://github.com/srinathln7/ethereum-data-service/tree/main/internal/model).
77
77
78
78
**Redis DB**
79
79
-**Role**: Central storage for processed Ethereum blockchain data.
80
-
-**Flow**: Stores formatted block data received from Data Formatter and Block Subscriber, enabling efficient data retrieval.
80
+
-**Flow**: Stores formatted block data received from upstream services with pre-define expiry time and enables efficient data retrieval.
81
81
-**Documentation**: For more details, please refer to the [storage README](https://github.com/srinathln7/ethereum-data-service/tree/main/internal/storage).
82
82
83
83
**API Service**
@@ -90,15 +90,14 @@ The VC-Ethereum Data Service architecture streamlines the retrieval, processing,
90
90
-**Flow**: Initiates HTTP requests to API Service to fetch specific blockchain data of interest.
91
91
92
92
**Redis Insight**
93
-
-**Role**: Analyzes and visualizes data stored in Redis DB, offering insights into key creationand memory usage.
93
+
-**Role**: Analyzes and visualizes data stored in Redis DB, offering insights into key creation, memory consumption and CPU usage.
94
94
-**Flow**: Provides real-time GUI monitoring of Redis operations, complementing `redis-cli`.
95
95
96
-
97
96
## Remarks
98
97
99
98
### TTL for Blockdata in Redis
100
99
101
-
In the design, we start the **Bootstrapper** and **BlockNotification** services simultaneously. The Bootstrapper fetches the latest block height (`h`), retrieves data from `h-50` to `h`, and exits gracefully. This process takes about 5 minutes, so we set the TTL for block info in Redis to 650 seconds (50 * 13 seconds, the average ETH block time). Concurrently, the block notifier publishes real-time block info to Redis with the same TTL. Initially, our datastore holds more than 50 blocks, but it eventually stabilizes at 50 blocks. Despite the initial load, Redis memory usage remains well within its capabilities.
100
+
In the design, we start the **Bootstrapper** and **BlockNotification** services simultaneously. The Bootstrapper fetches the latest block height (`h`), retrieves data from `h-50` to `h`, and exits gracefully. We set the TTL for all these retreived block information in Redis to 650 seconds (50 * 13 seconds, the average ETH block time). Concurrently, the block notifier publishes real-time block info to Redis with the same TTL. Initially, our datastore holds more than 50 blocks, but it eventually stabilizes at 50 blocks. Despite the initial load, Redis memory usage remains well within its capabilities.
102
101
103
102
### Data Formatter
104
103
The `Data Formatter` module integrates as a component rather than a standalone service, ensuring uniform data formatting across Bootstrapper and Block Subscriber. This approach maximizes code reuse and data integrity within the VC-Ethereum Data Service architecture.
Copy file name to clipboardExpand all lines: README.md
+9-12Lines changed: 9 additions & 12 deletions
Original file line number
Diff line number
Diff line change
@@ -1,10 +1,6 @@
1
1
# VC-Ethereum Data-API-Service
2
2
3
-
VC-Ethereum Data-API-Service stores block info, transaction hashes, and events for the latest 50 blocks and provides API endpoints for querying events by address, blocks by number, and transactions by hash.
4
-
5
-
## Objective
6
-
7
-
Refer to the [project objective](https://github.com/srinathln7/ethereum-data-service/blob/main/docs/CHALLENGE.md) and the project structure. This document outlines the purpose of the project and how the codebase is organized.
3
+
VC-Ethereum Data-API-Service stores block info, transaction hashes, and events for the latest 50 blocks and provides API endpoints for querying events by address, blocks by number, and transactions by hash. Please refer to the [project objective](https://github.com/srinathln7/ethereum-data-service/blob/main/docs/CHALLENGE.md) here which outlines the goal of the project and how the codebase is organized.
8
4
9
5
## Prerequisites
10
6
@@ -46,10 +42,14 @@ make builddown
46
42
47
43
To view logs in real-time for each service:
48
44
49
-
```sh
45
+
```
46
+
50
47
docker logs -f vc-api-server
48
+
51
49
docker logs -f vc-bootstrapper
50
+
52
51
docker logs -f vc-blocknotifier
52
+
53
53
docker logs -f vc-blocksubscriber
54
54
```
55
55
@@ -92,8 +92,7 @@ Alternatively, you can test the service in your browser.
92
92
93
93
## Observation and Analysis
94
94
95
-
To view real-time analytics of our local data store, click [here](http://localhost:5540) to navigate to the Redis Insight GUI. Then, manually connect the database by clicking `Add connection details manually` and using the following details
96
-
95
+
To view real-time analytics of our local data store, navigate to http://localhost:5540 in your browser to access the Redis Insight GUI. Then, manually connect the database by clicking `Add connection details manually` and using the following details
97
96
98
97
```
99
98
host: redis
@@ -106,9 +105,7 @@ This validates that Redis is an optimal choice for our current project requireme
106
105
107
106
## Limitations and Future Improvements
108
107
109
-
In our current architecture, each service operates with a single instance, making each service vulnerable to being a single point of failure. This becomes particularly critical because if Redis experiences downtime, it impacts the entire application. To address this, we can implement well-known strategies such as deploying Redis in a high-availability configuration using Redis Sentinel or Redis Cluster. Additionally, adopting container orchestration platforms like Kubernetes can enable automatic scaling and resilience by managing multiple instances of each service. Implementing load balancing across these instances can further enhance availability and fault tolerance and incorporating monitoring and alerting mechanisms helps in promptly identifying and mitigating issues before they impact the entire system. These approaches collectively aim to enhance the reliability and availability of our application architecture.
110
-
111
-
As part of future improvements, we can consider the following tasks:
108
+
In our current architecture, each service operates with a single instance, making each service vulnerable to being a single point of failure. This becomes particularly critical because if Redis experiences downtime, it impacts the entire application. To address this, we can implement strategies such as deploying Redis in a high-availability configuration using Redis Cluster. Additionally, we can deploy these service on Kubernetes and enable automatic scaling and resilience by managing multiple instances of each service. Implementing load balancing across these instances can further enhance availability and fault tolerance and incorporating monitoring and alerting mechanisms helps in promptly identifying and mitigating issues before they impact the entire system. These approaches collectively aim to enhance the reliability and availability of our application architecture.
112
109
113
110
### Easy-to-Query APIs
114
111
@@ -122,7 +119,7 @@ To secure the API, I would implement the following measures:
122
119
123
120
-**Rate Limiting:** Implement rate limiting to prevent abuse and protect the service from being overwhelmed by too many requests.
124
121
125
-
-**HTTPS:** Use HTTPS to encrypt communication between clients and the API.
122
+
-**HTTPS:** Use HTTPS to encrypt communication between clients and our API service.
0 commit comments