Skip to content

Commit 6261679

Browse files
committed
refine event address idx
1 parent 3112d56 commit 6261679

File tree

13 files changed

+42
-64
lines changed

13 files changed

+42
-64
lines changed

DESIGN.md

Lines changed: 8 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -51,7 +51,7 @@ The VC-Ethereum Data Service architecture streamlines the retrieval, processing,
5151
- **Documentation**: For more details on different JSON RPC calls supported, please refer to the [Ethereum JSON-RPC](https://ethereum.org/en/developers/docs/apis/json-rpc/).
5252

5353
**Bootstrapper**
54-
- **Role**: Retrieves historical block data from the Ethereum Node using HTTPS RPC.
54+
- **Role**: Retrieves the most recent 50 block data from the Ethereum Node using HTTPS RPC.
5555
- **Flow**: Initiates an HTTPS request to fetch the latest 50 blocks for initial synchronization.
5656
- **Documentation**: For more details, please refer to the [bootstrapper README](https://github.com/srinathln7/ethereum-data-service/tree/main/internal/services/bootstrapper).
5757

@@ -93,7 +93,13 @@ The VC-Ethereum Data Service architecture streamlines the retrieval, processing,
9393
- **Flow**: Provides real-time GUI monitoring of Redis operations, complementing `redis-cli`.
9494

9595

96-
## Remark
96+
## Remarks
97+
98+
### TTL for Blockdata in Redis
99+
100+
In the design, we start the **Bootstrapper** and **BlockNotification** services simultaneously. The Bootstrapper fetches the latest block height (`h`), retrieves data from `h-50` to `h`, and exits gracefully. This process takes about 5 minutes, so we set the TTL for block info in Redis to 650 seconds (50 * 13 seconds, the average ETH block time). Concurrently, the block notifier publishes real-time block info to Redis with the same TTL. Initially, our datastore holds more than 50 blocks, but it eventually stabilizes at 50 blocks. Despite the initial load, Redis memory usage remains well within its capabilities.
101+
102+
#### Data Formatter
97103

98104
The `Data Formatter` module integrates as a component rather than a standalone service, ensuring uniform data formatting across Bootstrapper and Block Subscriber. This approach maximizes code reuse and data integrity within the VC-Ethereum Data Service architecture.
99105

README.md

Lines changed: 4 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -1,6 +1,6 @@
11
# VC-Ethereum Data-API-Service
22

3-
The VC-Ethereum Data-API-Service provides essential APIs to retrieve block, transaction, and event data for the most recent 50 blocks from the Ethereum blockchain
3+
VC-Ethereum Data-API-Service stores block info, transaction hashes, and events for the latest 50 blocks and provides API endpoints for querying events by address, blocks by number, and transactions by hash.
44

55
## Objective
66

@@ -100,19 +100,19 @@ host: redis
100100
port: 6379
101101
```
102102

103-
After continuously running the application for several hours, we monitored Redis using the Redis-Insight tool and observed that Redis peak memory consumption reached approximately 50MB. The number of indexed keys maintained averaged between 35,000 to 40,000, with Redis utilizing less than 1% of CPU resources. This performance is well within Redis's capabilities, as outlined in the official [Redis FAQ](https://redis.io/docs/latest/develop/get-started/faq/#:~:text=Redis%20can%20handle%20up%20to), which states Redis can manage up to 2^32 keys and has been tested to handle at least 250 million keys per instance.
103+
After continuously running the application for several hours, we monitored Redis using the Redis-Insight tool and observed that Redis peak memory consumption reached approximately 50MB. The number of indexed keys maintained averaged between 30,000 to 40,000, with Redis utilizing less than 1% of CPU resources. This performance is well within Redis's capabilities, as outlined in the official [Redis FAQ](https://redis.io/docs/latest/develop/get-started/faq/#:~:text=Redis%20can%20handle%20up%20to), which states Redis can manage up to 2^32 keys and has been tested to handle at least 250 million keys per instance.
104104

105105
This validates that Redis is an optimal choice for our current project requirements.
106106

107107
## Limitations and Future Improvements
108108

109109
In our current architecture, each service operates with a single instance, making each service vulnerable to being a single point of failure. This becomes particularly critical because if Redis experiences downtime, it impacts the entire application. To address this, we can implement well-known strategies such as deploying Redis in a high-availability configuration using Redis Sentinel or Redis Cluster. Additionally, adopting container orchestration platforms like Kubernetes can enable automatic scaling and resilience by managing multiple instances of each service. Implementing load balancing across these instances can further enhance availability and fault tolerance and incorporating monitoring and alerting mechanisms helps in promptly identifying and mitigating issues before they impact the entire system. These approaches collectively aim to enhance the reliability and availability of our application architecture.
110110

111-
As part of future improvements, we can consider the following tasks:
111+
As part of future improvements, we consider the following tasks:
112112

113113
### Easy-to-Query APIs
114114

115-
To expose the stored data to customers in an easy-to-query API, I would consider implementing a GraphQL API (although I have worked with GraphQL in the past) on top of the existing API service. GraphQL provides a flexible and efficient way to query and retrieve data, allowing clients to request only the data they need.
115+
To expose the stored data to customers in an easy-to-query API, I would consider implementing a GraphQL API on top of the existing API service. GraphQL provides a flexible and efficient way to query and retrieve data, allowing clients to request only the data they need.
116116

117117
### API Security
118118

api/v1/handlers.go

Lines changed: 3 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -3,6 +3,7 @@ package v1
33
import (
44
"ethereum-data-service/internal/storage"
55
"net/http"
6+
"strings"
67

78
"github.com/gin-gonic/gin"
89
"github.com/redis/go-redis/v9"
@@ -51,7 +52,8 @@ func getEvents(rdb *redis.Client) gin.HandlerFunc {
5152
return
5253
}
5354

54-
events, err := storage.GetEventsByAddress(rdb, address)
55+
// We indexed address by first converting it to lower case to eliminate case sensitivity wrt. to address
56+
events, err := storage.GetEventsByAddress(rdb, strings.ToLower(address))
5557
if err != nil {
5658
c.JSON(http.StatusInternalServerError, gin.H{"error": "failed to get events from Redis", "details": err.Error()})
5759
return

docs/html/bootstrap.html

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -162,7 +162,7 @@ <h2 id="RunBootstrapSvc">func <a href="/src/ethereum-data-service/internal/servi
162162

163163

164164

165-
<h2 id="loadRecentBlockData">func <a href="/src/ethereum-data-service/internal/services/bootstrapper/bootstrap.go?s=1517:1636#L36">loadRecentBlockData</a>
165+
<h2 id="loadRecentBlockData">func <a href="/src/ethereum-data-service/internal/services/bootstrapper/bootstrap.go?s=1597:1716#L37">loadRecentBlockData</a>
166166
<a class="permalink" href="#loadRecentBlockData">&#xb6;</a>
167167

168168

docs/html/config.html

Lines changed: 4 additions & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -148,7 +148,7 @@ <h3>Package files</h3>
148148

149149

150150

151-
<h2 id="Config">type <a href="/src/ethereum-data-service/internal/config/config.go?s=86:1455#L1">Config</a>
151+
<h2 id="Config">type <a href="/src/ethereum-data-service/internal/config/config.go?s=86:1383#L1">Config</a>
152152
<a class="permalink" href="#Config">&#xb6;</a>
153153

154154

@@ -160,15 +160,11 @@ <h2 id="Config">type <a href="/src/ethereum-data-service/internal/config/config.
160160

161161
<span id="Config.API_PORT"></span> <span class="comment">// API_PORT is the port on which the API server will listen.</span>
162162
API_PORT <a href="/pkg/builtin/#string">string</a>
163-
<span id="Config.API_STATIC_FILE"></span> <span class="comment">// API_STATIC_FILE is the path to static files served by the API server.</span>
164-
API_STATIC_FILE <a href="/pkg/builtin/#string">string</a>
165163

166164
<span id="Config.ETH_HTTPS_URL"></span> <span class="comment">// ETH_HTTPS_URL is the HTTPS URL for accessing the Ethereum network.</span>
167165
ETH_HTTPS_URL <a href="/pkg/builtin/#string">string</a>
168166
<span id="Config.ETH_WSS_URL"></span> <span class="comment">// ETH_WSS_URL is the WebSocket URL for accessing the Ethereum network.</span>
169167
ETH_WSS_URL <a href="/pkg/builtin/#string">string</a>
170-
<span id="Config.ETH_AVG_BLOCK_TIME"></span> <span class="comment">// ETH_AVG_BLOCK_TIME is the average time duration (seconds) between Ethereum blocks.</span>
171-
ETH_AVG_BLOCK_TIME <a href="/pkg/time/">time</a>.<a href="/pkg/time/#Duration">Duration</a>
172168

173169
<span id="Config.REDIS_DB"></span> <span class="comment">// REDIS_DB is the Redis database number to use.</span>
174170
REDIS_DB <a href="/pkg/builtin/#int">int</a>
@@ -177,6 +173,8 @@ <h2 id="Config">type <a href="/src/ethereum-data-service/internal/config/config.
177173
<span id="Config.REDIS_PUBSUB_CH"></span> <span class="comment">// REDIS_PUBSUB_CH is the Redis Pub/Sub channel name for messaging.</span>
178174
REDIS_PUBSUB_CH <a href="/pkg/builtin/#string">string</a>
179175
<span id="Config.REDIS_KEY_EXPIRY_TIME"></span> <span class="comment">// REDIS_KEY_EXPIRY_TIME is the default expiration time (seconds) for keys stored in Redis.</span>
176+
<span class="comment">// It is calculated based on Ethereum avg block time (~13s). Currently set to 50*13=650s since</span>
177+
<span class="comment">// we need to store info only for about 50 blocks</span>
180178
REDIS_KEY_EXPIRY_TIME <a href="/pkg/time/">time</a>.<a href="/pkg/time/#Duration">Duration</a>
181179

182180
<span id="Config.NUM_BLOCKS_TO_SYNC"></span> <span class="comment">// NUM_BLOCKS_TO_SYNC is the number of recent blocks to sync during initialization.</span>
@@ -198,7 +196,7 @@ <h2 id="Config">type <a href="/src/ethereum-data-service/internal/config/config.
198196

199197

200198

201-
<h3 id="LoadConfig">func <a href="/src/ethereum-data-service/internal/config/config.go?s=1457:1491#L32">LoadConfig</a>
199+
<h3 id="LoadConfig">func <a href="/src/ethereum-data-service/internal/config/config.go?s=1385:1419#L30">LoadConfig</a>
202200
<a class="permalink" href="#LoadConfig">&#xb6;</a>
203201

204202

docs/html/storage.html

Lines changed: 3 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -265,7 +265,7 @@ <h2 id="GetTransactionByHash">func <a href="/src/ethereum-data-service/internal/
265265

266266

267267

268-
<h2 id="IdxBlockAndStore">func <a href="/src/ethereum-data-service/internal/storage/index.go?s=230:346#L4">IdxBlockAndStore</a>
268+
<h2 id="IdxBlockAndStore">func <a href="/src/ethereum-data-service/internal/storage/index.go?s=241:357#L5">IdxBlockAndStore</a>
269269
<a class="permalink" href="#IdxBlockAndStore">&#xb6;</a>
270270

271271

@@ -281,7 +281,7 @@ <h2 id="IdxBlockAndStore">func <a href="/src/ethereum-data-service/internal/stor
281281

282282

283283

284-
<h2 id="IdxEventsAndStore">func <a href="/src/ethereum-data-service/internal/storage/index.go?s=1425:1542#L32">IdxEventsAndStore</a>
284+
<h2 id="IdxEventsAndStore">func <a href="/src/ethereum-data-service/internal/storage/index.go?s=1436:1553#L33">IdxEventsAndStore</a>
285285
<a class="permalink" href="#IdxEventsAndStore">&#xb6;</a>
286286

287287

@@ -297,7 +297,7 @@ <h2 id="IdxEventsAndStore">func <a href="/src/ethereum-data-service/internal/sto
297297

298298

299299

300-
<h2 id="IdxTxAndStore">func <a href="/src/ethereum-data-service/internal/storage/index.go?s=803:916#L17">IdxTxAndStore</a>
300+
<h2 id="IdxTxAndStore">func <a href="/src/ethereum-data-service/internal/storage/index.go?s=814:927#L18">IdxTxAndStore</a>
301301
<a class="permalink" href="#IdxTxAndStore">&#xb6;</a>
302302

303303

internal/config/README.md

Lines changed: 1 addition & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -13,14 +13,12 @@ Holds configuration settings for the application.
1313
- **Fields**:
1414
- `DEFAULT_TIMEOUT time.Duration`: Default timeout for network requests.
1515
- `API_PORT string`: Port for the API server.
16-
- `API_STATIC_FILE string`: Path to static files served by the API server.
1716
- `ETH_HTTPS_URL string`: HTTPS URL for accessing the Ethereum network.
1817
- `ETH_WSS_URL string`: WebSocket URL for accessing the Ethereum network.
19-
- `ETH_AVG_BLOCK_TIME time.Duration`: Average time between Ethereum blocks.
2018
- `REDIS_DB int`: Redis database number to use.
2119
- `REDIS_ADDR string`: Address of the Redis server.
2220
- `REDIS_PUBSUB_CH string`: Redis Pub/Sub channel name for messaging.
23-
- `REDIS_KEY_EXPIRY_TIME time.Duration`: Expiration time for keys stored in Redis.
21+
- `REDIS_KEY_EXPIRY_TIME time.Duration`: Expiration time for keys stored in Redis and is calculated based on avg. ETH block time.
2422
- `NUM_BLOCKS_TO_SYNC int`: Number of recent blocks to sync during initialization.
2523
- `BOOTSTRAP_TIMEOUT time.Duration`: Time after which the bootstrap service exits itself gracefully.
2624

internal/config/config.go

Lines changed: 7 additions & 16 deletions
Original file line numberDiff line numberDiff line change
@@ -12,15 +12,11 @@ type Config struct {
1212

1313
// API_PORT is the port on which the API server will listen.
1414
API_PORT string
15-
// API_STATIC_FILE is the path to static files served by the API server.
16-
API_STATIC_FILE string
1715

1816
// ETH_HTTPS_URL is the HTTPS URL for accessing the Ethereum network.
1917
ETH_HTTPS_URL string
2018
// ETH_WSS_URL is the WebSocket URL for accessing the Ethereum network.
2119
ETH_WSS_URL string
22-
// ETH_AVG_BLOCK_TIME is the average time duration (seconds) between Ethereum blocks.
23-
ETH_AVG_BLOCK_TIME time.Duration
2420

2521
// REDIS_DB is the Redis database number to use.
2622
REDIS_DB int
@@ -29,6 +25,8 @@ type Config struct {
2925
// REDIS_PUBSUB_CH is the Redis Pub/Sub channel name for messaging.
3026
REDIS_PUBSUB_CH string
3127
// REDIS_KEY_EXPIRY_TIME is the default expiration time (seconds) for keys stored in Redis.
28+
// It is calculated based on Ethereum avg block time (~13s). Currently set to 50*13=650s since
29+
// we need to store info only for about 50 blocks
3230
REDIS_KEY_EXPIRY_TIME time.Duration
3331

3432
// NUM_BLOCKS_TO_SYNC is the number of recent blocks to sync during initialization.
@@ -42,8 +40,8 @@ type Config struct {
4240
func LoadConfig() (*Config, error) {
4341
requiredKeys := []string{
4442
"DEFAULT_TIMEOUT",
45-
"API_PORT", "API_STATIC_FILE",
46-
"ETH_HTTPS_URL", "ETH_WSS_URL", "ETH_AVG_BLOCK_TIME",
43+
"API_PORT",
44+
"ETH_HTTPS_URL", "ETH_WSS_URL",
4745
"REDIS_ADDR", "REDIS_DB", "REDIS_PUBSUB_CH", "REDIS_KEY_EXPIRY_TIME",
4846
"NUM_BLOCKS_TO_SYNC", "BOOTSTRAP_TIMEOUT",
4947
}
@@ -58,11 +56,6 @@ func LoadConfig() (*Config, error) {
5856
return nil, err
5957
}
6058

61-
avgBlockTime, err := strconv.Atoi(envMap["ETH_AVG_BLOCK_TIME"])
62-
if err != nil {
63-
return nil, err
64-
}
65-
6659
rdb, err := strconv.Atoi(envMap["REDIS_DB"])
6760
if err != nil {
6861
return nil, err
@@ -86,12 +79,10 @@ func LoadConfig() (*Config, error) {
8679
return &Config{
8780
DEFAULT_TIMEOUT: time.Duration(defaultTimeout) * time.Second,
8881

89-
API_PORT: envMap["API_PORT"],
90-
API_STATIC_FILE: envMap["API_STATIC_FILE"],
82+
API_PORT: envMap["API_PORT"],
9183

92-
ETH_HTTPS_URL: envMap["ETH_HTTPS_URL"],
93-
ETH_WSS_URL: envMap["ETH_WSS_URL"],
94-
ETH_AVG_BLOCK_TIME: time.Duration(avgBlockTime) * time.Second,
84+
ETH_HTTPS_URL: envMap["ETH_HTTPS_URL"],
85+
ETH_WSS_URL: envMap["ETH_WSS_URL"],
9586

9687
REDIS_DB: rdb,
9788
REDIS_KEY_EXPIRY_TIME: time.Duration(expiryTime) * time.Second,

internal/services/bootstrapper/README.md

Lines changed: 1 addition & 9 deletions
Original file line numberDiff line numberDiff line change
@@ -39,15 +39,7 @@ This function fetches the most recent Ethereum blocks and stores them in Redis.
3939
4. For each block:
4040
- Retrieves the block data.
4141
- Formats the block data.
42-
- Calculates an expiry time for storing the block in Redis.
43-
- Stores the block data in Redis with the calculated expiry time.
42+
- Stores the block data in Redis with the pre-defined expiry time.
4443
5. Logs the successful loading of blocks into Redis.
4544

46-
## Configuration
47-
48-
### config.Config
49-
50-
- `BOOTSTRAP_TIMEOUT`: The timeout duration for the bootstrap service.
51-
- `NUM_BLOCKS_TO_SYNC`: The number of recent blocks to fetch and store.
52-
- `ETH_AVG_BLOCK_TIME`: The average time taken to produce a new block in Ethereum (used to calculate expiry times).
5345

internal/services/bootstrapper/bootstrap.go

Lines changed: 2 additions & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -36,6 +36,7 @@ func RunBootstrapSvc(client *client.Client, cfg *config.Config) {
3636
totalTime := time.Since(startTime)
3737
log.Printf("Bootstrapper successfully completed in %s", totalTime)
3838

39+
// Once the bootstrapper finished loading 50 blocks, it shuts down succesfully
3940
log.Println("Shutting down bootstraper gracefully...")
4041
os.Exit(0)
4142

@@ -68,12 +69,7 @@ func loadRecentBlockData(ctx context.Context, ethClient *ethclient.Client, rdb *
6869
return err
6970
}
7071

71-
// Calculate expiry time for each block data
72-
// For example: Assume blocks are [B0....B49]. B0 data will expire after the first 13 seconds, B1 after the next 13 seconds, and so on.
73-
// On average, it takes 12 seconds to produce a new block in Ethereum. We set the expiry to 13 seconds to provide a safety margin.
74-
expiryTime := time.Duration((n - i + 1)) * cfg.ETH_AVG_BLOCK_TIME
75-
76-
err = storage.AddBlockDataToDB(ctx, rdb, blockDataInBytes, expiryTime)
72+
err = storage.AddBlockDataToDB(ctx, rdb, blockDataInBytes, cfg.REDIS_KEY_EXPIRY_TIME)
7773
if err != nil {
7874
return err
7975
}

internal/storage/README.md

Lines changed: 1 addition & 8 deletions
Original file line numberDiff line numberDiff line change
@@ -56,7 +56,7 @@ This function indexes each transaction by its hash in Redis.
5656

5757
### IdxEventsAndStore
5858

59-
This function indexes each event by its address in Redis.
59+
This function indexes each event by its address in Redis. Before indexing, all addresses are converted to lower case to eliminate any case sensitivity.
6060

6161
- **Parameters**:
6262
- `ctx context.Context`: The context for managing cancellation.
@@ -132,13 +132,6 @@ This function retrieves all block numbers stored in Redis.
132132
- `REDIS_PUBSUB_CH`: The Redis channel for publishing block data.
133133
- `REDIS_KEY_EXPIRY_TIME`: The expiry time for storing block data in Redis.
134134

135-
## Dependencies
136-
137-
- `github.com/redis/go-redis/v9`: Redis client library.
138-
- `encoding/json`: Package for JSON serialization and deserialization.
139-
- `github.com/ethereum/go-ethereum/core/types`: Ethereum types library.
140-
- `context`: Package for managing cancellation and timeouts.
141-
- `log`: Package for logging.
142135

143136
## Usage
144137

internal/storage/index.go

Lines changed: 5 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -5,6 +5,7 @@ import (
55
"encoding/json"
66
"ethereum-data-service/internal/model"
77
"fmt"
8+
"strings"
89
"time"
910

1011
"github.com/redis/go-redis/v9"
@@ -47,8 +48,10 @@ func IdxEventsAndStore(ctx context.Context, rdb *redis.Client, blockData *model.
4748
return fmt.Errorf("error marshalling event %+v: %v", event, err)
4849
}
4950
// For ex key:`event:0x6000da47483062A0D734Ba3dc7576Ce6A0B645C4_20167294_0xd59016cbaf7c580e83544ac5bd98584f7ec65b6984ddbd6a7647d6873c16f63a_234`
50-
// This is done to keep the key associated with every event unique
51-
addressKey := fmt.Sprint(EVENT_PREFIX, event.Address.Hex(), "_", event.BlockNumber, "_", txHash, "_", event.Index)
51+
// This is done to keep the key associated with every event unique. To eliminate any case senstivity, wrt to address we convert all address
52+
// to lower case before indexing. For ex: Addr `0x0C04fF41b11065EEd8c9EDA4d461BA6611591395` and `0x0C04ff41b11065eed8c9eda4d461ba6611591395`
53+
// all point to the same account. We do the same in the API call as well.
54+
addressKey := fmt.Sprint(EVENT_PREFIX, strings.ToLower(event.Address.Hex()), "_", event.BlockNumber, "_", txHash, "_", event.Index)
5255
if err := rdb.Set(ctx, addressKey, eventJSON, expiryTime).Err(); err != nil {
5356
return fmt.Errorf("error storing event %s_%d in Redis: %v", event.Address, event.TxIndex, err)
5457
}

sample.env

Lines changed: 2 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -3,19 +3,18 @@ DEFAULT_TIMEOUT=5 #seconds
33

44
# api-server
55
API_PORT=8080
6-
API_STATIC_FILE=./api/static
76

87
# ethereum-client
98
ETH_HTTPS_URL=https://mainnet.ethereum.validationcloud.io/v1/JFt58zlN7gcQlLYnMZcOD75LpethJgD6Eq5nKOxC9F0
109
ETH_WSS_URL=wss://mainnet.ethereum.validationcloud.io/v1/wss/JFt58zlN7gcQlLYnMZcOD75LpethJgD6Eq5nKOxC9F0
11-
ETH_AVG_BLOCK_TIME=13
10+
1211

1312

1413
# redis-client
1514
REDIS_ADDR=localhost:6379
1615
REDIS_DB=0
1716
REDIS_PUBSUB_CH=ETH_MAINNET
18-
REDIS_KEY_EXPIRY_TIME=650 # seconds
17+
REDIS_KEY_EXPIRY_TIME=650 # 50 * ETH_AVG_BLOCK_TIME (13s) ~ 650s
1918

2019

2120
# bootstraper-service

0 commit comments

Comments
 (0)