refine event address idx

Author: srinathln7

DESIGN.md

@@ -51,7 +51,7 @@ The VC-Ethereum Data Service architecture streamlines the retrieval, processing,
 - **Documentation**: For more details on different JSON RPC calls supported, please refer to the [Ethereum JSON-RPC](https://ethereum.org/en/developers/docs/apis/json-rpc/).
 
 **Bootstrapper**
-- **Role**: Retrieves historical block data from the Ethereum Node using HTTPS RPC.
+- **Role**: Retrieves the most recent 50 block data from the Ethereum Node using HTTPS RPC.
 - **Flow**: Initiates an HTTPS request to fetch the latest 50 blocks for initial synchronization.
 - **Documentation**: For more details, please refer to the [bootstrapper README](https://github.com/srinathln7/ethereum-data-service/tree/main/internal/services/bootstrapper).
 
@@ -93,7 +93,13 @@ The VC-Ethereum Data Service architecture streamlines the retrieval, processing,
 - **Flow**: Provides real-time GUI monitoring of Redis operations, complementing `redis-cli`.
 
 
-## Remark
+## Remarks
+
+### TTL for Blockdata in Redis
+
+In the design, we start the **Bootstrapper** and **BlockNotification** services simultaneously. The Bootstrapper fetches the latest block height (`h`), retrieves data from `h-50` to `h`, and exits gracefully. This process takes about 5 minutes, so we set the TTL for block info in Redis to 650 seconds (50 * 13 seconds, the average ETH block time). Concurrently, the block notifier publishes real-time block info to Redis with the same TTL. Initially, our datastore holds more than 50 blocks, but it eventually stabilizes at 50 blocks. Despite the initial load, Redis memory usage remains well within its capabilities.
+
+#### Data Formatter
 
 The `Data Formatter` module integrates as a component rather than a standalone service, ensuring uniform data formatting across Bootstrapper and Block Subscriber. This approach maximizes code reuse and data integrity within the VC-Ethereum Data Service architecture.
 

README.md

@@ -1,6 +1,6 @@
 # VC-Ethereum Data-API-Service
 
-The VC-Ethereum Data-API-Service provides essential APIs to retrieve block, transaction, and event data for the most recent 50 blocks from the Ethereum blockchain
+VC-Ethereum Data-API-Service stores block info, transaction hashes, and events for the latest 50 blocks and provides API endpoints for querying events by address, blocks by number, and transactions by hash.
 
 ## Objective
 
@@ -100,19 +100,19 @@ host: redis
 port: 6379
 ```
 
-After continuously running the application for several hours, we monitored Redis using the Redis-Insight tool and observed that Redis peak memory consumption reached approximately 50MB. The number of indexed keys maintained averaged between 35,000 to 40,000, with Redis utilizing less than 1% of CPU resources. This performance is well within Redis's capabilities, as outlined in the official [Redis FAQ](https://redis.io/docs/latest/develop/get-started/faq/#:~:text=Redis%20can%20handle%20up%20to), which states Redis can manage up to 2^32 keys and has been tested to handle at least 250 million keys per instance.
+After continuously running the application for several hours, we monitored Redis using the Redis-Insight tool and observed that Redis peak memory consumption reached approximately 50MB. The number of indexed keys maintained averaged between 30,000 to 40,000, with Redis utilizing less than 1% of CPU resources. This performance is well within Redis's capabilities, as outlined in the official [Redis FAQ](https://redis.io/docs/latest/develop/get-started/faq/#:~:text=Redis%20can%20handle%20up%20to), which states Redis can manage up to 2^32 keys and has been tested to handle at least 250 million keys per instance.
 
 This validates that Redis is an optimal choice for our current project requirements.
 
 ## Limitations and Future Improvements
 
 In our current architecture, each service operates with a single instance, making each service vulnerable to being a single point of failure. This becomes particularly critical because if Redis experiences downtime, it impacts the entire application. To address this, we can implement well-known strategies such as deploying Redis in a high-availability configuration using Redis Sentinel or Redis Cluster. Additionally, adopting container orchestration platforms like Kubernetes can enable automatic scaling and resilience by managing multiple instances of each service. Implementing load balancing across these instances can further enhance availability and fault tolerance and incorporating monitoring and alerting mechanisms helps in promptly identifying and mitigating issues before they impact the entire system. These approaches collectively aim to enhance the reliability and availability of our application architecture.
 
-As part of future improvements, we can consider the following tasks:
+As part of future improvements, we consider the following tasks:
 
 ### Easy-to-Query APIs
 
-To expose the stored data to customers in an easy-to-query API, I would consider implementing a GraphQL API (although I have worked with GraphQL in the past) on top of the existing API service. GraphQL provides a flexible and efficient way to query and retrieve data, allowing clients to request only the data they need.
+To expose the stored data to customers in an easy-to-query API, I would consider implementing a GraphQL API on top of the existing API service. GraphQL provides a flexible and efficient way to query and retrieve data, allowing clients to request only the data they need.
 
 ### API Security
 

api/v1/handlers.go

@@ -3,6 +3,7 @@ package v1
 import (
 	"ethereum-data-service/internal/storage"
 	"net/http"
+	"strings"
 
 	"github.com/gin-gonic/gin"
 	"github.com/redis/go-redis/v9"
@@ -51,7 +52,8 @@ func getEvents(rdb *redis.Client) gin.HandlerFunc {
 			return
 		}
 
-		events, err := storage.GetEventsByAddress(rdb, address)
+		// We indexed address by first converting it to lower case to eliminate case sensitivity wrt. to address
+		events, err := storage.GetEventsByAddress(rdb, strings.ToLower(address))
 		if err != nil {
 			c.JSON(http.StatusInternalServerError, gin.H{"error": "failed to get events from Redis", "details": err.Error()})
 			return

docs/html/bootstrap.html

@@ -162,7 +162,7 @@ <h2 id="RunBootstrapSvc">func <a href="/src/ethereum-data-service/internal/servi
 
 
 
-			<h2 id="loadRecentBlockData">func <a href="/src/ethereum-data-service/internal/services/bootstrapper/bootstrap.go?s=1517:1636#L36">loadRecentBlockData</a>
+			<h2 id="loadRecentBlockData">func <a href="/src/ethereum-data-service/internal/services/bootstrapper/bootstrap.go?s=1597:1716#L37">loadRecentBlockData</a>
 				<a class="permalink" href="#loadRecentBlockData">&#xb6;</a>
 
 

docs/html/config.html

@@ -148,7 +148,7 @@ <h3>Package files</h3>
 
 
 
-			<h2 id="Config">type <a href="/src/ethereum-data-service/internal/config/config.go?s=86:1455#L1">Config</a>
+			<h2 id="Config">type <a href="/src/ethereum-data-service/internal/config/config.go?s=86:1383#L1">Config</a>
 				<a class="permalink" href="#Config">&#xb6;</a>
 
 
@@ -160,15 +160,11 @@ <h2 id="Config">type <a href="/src/ethereum-data-service/internal/config/config.
 
 <span id="Config.API_PORT"></span>    <span class="comment">// API_PORT is the port on which the API server will listen.</span>
     API_PORT <a href="/pkg/builtin/#string">string</a>
-<span id="Config.API_STATIC_FILE"></span>    <span class="comment">// API_STATIC_FILE is the path to static files served by the API server.</span>
-    API_STATIC_FILE <a href="/pkg/builtin/#string">string</a>
 
 <span id="Config.ETH_HTTPS_URL"></span>    <span class="comment">// ETH_HTTPS_URL is the HTTPS URL for accessing the Ethereum network.</span>
     ETH_HTTPS_URL <a href="/pkg/builtin/#string">string</a>
 <span id="Config.ETH_WSS_URL"></span>    <span class="comment">// ETH_WSS_URL is the WebSocket URL for accessing the Ethereum network.</span>
     ETH_WSS_URL <a href="/pkg/builtin/#string">string</a>
-<span id="Config.ETH_AVG_BLOCK_TIME"></span>    <span class="comment">// ETH_AVG_BLOCK_TIME is the average time duration (seconds) between Ethereum blocks.</span>
-    ETH_AVG_BLOCK_TIME <a href="/pkg/time/">time</a>.<a href="/pkg/time/#Duration">Duration</a>
 
 <span id="Config.REDIS_DB"></span>    <span class="comment">// REDIS_DB is the Redis database number to use.</span>
     REDIS_DB <a href="/pkg/builtin/#int">int</a>
@@ -177,6 +173,8 @@ <h2 id="Config">type <a href="/src/ethereum-data-service/internal/config/config.
 <span id="Config.REDIS_PUBSUB_CH"></span>    <span class="comment">// REDIS_PUBSUB_CH is the Redis Pub/Sub channel name for messaging.</span>
     REDIS_PUBSUB_CH <a href="/pkg/builtin/#string">string</a>
 <span id="Config.REDIS_KEY_EXPIRY_TIME"></span>    <span class="comment">// REDIS_KEY_EXPIRY_TIME is the default expiration time (seconds) for keys stored in Redis.</span>
+    <span class="comment">// It is calculated based on Ethereum avg block time (~13s). Currently set to 50*13=650s since</span>
+    <span class="comment">// we need to store info only for about 50 blocks</span>
     REDIS_KEY_EXPIRY_TIME <a href="/pkg/time/">time</a>.<a href="/pkg/time/#Duration">Duration</a>
 
 <span id="Config.NUM_BLOCKS_TO_SYNC"></span>    <span class="comment">// NUM_BLOCKS_TO_SYNC is the number of recent blocks to sync during initialization.</span>
@@ -198,7 +196,7 @@ <h2 id="Config">type <a href="/src/ethereum-data-service/internal/config/config.
 
 
 
-				<h3 id="LoadConfig">func <a href="/src/ethereum-data-service/internal/config/config.go?s=1457:1491#L32">LoadConfig</a>
+				<h3 id="LoadConfig">func <a href="/src/ethereum-data-service/internal/config/config.go?s=1385:1419#L30">LoadConfig</a>
 					<a class="permalink" href="#LoadConfig">&#xb6;</a>
 
 

docs/html/storage.html

@@ -265,7 +265,7 @@ <h2 id="GetTransactionByHash">func <a href="/src/ethereum-data-service/internal/
 
 
 
-			<h2 id="IdxBlockAndStore">func <a href="/src/ethereum-data-service/internal/storage/index.go?s=230:346#L4">IdxBlockAndStore</a>
+			<h2 id="IdxBlockAndStore">func <a href="/src/ethereum-data-service/internal/storage/index.go?s=241:357#L5">IdxBlockAndStore</a>
 				<a class="permalink" href="#IdxBlockAndStore">&#xb6;</a>
 
 
@@ -281,7 +281,7 @@ <h2 id="IdxBlockAndStore">func <a href="/src/ethereum-data-service/internal/stor
 
 
 
-			<h2 id="IdxEventsAndStore">func <a href="/src/ethereum-data-service/internal/storage/index.go?s=1425:1542#L32">IdxEventsAndStore</a>
+			<h2 id="IdxEventsAndStore">func <a href="/src/ethereum-data-service/internal/storage/index.go?s=1436:1553#L33">IdxEventsAndStore</a>
 				<a class="permalink" href="#IdxEventsAndStore">&#xb6;</a>
 
 
@@ -297,7 +297,7 @@ <h2 id="IdxEventsAndStore">func <a href="/src/ethereum-data-service/internal/sto
 
 
 
-			<h2 id="IdxTxAndStore">func <a href="/src/ethereum-data-service/internal/storage/index.go?s=803:916#L17">IdxTxAndStore</a>
+			<h2 id="IdxTxAndStore">func <a href="/src/ethereum-data-service/internal/storage/index.go?s=814:927#L18">IdxTxAndStore</a>
 				<a class="permalink" href="#IdxTxAndStore">&#xb6;</a>
 
 

internal/config/README.md

@@ -13,14 +13,12 @@ Holds configuration settings for the application.
 - **Fields**:
   - `DEFAULT_TIMEOUT time.Duration`: Default timeout for network requests.
   - `API_PORT string`: Port for the API server.
-  - `API_STATIC_FILE string`: Path to static files served by the API server.
   - `ETH_HTTPS_URL string`: HTTPS URL for accessing the Ethereum network.
   - `ETH_WSS_URL string`: WebSocket URL for accessing the Ethereum network.
-  - `ETH_AVG_BLOCK_TIME time.Duration`: Average time between Ethereum blocks.
   - `REDIS_DB int`: Redis database number to use.
   - `REDIS_ADDR string`: Address of the Redis server.
   - `REDIS_PUBSUB_CH string`: Redis Pub/Sub channel name for messaging.
-  - `REDIS_KEY_EXPIRY_TIME time.Duration`: Expiration time for keys stored in Redis.
+  - `REDIS_KEY_EXPIRY_TIME time.Duration`: Expiration time for keys stored in Redis and is calculated based on avg. ETH block time.
   - `NUM_BLOCKS_TO_SYNC int`: Number of recent blocks to sync during initialization.
   - `BOOTSTRAP_TIMEOUT time.Duration`: Time after which the bootstrap service exits itself gracefully.
 

internal/config/config.go

@@ -12,15 +12,11 @@ type Config struct {
 
 	// API_PORT is the port on which the API server will listen.
 	API_PORT string
-	// API_STATIC_FILE is the path to static files served by the API server.
-	API_STATIC_FILE string
 
 	// ETH_HTTPS_URL is the HTTPS URL for accessing the Ethereum network.
 	ETH_HTTPS_URL string
 	// ETH_WSS_URL is the WebSocket URL for accessing the Ethereum network.
 	ETH_WSS_URL string
-	// ETH_AVG_BLOCK_TIME is the average time duration (seconds) between Ethereum blocks.
-	ETH_AVG_BLOCK_TIME time.Duration
 
 	// REDIS_DB is the Redis database number to use.
 	REDIS_DB int
@@ -29,6 +25,8 @@ type Config struct {
 	// REDIS_PUBSUB_CH is the Redis Pub/Sub channel name for messaging.
 	REDIS_PUBSUB_CH string
 	// REDIS_KEY_EXPIRY_TIME is the default expiration time (seconds) for keys stored in Redis.
+	// It is calculated based on Ethereum avg block time (~13s). Currently set to 50*13=650s since
+	// we need to store info only for about 50 blocks
 	REDIS_KEY_EXPIRY_TIME time.Duration
 
 	// NUM_BLOCKS_TO_SYNC is the number of recent blocks to sync during initialization.
@@ -42,8 +40,8 @@ type Config struct {
 func LoadConfig() (*Config, error) {
 	requiredKeys := []string{
 		"DEFAULT_TIMEOUT",
-		"API_PORT", "API_STATIC_FILE",
-		"ETH_HTTPS_URL", "ETH_WSS_URL", "ETH_AVG_BLOCK_TIME",
+		"API_PORT",
+		"ETH_HTTPS_URL", "ETH_WSS_URL",
 		"REDIS_ADDR", "REDIS_DB", "REDIS_PUBSUB_CH", "REDIS_KEY_EXPIRY_TIME",
 		"NUM_BLOCKS_TO_SYNC", "BOOTSTRAP_TIMEOUT",
 	}
@@ -58,11 +56,6 @@ func LoadConfig() (*Config, error) {
 		return nil, err
 	}
 
-	avgBlockTime, err := strconv.Atoi(envMap["ETH_AVG_BLOCK_TIME"])
-	if err != nil {
-		return nil, err
-	}
-
 	rdb, err := strconv.Atoi(envMap["REDIS_DB"])
 	if err != nil {
 		return nil, err
@@ -86,12 +79,10 @@ func LoadConfig() (*Config, error) {
 	return &Config{
 		DEFAULT_TIMEOUT: time.Duration(defaultTimeout) * time.Second,
 
-		API_PORT:        envMap["API_PORT"],
-		API_STATIC_FILE: envMap["API_STATIC_FILE"],
+		API_PORT: envMap["API_PORT"],
 
-		ETH_HTTPS_URL:      envMap["ETH_HTTPS_URL"],
-		ETH_WSS_URL:        envMap["ETH_WSS_URL"],
-		ETH_AVG_BLOCK_TIME: time.Duration(avgBlockTime) * time.Second,
+		ETH_HTTPS_URL: envMap["ETH_HTTPS_URL"],
+		ETH_WSS_URL:   envMap["ETH_WSS_URL"],
 
 		REDIS_DB:              rdb,
 		REDIS_KEY_EXPIRY_TIME: time.Duration(expiryTime) * time.Second,

internal/services/bootstrapper/README.md

@@ -39,15 +39,7 @@ This function fetches the most recent Ethereum blocks and stores them in Redis.
   4. For each block:
      - Retrieves the block data.
      - Formats the block data.
-     - Calculates an expiry time for storing the block in Redis.
-     - Stores the block data in Redis with the calculated expiry time.
+     - Stores the block data in Redis with the pre-defined expiry time.
   5. Logs the successful loading of blocks into Redis.
 
-## Configuration
-
-### config.Config
-
-- `BOOTSTRAP_TIMEOUT`: The timeout duration for the bootstrap service.
-- `NUM_BLOCKS_TO_SYNC`: The number of recent blocks to fetch and store.
-- `ETH_AVG_BLOCK_TIME`: The average time taken to produce a new block in Ethereum (used to calculate expiry times).
 

internal/services/bootstrapper/bootstrap.go

@@ -36,6 +36,7 @@ func RunBootstrapSvc(client *client.Client, cfg *config.Config) {
 	totalTime := time.Since(startTime)
 	log.Printf("Bootstrapper successfully completed in %s", totalTime)
 
+	// Once the bootstrapper finished loading 50 blocks, it shuts down succesfully
 	log.Println("Shutting down bootstraper gracefully...")
 	os.Exit(0)
 
@@ -68,12 +69,7 @@ func loadRecentBlockData(ctx context.Context, ethClient *ethclient.Client, rdb *
 			return err
 		}
 
-		// Calculate expiry time for each block data
-		// For example: Assume blocks are [B0....B49]. B0 data will expire after the first 13 seconds, B1 after the next 13 seconds, and so on.
-		// On average, it takes 12 seconds to produce a new block in Ethereum. We set the expiry to 13 seconds to provide a safety margin.
-		expiryTime := time.Duration((n - i + 1)) * cfg.ETH_AVG_BLOCK_TIME
-
-		err = storage.AddBlockDataToDB(ctx, rdb, blockDataInBytes, expiryTime)
+		err = storage.AddBlockDataToDB(ctx, rdb, blockDataInBytes, cfg.REDIS_KEY_EXPIRY_TIME)
 		if err != nil {
 			return err
 		}

internal/storage/README.md

@@ -56,7 +56,7 @@ This function indexes each transaction by its hash in Redis.
 
 ### IdxEventsAndStore
 
-This function indexes each event by its address in Redis.
+This function indexes each event by its address in Redis. Before indexing, all addresses are converted to lower case to eliminate any case sensitivity.
 
 - **Parameters**:
   - `ctx context.Context`: The context for managing cancellation.
@@ -132,13 +132,6 @@ This function retrieves all block numbers stored in Redis.
 - `REDIS_PUBSUB_CH`: The Redis channel for publishing block data.
 - `REDIS_KEY_EXPIRY_TIME`: The expiry time for storing block data in Redis.
 
-## Dependencies
-
-- `github.com/redis/go-redis/v9`: Redis client library.
-- `encoding/json`: Package for JSON serialization and deserialization.
-- `github.com/ethereum/go-ethereum/core/types`: Ethereum types library.
-- `context`: Package for managing cancellation and timeouts.
-- `log`: Package for logging.
 
 ## Usage
 

internal/storage/index.go

@@ -5,6 +5,7 @@ import (
 	"encoding/json"
 	"ethereum-data-service/internal/model"
 	"fmt"
+	"strings"
 	"time"
 
 	"github.com/redis/go-redis/v9"
@@ -47,8 +48,10 @@ func IdxEventsAndStore(ctx context.Context, rdb *redis.Client, blockData *model.
 				return fmt.Errorf("error marshalling event %+v: %v", event, err)
 			}
 			// For ex key:`event:0x6000da47483062A0D734Ba3dc7576Ce6A0B645C4_20167294_0xd59016cbaf7c580e83544ac5bd98584f7ec65b6984ddbd6a7647d6873c16f63a_234`
-			// This is done to keep the key associated with every event unique
-			addressKey := fmt.Sprint(EVENT_PREFIX, event.Address.Hex(), "_", event.BlockNumber, "_", txHash, "_", event.Index)
+			// This is done to keep the key associated with every event unique. To eliminate any case senstivity, wrt to address we convert all address
+			// to lower case before indexing. For ex: Addr `0x0C04fF41b11065EEd8c9EDA4d461BA6611591395` and `0x0C04ff41b11065eed8c9eda4d461ba6611591395`
+			// all point to the same account. We do the same in the API call as well.
+			addressKey := fmt.Sprint(EVENT_PREFIX, strings.ToLower(event.Address.Hex()), "_", event.BlockNumber, "_", txHash, "_", event.Index)
 			if err := rdb.Set(ctx, addressKey, eventJSON, expiryTime).Err(); err != nil {
 				return fmt.Errorf("error storing event %s_%d in Redis: %v", event.Address, event.TxIndex, err)
 			}

sample.env

@@ -3,19 +3,18 @@ DEFAULT_TIMEOUT=5 #seconds
 
 # api-server
 API_PORT=8080
-API_STATIC_FILE=./api/static 
 
 # ethereum-client
 ETH_HTTPS_URL=https://mainnet.ethereum.validationcloud.io/v1/JFt58zlN7gcQlLYnMZcOD75LpethJgD6Eq5nKOxC9F0
 ETH_WSS_URL=wss://mainnet.ethereum.validationcloud.io/v1/wss/JFt58zlN7gcQlLYnMZcOD75LpethJgD6Eq5nKOxC9F0
-ETH_AVG_BLOCK_TIME=13
+
 
 
 # redis-client 
 REDIS_ADDR=localhost:6379
 REDIS_DB=0 
 REDIS_PUBSUB_CH=ETH_MAINNET 
-REDIS_KEY_EXPIRY_TIME=650  # seconds
+REDIS_KEY_EXPIRY_TIME=650  # 50 * ETH_AVG_BLOCK_TIME (13s) ~ 650s
 
 
 # bootstraper-service