Skip to content

Improve e2store README #4035

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 1 commit into from
Apr 26, 2025
Merged

Improve e2store README #4035

Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
103 changes: 87 additions & 16 deletions packages/e2store/README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -6,18 +6,27 @@
[![Code Coverage][e2store-coverage-badge]][e2store-coverage-link]
[![Discord][discord-badge]][discord-link]

| A collection of utility functions for Ethereum. |
| ----------------------------------------------- |
| A collection of utility functions for Ethereum data storage formats. |
| ------------------------------------------------------------------- |

`@ethereumjs/e2store` provides utilities for working with Ethereum data storage formats, including E2HS, Era1, and Era files. These formats are commonly used for storing historical blockchain data, beacon states, and block data in an efficient, provable, and standardized way.

## Table of Contents

- [@ethereumjs/e2store `v10`](#ethereumjse2store-v10)
- [Table of Contents](#table-of-contents)
- [Installation](#installation)
- [Usage](#usage)
- [Export History as Era1](#export-history-as-era1)
- [E2HS](#e2hs)
- [Format E2HS](#format-e2hs)
- [Read Tuples from E2HS](#read-tuples-from-e2hs)
- [Read E2HS Tuple at Index](#read-e2hs-tuple-at-index)
- [Era1](#era1)
- [Export History as Era1](#export-history-as-era1)
- [Read Era1 file](#read-era1-file)
- [Read Era file](#read-era-file)
- [Era](#era)
- [Read Era file](#read-era-file)
- [Common Use Cases](#common-use-cases)
- [EthereumJS](#ethereumjs)
- [License](#license)

Expand All @@ -37,23 +46,69 @@ All helpers are re-exported from the root level and deep imports are not necessa
import { formatEntry } from "@ethereumjs/e2store"
```

### Export History as Era1
### E2HS

E2HS is a format for storing historical blockchain data along with proofs. It provides efficient access to block headers, bodies, and receipts, and can be used to bootstrap a Portal History Network DB.

#### Format E2HS

```ts
import { formatE2HS } from "@ethereumjs/e2store"

// Format data into E2HS format
// data is an array of block tuple data and an epoch index
const e2hs = await formatE2HS(data)
```

#### Read Tuples from E2HS

```ts
import { readTuplesFromE2HS, parseEH2SBlockTuple } from "@ethereumjs/e2store"

// Read all tuples from an E2HS file
const tuples = await readTuplesFromE2HS(filePath)
for await (const tuple of tuples) {
const { headerWithProof, body, receipts } = parseEH2SBlockTuple(tuple)
console.log(headerWithProof)
console.log(body)
console.log(receipts)
}
```

#### Read E2HS Tuple at Index

```ts
import { readE2HSTupleAtIndex, parseEH2SBlockTuple } from "@ethereumjs/e2store"

// Read a specific tuple by index
const tuple = await readE2HSTupleAtIndex(filePath, index)
const { headerWithProof, body, receipts } = parseEH2SBlockTuple(tuple)
console.log(headerWithProof)
console.log(body)
console.log(receipts)
```

### Era1

Era1 files store historical data in epochs of 8192 blocks, making it efficient to access large ranges of historical data. Era1 block tuples contain a header, body, receipts, and total difficulty. The data can be verified by reconstructing the epoch accumulator, and validating again the accumulator root, also contained in the era1 file.

Export history in epochs of 8192 blocks as Era1 files
#### Export History as Era1

Export history from an EthereumJS client DB in epochs of 8192 blocks as Era1 files:

```ts
import { exportEpochAsEra1 } from "@ethereumjs/e2store"

const dataDir = PATH_TO_ETHEREUMJS_CLIENT_DB
const epoch = 0

// generates ${dataDir}/era1/epoch-0.era1
// Generates ${dataDir}/era1/epoch-0.era1
await exportEpochAsEra1(epoch, dataDir)
```

### Read Era1 file

`readERA1` returns an async iterator of block tuples (header + body + receipts + totalDifficulty)
`readERA1` returns an async iterator of block tuples (header + body + receipts + totalDifficulty):

```ts
import {
Expand All @@ -68,44 +123,60 @@ import {

const era1File = readBinaryFile(PATH_TO_ERA1_FILE)

// validate era1 file
// Validate era1 file
// Rebuilds epoch accumulator and validates the accumulator root
const isValid = validateERA1(era1File)
if (!isValid) {
throw new Error('Invalid Era1 file')
}

// read blocks from era1 file
// Read blocks from era1 file
const blocks = readERA1(era1File)

for await (const blockTuple of blocks) {
const { header, body, receipts } = await parseBlockTuple(blockTuple)
const block = blockFromTuple({ header, body })
console.log(block.header.number)
console.log(`Block number: ${block.header.number}`)
}

// reconstruct epoch accumulator
// Reconstruct epoch accumulator
const headerRecords = await getHeaderRecords(era1File)
const epochAccumulator = EpochAccumulator.encode(headerRecords)
const epochAccumulatorRoot = EpochAccumulator.merkleRoot(headerRecords)
```

### Read Era file
### Era

Era files are used to store beacon chain data, including beacon states and blocks.

#### Read Era file

```ts
import { readBeaconState } from "@ethereumjs/e2store"
import { readBeaconState, readBlocksFromEra } from "@ethereumjs/e2store"

const eraFile = readBinaryFile(PATH_TO_ERA_FILE)

// Extract BeaconState
const state = await readBeaconState(eraFile)
console.log(state.slot)
console.log(`Current slot: ${state.slot}`)

// Read Beacon Blocks from era file
let count = 0
for await (const block of readBlocksFromEra(eraFile)) {
console.log(block.message.slot)
console.log(`Block slot: ${block.message.slot}`)
count++
if (count > 10) break
}
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nice, can we also add all the examples from the README to a dedicated examples folder so that they are auto-inserted from now on when npm run build:examples is run?

So, the way that this works is that you add the comment line (and you can leave the example code itself in first round) from the screenshot below in the README example block(s):

grafik

And when you then run the npm command (and code is not yet there) you can test if the stuff is inserted automatically.

```

## Common Use Cases

1. **Historical Data Access**: Use E2HS and Era1 formats to efficiently access historical blockchain data
2. **Beacon Chain Analysis**: Read and analyze beacon chain states and blocks using Era files
3. **Data Export**: Export historical data in standardized formats for analysis or archival
4. **Portal History Network**: Bootstrap a Portal History Network DB using E2HS
5. **Execution Client Sync**: Sync an execution client without devp2p using Era1 or E2HS files

## EthereumJS

See our organizational [documentation](https://ethereumjs.readthedocs.io) for an introduction to `EthereumJS` as well as information on current standards and best practices. If you want to join for work or carry out improvements on the libraries, please review our [contribution guidelines](https://ethereumjs.readthedocs.io/en/latest/contributing.html) first.
Expand Down