Skip to content

Commit ed5841f

Browse files
committed
update READMEs
1 parent badd12c commit ed5841f

File tree

3 files changed

+10
-9
lines changed

3 files changed

+10
-9
lines changed

README.md

Lines changed: 8 additions & 7 deletions
Original file line numberDiff line numberDiff line change
@@ -1,14 +1,15 @@
11
# Metaflow Terraform module
22

3-
Provides the core functionality for Metaflow which includes:
3+
Terraform module that provisions AWS resources to run [Metaflow](https://metaflow.org/) in production.
44

5-
- on demand processing (`computation`)
6-
- blob and tabular storage (`datastore`)
7-
- an API to record and query past executions (`metadata-service`)
8-
- orchestrated processing (`step-functions`)
9-
- other bits of infra like Amazon Elastic Container Registry (ECR) to hold the Docker image we wish to use with Metaflow.
5+
This module consists of submodules that can be used separately as well:
6+
7+
- AWS Batch cluster to run Metaflow steps ([`metaflow-computation`](./modules/computation))
8+
- blob storage and metadata database ([`metaflow-datastore`](./modules/datastore))
9+
- a service providing API to record and query past executions ([`metaflow-metadata-service`](./modules/metadata-service))
10+
- resources to deploy Metaflow flows on Step Functions processing ([`metaflow-step-functions`](./modules/step-functions))
11+
- Metaflow UI([`metaflow-ui`](./modules/ui))
1012

11-
This module is composed of submodules which break up the responsibility into logical parts listed above.
1213
You can either use this high-level module, or submodules individually. See each module's corresponding `README.md` for more details.
1314

1415
This module requires an Amazon VPC to be set up by the module user beforehand. The output of the project `infra` is an example configuration of an Amazon VPC that can be passed to this module.

modules/computation/README.md

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -3,7 +3,7 @@
33
This module sets up the resources to run Metaflow steps on AWS Batch. One can modify how many resources
44
we want to have available, as well as configure autoscaling
55

6-
This module is not required to use Metaflow, as you can also run steps locally and leverage the Datastore
6+
This module is not required to use Metaflow, as you can also run steps locally, or in a Kubernetes cluster instead.
77

88
To read more, see [the Metaflow docs](https://docs.metaflow.org/metaflow-on-aws/metaflow-on-aws#compute)
99

modules/metadata-service/README.md

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1,6 +1,6 @@
11
# Metadata Service
22

3-
The Metadata Service is a central store for the Metaflow metadata. Namely, it contains information about past runs, and pointers to data artifacts they produced. Metaflow client talks to the Metadata service over an HTTP API endpoint. Metadata service is not strictly required to use Metaflow, especially in the local mode, but it enables a lot of useful functionality, especially if there is more than person using Metaflow in your team.
3+
The Metadata Service is a central store for the Metaflow metadata. Namely, it contains information about past runs, and pointers to data artifacts they produced. Metaflow client talks to the Metadata service over an HTTP API endpoint. Metadata service is not strictly required to use Metaflow (you can use Metaflow in the "local" mode without it), but it enables a lot of useful functionality, especially if there is more than person using Metaflow in your team.
44

55
This terraform module provisions infrastructure to run Metadata service on AWS Fargate.
66

0 commit comments

Comments
 (0)