k8s-multicluster-gitops

Infrastructure as Code (IaC) for provisioning and managing multiple Kubernetes clusters across multiple cloud accounts, using GitOps principles with ArgoCD.

Requirements

Guides on setting up Kubernetes clusters in the cloud are common, but few cover managing clusters across multiple providers and accounts, which is a key need for large enterprises.

This project aims to provide a practical example:

✅ Provisioning and managing Kubernetes clusters across multiple cloud providers (AWS, Azure, GCP).
✅ Deploying and maintaining Kubernetes clusters across multiple accounts and regions.

Architecture

You will likely need to deploy multiple Kubernetes clusters across various cloud providers and accounts.

Each cloud provider has a designated "management account" where subdomains are hosted:

aws.mylabs.dev - AWS
az.mylabs.dev - Azure
gcp.mylabs.dev - Google Cloud Platform

The second-level domain mylabs.dev is hosted externally (e.g., Cloudflare), and it's the user's responsibility to configure DNS delegation properly.

An IAM role (or its equivalent for each cloud provider) will be created in the management account. This role will allow GitHub Actions/mise to manage resources in the management account and will also be used to access other accounts where Kubernetes clusters are deployed.

Cloud Providers - Multi-Account Setup

Let's assume you have two AWS accounts, two Azure accounts, and two GCP accounts, and you want to deploy two Kubernetes clusters (EKS, AKS, GKE) in each account:

Cloud Provider	Account 01	Account 02
AWS (`aws.mylabs.dev`, `k8s.aws.mylabs.dev`)	`k01.k8s.aws.mylabs.dev` (US), `k02.k8s.aws.mylabs.dev` (EU)	`k03.k8s.aws.mylabs.dev` (US), `k04.k8s.aws.mylabs.dev` (EU)
Azure (`az.mylabs.dev`, `k8s.az.mylabs.dev`)	`k01.k8s.az.mylabs.dev` (US), `k02.k8s.az.mylabs.dev` (EU)	`k03.k8s.az.mylabs.dev` (US), `k04.k8s.az.mylabs.dev` (EU)
GCP (`gcp.mylabs.dev`, `k8s.gcp.mylabs.dev`)	`k01.k8s.gcp.mylabs.dev` (US), `k02.k8s.gcp.mylabs.dev` (EU)	`k03.k8s.gcp.mylabs.dev` (US), `k04.k8s.gcp.mylabs.dev` (EU)

AWS

You should have access to all your AWS accounts via the AWS CLI (using AWS_ACCESS_KEY_ID, AWS_SECRET_ACCESS_KEY, ...)

Management account

Select one of your AWS accounts to serve as the management account.

AWS CLI User

Create an aws-cli IAM user in your management AWS account. Then, add the user's ARN to the mise.toml file under the env.AWS_USER_ARN variable.

AWS_USER_NAME="aws-cli"
AWS_POLICY_ARN="arn:aws:iam::aws:policy/AdministratorAccess"

# Create IAM user
aws iam create-user --user-name "${AWS_USER_NAME}"

# Attach a policy to the user
aws iam attach-user-policy --user-name "${AWS_USER_NAME}" --policy-arn "${AWS_POLICY_ARN}"

# Create access keys for the user
aws iam create-access-key --user-name "${AWS_USER_NAME}"

# Get the ARN of the user
AWS_USER_ARN=$(aws iam list-users --query "Users[? UserName==\`${AWS_USER_NAME}\`].Arn" --output text)
sed -i "s@^AWS_USER_ARN.*@AWS_USER_ARN = \"${AWS_USER_ARN}\"@" mise.local.toml

The aws-cli user is created in the management AWS account. It will be used to access all AWS accounts via the AWS CLI by assuming the appropriate IAM role.

Route35 Hosted Zone + GitHub Action IAM Role

The following steps will create a Route 53 hosted zone for the subdomain aws.mylabs.dev and a GitHub Actions IAM role that can be assumed by the GitHub Actions workflow. The GitHub Actions IAM role will be used to manage resources in the management account and to access other accounts where Kubernetes clusters are deployed.

Create a Route 53 hosted zone for aws.mylabs.dev:

Ensure that the necessary environment variables are set for the AWS CLI (e.g., AWS_ACCESS_KEY_ID, AWS_SECRET_ACCESS_KEY).

mise run create:aws-mgmt:cf-route53-gh-action-iam-role-oidc

For more details please inspect the mise.toml file.

Next, manually set up the DNS delegation between your second-level domain mylabs.dev and the aws.mylabs.dev hosted zone in Route 53.

Example:

Tenant Accounts

Create an IAM role in each tenant account that allows the management account to assume a role in that tenant account.

Make sure to use AWS credentials (AWS_ACCESS_KEY_ID, AWS_SECRET_ACCESS_KEY) for the tenant account.

mise run create:aws-tenant:cf-iam-role

Azure

GCP

K8s Clusters

All kubeconfig files will be stored in the clusters/.kubeconfigs directory.

Kind

The kind clusters are created using the kind tool, which is a tool for running Kubernetes clusters in Docker containers.

mise run create:kind:kind01-internal
mise run delete:kind:kind01-internal
mise run create:kind:kind02-internal
mise run delete:kind:kind02-internal
mise run create-kind-all
mise run delete-kind-all

K3d

The k3d clusters are created using the k3d tool, which is a lightweight wrapper to run k3s in Docker containers.

mise run create:k3d:k3d01-internal
mise run delete:k3d:k3d01-internal
mise run create:k3d:k3d02-internal
mise run delete:k3d:k3d02-internal
mise run create-k3d-all
mise run delete-k3d-all

You can also create all the clusters at once using the create-all and delete-all commands:
mise run "create-kind-all" ::: "create-k3d-all"
mise run "delete-kind-all" ::: "delete-k3d-all"

Architecture diagrams

DNS diagram

flowchart TB
  subgraph "Cloudflare"
    mylabs.dev@{ icon: "logos:cloudflare-icon", form: "square", label: "mylabs.dev", pos: "b", h: 60 }
  end

  subgraph "AWS"
    subgraph "AWS Management Account"
      aws.mylabs.dev@{ icon: "logos:aws-route53", form: "circle", label: "aws.mylabs.dev", pos: "b", h: 60 }
      k8s.aws.mylabs.dev@{ icon: "logos:aws-route53", form: "square", label: "k8s.aws.mylabs.dev", pos: "b", h: 60 }
    end
    subgraph "AWS Account 02"
      k03.k8s.aws.mylabs.dev@{ icon: "logos:aws-route53", form: "square", label: "k03.k8s.aws.mylabs.dev", pos: "b", h: 60 }
      k04.k8s.aws.mylabs.dev@{ icon: "logos:aws-route53", form: "square", label: "k04.k8s.aws.mylabs.dev", pos: "b", h: 60 }
    end
    subgraph "AWS Account 01"
      k01.k8s.aws.mylabs.dev@{ icon: "logos:aws-route53", form: "square", label: "k01.k8s.aws.mylabs.dev", pos: "b", h: 60 }
      k02.k8s.aws.mylabs.dev@{ icon: "logos:aws-route53", form: "square", label: "k02.k8s.aws.mylabs.dev", pos: "b", h: 60 }
    end
  end

  subgraph "Azure"
    subgraph "Azure Management Account"
      az.mylabs.dev@{ icon: "logos:azure-icon", form: "circle", label: "az.mylabs.dev", pos: "b", h: 60 }
      k8s.az.mylabs.dev@{ icon: "logos:azure-icon", form: "square", label: "k8s.az.mylabs.dev", pos: "b", h: 60 }
    end
    subgraph "Azure Account 02"
      k03.k8s.az.mylabs.dev@{ icon: "logos:azure-icon", form: "square", label: "k03.k8s.az.mylabs.dev", pos: "b", h: 60 }
      k04.k8s.az.mylabs.dev@{ icon: "logos:azure-icon", form: "square", label: "k04.k8s.az.mylabs.dev", pos: "b", h: 60 }
    end
    subgraph "Azure Account 01"
      k01.k8s.az.mylabs.dev@{ icon: "logos:azure-icon", form: "square", label: "k01.k8s.az.mylabs.dev", pos: "b", h: 60 }
      k02.k8s.az.mylabs.dev@{ icon: "logos:azure-icon", form: "square", label: "k02.k8s.az.mylabs.dev", pos: "b", h: 60 }
    end
  end

  subgraph "GCP"
    subgraph "GCP Management Account"
      gcp.mylabs.dev@{ icon: "logos:google-cloud", form: "circle", label: "gcp.mylabs.dev", pos: "b", h: 60 }
      k8s.gcp.mylabs.dev@{ icon: "logos:google-cloud", form: "square", label: "k8s.gcp.mylabs.dev", pos: "b", h: 60 }
    end
    subgraph "GCP Account 02"
      k03.k8s.gcp.mylabs.dev@{ icon: "logos:google-cloud", form: "square", label: "k03.k8s.gcp.mylabs.dev", pos: "b", h: 60 }
      k04.k8s.gcp.mylabs.dev@{ icon: "logos:google-cloud", form: "square", label: "k04.k8s.gcp.mylabs.dev", pos: "b", h: 60 }
    end
    subgraph "GCP Account 01"
      k01.k8s.gcp.mylabs.dev@{ icon: "logos:google-cloud", form: "square", label: "k01.k8s.gcp.mylabs.dev", pos: "b", h: 60 }
      k02.k8s.gcp.mylabs.dev@{ icon: "logos:google-cloud", form: "square", label: "k02.k8s.gcp.mylabs.dev", pos: "b", h: 60 }
    end
  end

  mylabs.dev --> aws.mylabs.dev
  aws.mylabs.dev --> k8s.aws.mylabs.dev
  k8s.aws.mylabs.dev --> k01.k8s.aws.mylabs.dev
  k8s.aws.mylabs.dev --> k02.k8s.aws.mylabs.dev
  k8s.aws.mylabs.dev --> k03.k8s.aws.mylabs.dev
  k8s.aws.mylabs.dev --> k04.k8s.aws.mylabs.dev
  mylabs.dev --> az.mylabs.dev
  az.mylabs.dev --> k8s.az.mylabs.dev
  k8s.az.mylabs.dev --> k01.k8s.az.mylabs.dev
  k8s.az.mylabs.dev --> k02.k8s.az.mylabs.dev
  k8s.az.mylabs.dev --> k03.k8s.az.mylabs.dev
  k8s.az.mylabs.dev --> k04.k8s.az.mylabs.dev
  mylabs.dev --> gcp.mylabs.dev
  gcp.mylabs.dev --> k8s.gcp.mylabs.dev
  k8s.gcp.mylabs.dev --> k01.k8s.gcp.mylabs.dev
  k8s.gcp.mylabs.dev --> k02.k8s.gcp.mylabs.dev
  k8s.gcp.mylabs.dev --> k03.k8s.gcp.mylabs.dev
  k8s.gcp.mylabs.dev --> k04.k8s.gcp.mylabs.dev

AWS diagram

Suppose you have three AWS accounts and want to provision one EKS cluster in each. The setup would look like this:

AWS Management Account - centralized account responsible for provisioning and managing infrastructure, hosting two EKS clusters, and assuming IAM roles in tenant accounts
- k01.k8s.aws.mylabs.dev
- k02.k8s.aws.mylabs.dev
AWS Tenant Account 01 – tenant account hosting two EKS clusters. These clusters are created using a local IAM role, which is assumed by the IAM role from the management account
- k03.k8s.aws.mylabs.dev
- k04.k8s.aws.mylabs.dev
AWS Tenant Account 02 – tenant account hosting two EKS clusters. These clusters are created using a local IAM role, which is assumed by the IAM role from the management account
- k05.k8s.aws.mylabs.dev
- k06.k8s.aws.mylabs.dev

flowchart TB

  subgraph "AWS"
    subgraph "AWS Management Account"
      aws-cli@{ icon: "logos:panda", form: "rounded", label: "aws-cli", pos: "b", h: 60 }
      aws-mgmt-iam-role@{ icon: "logos:aws-iam", form: "square", label: "aws-mgmt-iam-role", pos: "b", h: 60 }
      aws-01-iam-role@{ icon: "logos:aws-iam", form: "square", label: "aws-01-iam-role", pos: "b", h: 60 }
      k01.k8s.aws.mylabs.dev@{ icon: "logos:aws-eks", form: "square", label: "k01.k8s.aws.mylabs.dev", pos: "b", h: 60 }
      k02.k8s.aws.mylabs.dev@{ icon: "logos:aws-eks", form: "square", label: "k02.k8s.aws.mylabs.dev", pos: "b", h: 60 }
    end
    subgraph "AWS Account 01"
      aws-02-iam-role@{ icon: "logos:aws-iam", form: "square", label: "aws-02-iam-role", pos: "b", h: 60 }
      k03.k8s.aws.mylabs.dev@{ icon: "logos:aws-eks", form: "square", label: "k03.k8s.aws.mylabs.dev", pos: "b", h: 60 }
      k04.k8s.aws.mylabs.dev@{ icon: "logos:aws-eks", form: "square", label: "k04.k8s.aws.mylabs.dev", pos: "b", h: 60 }
    end
    subgraph "AWS Account 02"
      aws-03-iam-role@{ icon: "logos:aws-iam", form: "square", label: "aws-03-iam-role", pos: "b", h: 60 }
      k05.k8s.aws.mylabs.dev@{ icon: "logos:aws-eks", form: "square", label: "k05.k8s.aws.mylabs.dev", pos: "b", h: 60 }
      k06.k8s.aws.mylabs.dev@{ icon: "logos:aws-eks", form: "square", label: "k06.k8s.aws.mylabs.dev", pos: "b", h: 60 }
    end
  end

  aws-cli -- "Assume Role (STS)" --> aws-mgmt-iam-role
  aws-mgmt-iam-role -- "Assume Role (STS)" --> aws-01-iam-role
  aws-01-iam-role --> k01.k8s.aws.mylabs.dev
  aws-01-iam-role --> k02.k8s.aws.mylabs.dev
  aws-mgmt-iam-role -- "Assume Role (STS)" --> aws-02-iam-role
  aws-02-iam-role --> k03.k8s.aws.mylabs.dev
  aws-02-iam-role --> k04.k8s.aws.mylabs.dev
  aws-mgmt-iam-role -- "Assume Role (STS)" --> aws-03-iam-role
  aws-03-iam-role --> k05.k8s.aws.mylabs.dev
  aws-03-iam-role --> k06.k8s.aws.mylabs.dev

Tests

docker run --rm -it --env GITHUB_TOKEN \
  -v "$PWD:/mnt" -v "/var/run/docker.sock:/var/run/docker.sock" \
  --workdir /mnt \
  bash bash -c 'set -euo pipefail && \
    apk add docker && \
    wget -q https://mise.run -O - | sh && \
    eval "$(~/.local/bin/mise activate bash)" && \
    mise run "create-kind-all" ::: "create-k3d-all" && \
    mise run "delete-kind-all" ::: "delete-k3d-all" \
  '

Name		Name	Last commit message	Last commit date
Latest commit History 51 Commits
.github		.github
cloudformation		cloudformation
images		images
mise		mise
scripts		scripts
.checkov.yml		.checkov.yml
.gitignore		.gitignore
.markdownlint.yml		.markdownlint.yml
.mega-linter.yml		.mega-linter.yml
LICENSE		LICENSE
README.md		README.md
SECURITY.md		SECURITY.md
lychee.toml		lychee.toml
mise.toml		mise.toml

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Repository files navigation

k8s-multicluster-gitops

Requirements

Architecture

Cloud Providers - Multi-Account Setup

AWS

Management account

AWS CLI User

Route35 Hosted Zone + GitHub Action IAM Role

Tenant Accounts

Azure

GCP

K8s Clusters

Kind

K3d

Architecture diagrams

DNS diagram

AWS diagram

Tests

About

Uh oh!

Releases

Packages

Uh oh!

Contributors 4

Uh oh!

Languages

License

ruzickap/k8s-multicluster-gitops

Folders and files

Latest commit

History

Repository files navigation

k8s-multicluster-gitops

Requirements

Architecture

Cloud Providers - Multi-Account Setup

AWS

Management account

AWS CLI User

Route35 Hosted Zone + GitHub Action IAM Role

Tenant Accounts

Azure

GCP

K8s Clusters

Kind

K3d

Architecture diagrams

DNS diagram

AWS diagram

Tests

About

Topics

Resources

License

Security policy

Uh oh!

Stars

Watchers

Forks

Releases

Packages 0

Uh oh!

Contributors 4

Uh oh!

Languages

Packages