IAM and network on AWS
Every identity an EKS install of Ramen uses, what each may do, and every network path, read from
deploy/terraform/aws/{main,iam}.tf, deploy/terraform/aws-satellite/, deploy/cloudformation/ramen.yaml (which
mirrors the Terraform module), the chart deploy/helm/ramen and the console's AWS adapter
(console/src/ramen_console/cloud/aws*.py). Terraform creates nothing per group. The install itself:
Deploy on AWS.
Identities at a glance
| Identity | Kind | Created by | Used by |
|---|---|---|---|
/ramen/ramen-console |
IAM role, IRSA for system:serviceaccount:ramen-system:console |
Terraform / CloudFormation | the console pod |
ramen-eks-cluster |
IAM role for the EKS service | Terraform | the control plane |
ramen-eks-node |
IAM role for EC2 | Terraform | the managed node group |
/ramen/ramen-<group>-<zone> |
IAM role, IRSA for ramen-<group>-<zone>/worker, boundary ramen-worker-boundary |
the console, when a zone is bound | that zone's worker pods |
/ramen/build/ramen-build-<group> |
IAM role, IRSA for ramen-system/ramen-build-<group>, boundary ramen-build-boundary |
the console, on the group's first build | the group's image-build Job |
/ramen/ramen-evals |
IAM role, IRSA for ramen-system/ramen-evals |
Terraform | the evals and SkillSpector scan Jobs |
/ramen/ramen-alb-controller, /ramen/ramen-fluent-bit |
IAM roles, IRSA in kube-system |
Terraform | the AWS Load Balancer Controller, Fluent Bit |
/ramen/ramen-ebs-csi |
IAM role, IRSA for kube-system/ebs-csi-controller-sa |
Terraform / CloudFormation | the aws-ebs-csi-driver addon's controller (the chart's PVCs) |
Every IRSA role trusts the cluster's OIDC provider with an exact sub (namespace and service account) and aud =
sts.amazonaws.com.
The console
Inline policy ramen-console on /ramen/ramen-console:
| Actions | Resources |
|---|---|
s3:ListBucket, s3:GetBucketLocation; s3:GetObject, PutObject, DeleteObject |
the groups bucket; its objects |
secretsmanager:CreateSecret, PutSecretValue, GetSecretValue, DescribeSecret, DeleteSecret, RestoreSecret, TagResource, UpdateSecret |
secret:ramen/* (group secrets) and secret:ramen-console/* (its own) |
secretsmanager:ListSecrets, dynamodb:ListTables, iam:ListRoles, wafv2:ListWebACLs, ListIPSets, wafv2:GetWebACLForResource, elasticloadbalancing:DescribeLoadBalancers, DescribeTags, SetWebAcl, logs:StartQuery, GetQueryResults, StopQuery, DescribeLogGroups |
* (not resource-scopable) |
| DynamoDB item and table actions | the state table and its indexes |
eks:DescribeCluster |
the cluster (the OIDC issuer for worker roles) |
iam:DeleteRole, GetRole, UpdateAssumeRolePolicy, TagRole, DeleteRolePolicy, GetRolePolicy, ListRolePolicies, ListAttachedRolePolicies, DetachRolePolicy |
role/ramen/* |
iam:CreateRole, PutRolePolicy, AttachRolePolicy |
role/ramen/*, only with iam:PermissionsBoundary = ramen-worker-boundary or ramen-build-boundary |
ecr:CreateRepository, DescribeRepositories, TagResource, DeleteRepository, BatchDeleteImage, ListImages |
repository/ramen/groups/* only — no push action anywhere |
wafv2: create/get/update/delete IP sets and the web ACL, TagResource; AssociateWebACL, DisassociateWebACL |
the web ACL ramen, IP sets ramen-*, the IngressGroup's ALBs |
No iam:PassRole: IRSA binds a role to a service account by annotation, nothing is passed to a service. With
console_bedrock = true (the default; CloudFormation parameter ConsoleBedrock) a second inline policy
ramen-console-bedrock allows bedrock:InvokeModel and InvokeModelWithResponseStream on foundation models and the
account's inference profiles, and bedrock:CallWithBearerToken: the judge proxy calls Bedrock for LLM rails.
Inside the cluster the console has the same ClusterRole ramen-console, per-zone ramen-console-zone RoleBinding and
ramen-console-build Role as on GCP (IAM and network on GCP), the
ramen-console-tls Role in ramen-system (D86: the CA ConfigMap ramen-ca, the chart services' ramen-tls-<service>
Secrets, restarts of those Deployments), plus Ingress rights (the ALB routes are Ingresses on AWS).
Worker roles
/ramen/ramen-<group>-<zone>, tagged ramen=worker, group=<group>, zone=<zone>. Baseline inline policy:
s3:ListBucket with s3:prefix <group>/ or <group>/*, s3:GetObject on <bucket>/<group>/*,
secretsmanager:GetSecretValue on secret:ramen/<group>/*. Approved permission requests add actions to the inline
policy (service-account permissions).
The permissions boundary ramen-worker-boundary caps all of it, keyed on the role's tag:
s3:ListBucketonly withs3:prefix${aws:PrincipalTag/group}/…; object actions only on<bucket>/${aws:PrincipalTag/group}/*;secretsmanager:GetSecretValue,DescribeSecretonly onsecret:ramen/${aws:PrincipalTag/group}/*;- the catalogue's account-wide actions (logs, CloudWatch metrics, SNS publish, SQS consume, DynamoDB items,
kms:Decrypt,bedrock:InvokeModel) — reachable only when a request for them was approved.
So no inline policy, however it is written, reaches another group's prefix or secrets.
Image builds, evals and scans
- Builds (D85). The console creates ECR
ramen/groups/<group>/workerand role/ramen/build/ramen-build-<group>(tagramen-build=<group>) with inline policyramen-build:ecr:GetAuthorizationToken, pull and push on that repository, pull on the worker base repository. The boundaryramen-build-boundaryallows the registry login, pull onrepository/ramen/*and push only onrepository/ramen/groups/${aws:PrincipalTag/ramen-build}/*. The platform repositoriesramen/console,ramen/workerandramen/skillspectorare out of every build role's reach, and out of the console's (it holds no push action). - Evals and scans.
/ramen/ramen-evalsmaylogs:CreateLogStreamandlogs:PutLogEventson the Container Insights application log group, nothing else. The KSA mounts no API token and is bound to no Role.
Cluster, nodes and add-ons
ramen-eks-cluster holds AmazonEKSClusterPolicy. ramen-eks-node holds AmazonEKSWorkerNodePolicy,
AmazonEKS_CNI_Policy and AmazonEC2ContainerRegistryReadOnly (AWS-managed; pulls every repository of the account).
The Load Balancer Controller role carries the controller's official policy (vendored, v2.13.0); Fluent Bit's carries
CloudWatchAgentServerPolicy. ramen-ebs-csi carries AmazonEBSCSIDriverPolicy (AWS-managed: create, attach,
snapshot and delete EBS volumes, the create calls limited to volumes the driver tags), for the managed
aws-ebs-csi-driver addon that provisions the chart's PVCs (promptfoo, Phoenix) on the chart's encrypted gp3 class;
satellites run no PVCs and get no driver. The cluster's access mode is API_AND_CONFIG_MAP with the creator as admin.
Network
| Path | From → to | Port | Control |
|---|---|---|---|
| Clients → edge | internet → internet-facing ALB (IngressGroup ramen, target-type: ip) |
443 only | ACM certificate (self-signed import, SAN *.<region>.elb.amazonaws.com); WAF web ACL ramen |
| Edge → console | ALB → console pods | 8000 | Ingress console on /; HTTPS to the pod and HTTPS health check on /healthz (D86; the ALB does not verify the certificate) |
| Edge → workers | ALB → worker pods | 8080 | per-zone Ingresses worker (gRPC paths, GRPC target group) and worker-http (/mcp, OAuth metadata, HTTP1 target group), both matching ramen-group + ramen-zone, both backend-protocol: HTTPS with HTTPS health checks (D86); IP rules = a WAF IP set per group and a web ACL rule blocking that group's header from other sources |
| Console → workers | console pod → worker pod IPs (pod subnets, D89) | 8080 | gRPC over TLS verified against the install CA as the zone's <group>.<zone>.worker.ramen.internal (D86, D92); admin key; NetworkPolicy worker (see below) |
| Workers → console, Redis, collector, internet | as on GCP | 8000, 6379, 4318, any | TLS on each (D86); worker-egress (needs cluster.serviceCidr: aws eks describe-cluster --name ramen --query cluster.kubernetesNetworkConfig.serviceIpv4Cidr --output text; EKS picks 172.20.0.0/16 when the VPC is in 10.0.0.0/8, as the default 10.42.0.0/16 is, else 10.100.0.0/16). The install command in Deploy on AWS does not set it, so there is no worker-egress policy until you add --set cluster.serviceCidr=… |
| Home → satellite | home VPC (as the home node's address) → satellite worker pods | 8080 | VPC peering with routes both ways; ingress rule on the satellite's cluster security group (which the pod interfaces carry) from home_vpc_cidr; the satellite zone's worker policy admits the home public subnets |
- VPC.
vpc_cidr(default10.42.0.0/16), an internet gateway, no NAT, one route table for all four subnets, and the EKS API endpoint public and private. Subnets (D89; the satellite's are the same in its10.43.0.0/16):
| Subnet | Default CIDR | Holds | Public IP on launch | ALB may use it |
|---|---|---|---|---|
ramen-public-0, -1 |
10.42.0.0/24, 10.42.1.0/24 |
nodes (their primary addresses), the ALB's interfaces, the EKS control plane's interfaces | yes | yes (kubernetes.io/role/elb) |
ramen-pods-0, -1 |
10.42.64.0/20, 10.42.80.0/20 |
every pod that is not hostNetwork |
no | no (untagged) |
Pods take addresses from the pod subnet of their node's availability zone through VPC CNI custom networking: the
managed vpc-cni addon sets AWS_VPC_K8S_CNI_CUSTOM_NETWORK_CFG and ENI_CONFIG_LABEL_DEF =
topology.kubernetes.io/zone and creates one ENIConfig per zone (pod subnet + the cluster security group).
Pod traffic leaving the VPC is rewritten to the node's address (AWS_VPC_K8S_CNI_EXTERNALSNAT=false), so there is
no NAT gateway. A node's first interface holds no pods: a t3.small takes 8 pods (was 11).
- Security groups are EKS's cluster security group and the ALB's, which the Load Balancer Controller manages
(443 from anywhere, then to the pods). Terraform adds none on the home cluster.
- NetworkPolicies. The chart and the console write the same policies as on GCP. Two things differ on EKS:
1. A policy is only enforced when the cluster runs a policy engine. The Amazon VPC CNI enforces NetworkPolicies
only with its network-policy agent on (enableNetworkPolicy); since 1.0.0c Terraform (home and satellite) and
the CloudFormation template install the managed vpc-cni addon with it on. Checked live on 1.0.0b35: an
unrelated pod is dropped at Redis (Threat model, S6b).
2. The ALB reaches pods from its interfaces in the public subnets, so the "edge" peer is a CIDR list: chart
networkPolicy.awsAlbCidrs for the ramen-system policies, which the chart also hands the console as
RAMEN_AWS_ALB_CIDRS for each zone's worker policy. Terraform sets it: terraform output -raw network_set
(CloudFormation: output NetworkSet) is --set networkPolicy.awsAlbCidrs={10.42.0.0/24,10.42.1.0/24}, part of
the install command in Deploy on AWS. With satellites, the console's list also
needs each satellite's public subnets (console.env.RAMEN_AWS_ALB_CIDRS, comma separated; output alb_cidrs).
The pre-D89 networkPolicy.awsVpcCidr / RAMEN_AWS_VPC_CIDR (a whole VPC) still work when the new value is
unset, but a VPC range holds the pod subnets too and admits every pod.
Who can reach a worker pod on 8080, then: the ALB (public subnets), the console (namespace ramen-system), the
zone's own pods, and anything using a node's address — kubelet probes, hostNetwork pods, and on a satellite the
home cluster's pods, whose cross-VPC traffic carries the home node's address (the console has no fixed address of
its own to narrow this to; the worker's own gates still apply). The console's ALB list is one for every zone, so a
home zone likewise admits each satellite's public subnets. A pod in any other namespace has a
pod-subnet address and is dropped (Threat model, S6b). The
network-policy agent lets the kubelet's probes through as well (Redis, whose policy has no ipBlock, passes its
tcpSocket probe on EKS).
- IP rules are enforced by the node (RAMEN_ALLOWED_CIDRS) and by WAF. WAF IP sets take /1–/32, so
0.0.0.0/0 is written as two /1s.
Satellites
deploy/terraform/aws-satellite adds an EKS cluster in another region of the same account, a VPC (default
10.43.0.0/16) peered with the home one, routes both ways and the 8080 rule above, an EKS access entry mapping the
console role to the Kubernetes group ramen-console (bound to the ClusterRole ramen-console by the satellite chart,
and in each zone namespace through the home chart's console.k8sSubjects), and
a separate inline policy on the console role for the satellite's EKS, WAF and ALB actions. ECR is regional: satellite
nodes pull from the home region's repositories. Deploy on AWS → Satellite clusters.
Switches that change IAM
| Terraform variable / CloudFormation parameter | Default | Effect |
|---|---|---|
console_bedrock / ConsoleBedrock |
true / "true" |
the console's Bedrock policy for the judge proxy |
otel_collector, evals_engine, skillspector_engine |
none |
the optional engines (no IAM of their own); CloudFormation has no parameter for them — add the chart's --set otel.collector=…,evals.engine=…,skillspector.engine=… yourself |