Open edX on AWS EKS

What is the need?

Learning and Development (L&D) has grown into a major business division of many multinational companies. Upskilling your employees not only increase your productivity and help improve customer service and satisfaction but also enhance the employee confidence in themselves as well their trust in the company.

In an attempt to establish a functional L&D division, what every company needs is a Learning Management Platform. A learner-centric, massively scalable solution for this is provided by edX as an open-source free-to-use platform: Open edX. You can find the Open edX LMS (student-facing, delivering courseware), and Studio (course authoring) components in their core repository.

Even though it is a great solution, there is a word in the community that it can really tricky to install. To counter this problem, Overhang Tutor came into the picture. Tutor is a docker based Open edX distribution with an intent to make it easy to deploy, customize, upgrade and scale Open edX. And so, in this story, we will be using this tutor to deploy our own Open edX platform onto AWS EKS.

Let’s get started

Step 1 : Pre-requisites

The first step to using tutor is to install the library. It can be done through binaries or from the source. Depending upon your system configurations, proceed on installing Tutor by following these installation instructions. This is the only installation needed from Tutor side

Once done, move on to installing the AWS and Kubernetes requirements. For this, we need the following

• aws cli
• eksctl
• kubectl

Use the hyperlinks to navigate to the installation instructions

Step 2 : Create EKS Cluster

eksctl create cluster --name openedx --version 1.14 --region eu-west-1 

Use the above-mentioned code to create an EKS control plane in Ireland (eu-west-1) region with the name openedx . This will create a CloudFormation stack so can also check the resources created by it over there.

Next, we add nodes to our cluster using the following command

eksctl create nodegroup --cluster openedx --region eu-west-1 --name edxnodes --node-type t2.medium --nodes 3 --nodes-min 3 --nodes-max 4 --node-volume-size 25 --managed --alb-ingress-access

This will add 3 instances of t2.medium type which can scale up to 4 instances. 25 here represents the memory attached per instance. The managed tag creates the instances in an autoscaling group and manages the instances for us and the alb-ingress-access will be used in the next step.

Once the cluster is created successfully, update then kube config to use kubectl for managing the cluster using the following command

aws eks — region eu-west-1 update-kubeconfig — name openedx

Step 3 : Configuring Ingress Controller

Now we need to install an ingress controller on our cluster to make sure that our application is accessible on a DNS. But before installing, we need to verify some tag entries in our subnets

• Verify that the following tag is available in all the subnets:kubernetes.io/cluster/<cluster-name> = shared
• Verify that the following tag is available in the public subnets:kubernetes.io/role/elb = 1
• Verify that the following tag is available in the private subnets:kubernetes.io/role/internal-elb = 1

Add these tags if not already present. Once done with that, create an OIDC provider. To use IAM roles for service accounts in your cluster, you must create an OIDC identity provider in the IAM console. To do that, use the following command

eksctl utils associate-iam-oidc-provider \
  --region eu-west-1 \
  --cluster openedx \
  --approve

Now create an IAM policy which will be attached to the service account

aws iam create-policy \
   --policy-name ALBIngressControllerIAMPolicy \
   --policy-document https://raw.githubusercontent.com/kubernetes-sigs/aws-alb-ingress-controller/v1.1.4/docs/examples/iam-policy.json

Take note of the ARN for this policy.

Next, we create a cluster role, a service account and bind them using a cluster role binding. All these components can be found in a single template and can be deployed as

kubectl apply -f https://raw.githubusercontent.com/kubernetes-sigs/aws-alb-ingress-controller/v1.1.4/docs/examples/rbac-role.yaml

Now we create an IAM Service Account and attach the policy we created earlier using the following command

eksctl create iamserviceaccount \
   --region eu-west-1 \
   --name alb-ingress-controller \
   --namespace kube-system \
   --cluster openedx \
   --attach-policy-arn <ARN-for-ALBIngressControllerIAMPolicy> \
   --override-existing-serviceaccounts \
   --approve

Finally, we deploy the Ingress Controller pods using the commands

kubectl apply -f https://raw.githubusercontent.com/kubernetes-sigs/aws-alb-ingress-controller/v1.1.4/docs/examples/alb-ingress-controller.yaml

We need to edit this deployment using kubectl edit deployment.apps/alb-ingress-controller -n kube-system command to associate the deployment with the right cluster, vpc and region. In the deployment, add the three lines mentioned in bold (make sure you replace the correct VPC id)

spec:
   containers:
   - args:
      - --ingress-class=alb
      - --cluster-name=openedx
      - --aws-vpc-id=vpc-XXXXXXXXXXXXXX
      - --aws-region=eu-west-1
     image: docker.io/amazon/aws-alb-ingress-controller:v1.1.4
     imagePullPolicy: IfNotPresent
     name: alb-ingress-controller

A few lines before and after the edited section has been added to help you navigate to the right section where you need to make the entry

Confirm that the ingress controller pod is running through

kubectl get pods -n kube-system

Step 4 : Configuring DNS

To access your platform, you will need a DNS. For my installation, I bought a domain at Gandi and changed the Nameserver to Route 53.

We will create two entries in the recordset later in the installation.

Step 5 : Deploying Open edX

Tutor provides a single step, tutor k8s quickstart, to install and initialize the services, but we will break it down into three steps to as we will have to do a minor manual step in between.

Start by saving the configurations

tutor config save --interactive

You will be asked for entries like

• LMS hostname
• CMS hostname
• Platform name
• Contact email
• Language

It will also ask whether you want to configure HTTPS certs, for which I declined. If you want to configure HTTPS, install cert-manager prior to this step using these steps and set the web proxy flag to true using the command tutor config save — set WEB_PROXY=true.

Now, to deploy all configured Open edX services, use the below-mentioned command

tutor k8s start

This will deploy all the resources but will not initialize the platform. Here come our manual steps. First step is to annotate the ingress resource created by tutor such that it can be identified by the ingress controller and creates an internet-facing application load balancer for us. You may edit the ingress using kubectl edit command or just execute the following

kubectl annotate ingress.extensions/web -n openedx kubernetes.io/ingress.class=alb alb.ingress.kubernetes.io/scheme=internet-facing

Now wait till an external address gets assigned to ingress.extensions/web, this will be a DNS name for the ALB. You may also head on to the LoadBalancer section in EC2 service of AWS console to view this. Now make two CNAME entries in the Record Set for your domain

1. Redirect traffic on LMS hostname to the DNS of ALB
2. Redirect all traffic on *.LMS hostname to LMS hostname e.g. *.example.com to example.com if example.com is the LMS hostname

Now we can initialize the platform using the following command

tutor k8s init

Once the deployment gets completed, head on to the LMS hostname to view the platform!

Automation

I had to first test this platform locally before actually using it for training. Also, I needed to make sure that I could replicate it in multiple accounts if needed, so I decided to create an Ansible playbook around it.

You can use this playbook as well, it is available in my public repo. For installing individual components, use the tags configured in the playbook. Details about the tags are provided in the README.