Photo by
Henrique Félix
on
Unsplash
The
cf-for-k8s
project is an interesting twist to running Cloud Foundry on Kubernetes using more “Kubernetes”-native projects like Istio, Eirini, and Fluentd.
If you are interested in giving
cf-for-k8s
a spin on your own laptop the instructions below will create a Kubernetes (Minikube) and install
cf-for-k8s
.
These instructions borrow
heavily
from
https://github.com/cloudfoundry/cf-for-k8s
but I’ve attempted to streamline the process by making a number of assumptions for you including the use of MacOS and homebrew. Check out the link for additional instructions and options that were removed to streamline the install.
<minikube ip>.netip.cc
will be used as the domain for the installation so no additional DNS solutions are required.
Install CLI Tools & Minikube
brew tap k14s/tapbrew install ytt kbld kapp imgpkg kwt vendir yq
brew install kubectl
brew install cloudfoundry/tap/[email protected]
brew install cloudfoundry/tap/bosh-cli
brew install minikube
Step 2 – Start Kubernetes
Start Minikube
minikube start --cpus=6 --memory=8g --kubernetes-version="1.19.2" --driver=docker
Enable metrics-server:
minikube addons enable metrics-server
Obtain minikube IP that will be the basis for your domain URL.
minikube ip
MINIKUBE_IP=$(minikube ip)
Use minikube tunnel to expose the LoadBalancer service for the ingress gateway:
sudo minikube tunnel # be sure to run in another session
This should be run in a separate terminal as this will block.
sudo give capabilities to the tunnel to open ports 80 and 443 for the gorouters.
The kapp deploy command will not exit successfully until this command is run to allow minikube to create the LoadBalancer service.
Step 4 – Setup Config Files & Deploy Cloud Foundry
Clone and initialize this git repository:
TMP_DIR="${PWD}/cf-for-k8s-tmp"
mkdir -p ${TMP_DIR}
git clone https://github.com/cloudfoundry/cf-for-k8s.git -b main
cd cf-for-k8s
Create a “CF Installation Values” file and configure it values.yml file using the BOSH CLI to generate self signed certs and passwords.
./hack/generate-values.sh -d $MINIKUBE_IP.netip.cc > ${TMP_DIR}/cf-values.yml
Create an OCI-compliant app registry and configure the values file, hub.docker.com is pretty easy to get started:
Create an account in hub.docker.com. Note down the user name and password you used during signup.
Create a repository in your account. Note down the repository name.
Add the following registry config block to the end of ../cf-for-k8s-tmp/cf-values.yml file, swapping in the <my_username> and <my_password> values from the previous two steps:
app_registry:
hostname: https://index.docker.io/v1/
repository_prefix: "<my_username>"
username: "<my_username>"
password: "<my_password>"
remove_resource_requirements: true
enable_automount_service_account_token: true
use_first_party_jwt_tokens: true
Run the following commands to install Cloud Foundry on your Kubernetes (Minikube) cluster:
i. Render the final K8s template to raw K8s configuration:
ytt -f config -f ${TMP_DIR}/cf-values.yml > ${TMP_DIR}/cf-for-k8s-rendered.yml
ii. Install using kapp and pass the above K8s configuration file
kapp deploy -a cf -f ${TMP_DIR}/cf-for-k8s-rendered.yml -y
Step 5 – Validate the Deployment
Target your CF CLI to point to the new CF instance and login using the admin credentials for key cf_admin_password in ${TMP_DIR}/cf-values.yml:
cf api --skip-ssl-validation https://api.$MINIKUBE_IP.netip.cc
cf auth admin "$(yq read ${TMP_DIR}/cf-values.yml 'cf_admin_password')"
Create an org/space for your app:
cf create-org test-org
cf create-space -o test-org test-space
cf target -o test-org -s test-space
Deploy a source code based app:
cf push test-node-app -p tests/smoke/assets/test-node-app
You should see the following output from the above command:
Pushing app test-node-app to org test-org / space test-space as admin...
Getting app info...
Creating app with these attributes...
... omitted for brevity ...
type: web
instances: 1/1
memory usage: 1024M
routes: test-node-app.<cf-domain>
state since cpu memory disk details
0 running 2020-03-18T02:24:51Z 0.0% 0 of 1G 0 of 1G
Validate the app is reachable over https:
curl -k https://test-node-app.$MINIKUBE_IP.netip.cc
You should see the following output:
Hello World
Now do a little dance that you were able to get CF running on Kubernetes!
Work Arounds
Work Around 1
Older cf-cli was installed. To overwrite:
brew link --overwrite [email protected]
Work Around 2
Older Minikube was installed. If you don’t need it anymore and getting weird errors, delete the cluster made by the older Minikube and create a new one:
minikube delete
minikube start
Work Around 3
Error:
Exiting due to MK_USAGE: Docker Desktop has only 1998MB memory but you specified 8192MB
Go into Docker Desktop > Preferences > Advanced and move the slider on Memory over to at least 8GB, 16GB if you have it.
Work Around 4
Error:
ytt: Error: Unknown comment syntax at line cf-values.yml:443: ' Below needed for Minikube':
Don’t attempt to put comments starting with # into cf-values.yml, while valid yaml the parser errors out.
Work Around 5
The instructions for creation of a Docker Hub account ask to create a repository, however there is nowhere to put the name of the repo. Attempting to use it in the repository_prefix results in a unsupported status code 401 error for builder/cf-default-builder (kpack.io/v1alpha1) namespace: cf-workloads-staging
Work Around 6
CF CLI cf auth command fails because it cannot pull the password. This may be due to differences in yq, the instructions say the command is:
cf auth admin "$(yq -r '.cf_admin_password' ${TMP_DIR}/cf-values.yml)"
But my copy of yq installed doesn’t know -r (does have read) and even with the read command the next two parameters are reversed.
Work Around 7
After minikube stop and subsequent minikube start, the UAA pod may not come up clean:
NAMESPACE NAME READY STATUS RESTARTS AGE
cf-system uaa-6bf497b5b9-snlg6 0/3 Init:CrashLoopBackOff 4 101m
Looking at the logs you’ll see it failed to mount the secrets:
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Warning FailedMount 6m15s kubelet MountVolume.SetUp failed for volume "cc-admin-client-credentials-file" : failed to sync secret cache: timed out waiting for the condition
Warning FailedMount 6m15s kubelet MountVolume.SetUp failed for volume "ca-certs-files" : failed to sync secret cache: timed out waiting for the condition
Warning FailedMount 6m15s kubelet MountVolume.SetUp failed for volume "istiod-ca-cert" : failed to sync configmap cache: timed out waiting for the condition
Warning FailedMount 6m15s kubelet MountVolume.SetUp failed for volume "jwt-policy-signing-keys-file" : failed to sync secret cache: timed out waiting for the condition
Warning FailedMount 6m15s kubelet MountVolume.SetUp failed for volume "uaa-token-gc8nw" : failed to sync secret cache: timed out waiting for the condition
Warning FailedMount 6m15s kubelet MountVolume.SetUp failed for volume "admin-client-credentials-file" : failed to sync secret cache: timed out waiting for the condition
Warning FailedMount 6m14s (x2 over 6m15s) kubelet MountVolume.SetUp failed for volume "encryption-keys-file" : failed to sync secret cache: timed out waiting for the condition
Warning FailedMount 6m14s (x2 over 6m15s) kubelet MountVolume.SetUp failed for volume "cf-api-controllers-client-credentials-file" : failed to sync secret cache: timed out waiting for the condition
Warning FailedMount 6m14s (x2 over 6m15s) kubelet MountVolume.SetUp failed for volume "cf-admin-user-credentials-file" : failed to sync secret cache: timed out waiting for the condition
Warning FailedMount 6m14s kubelet MountVolume.SetUp failed for volume "uaa-config" : failed to sync configmap cache: timed out waiting for the condition
Warning FailedMount 6m14s kubelet MountVolume.SetUp failed for volume "saml-keys-file" : failed to sync secret cache: timed out waiting for the condition
Warning FailedMount 6m14s kubelet MountVolume.SetUp failed for volume "smtp-credentials-file" : failed to sync secret cache: timed out waiting for the condition
Warning FailedMount 6m1s (x10 over 6m6s) kubelet (combined from similar events): MountVolume.SetUp failed for volume "cc-admin-client-credentials-file" : failed to sync secret cache: timed out...
The quick fix is to kubectl describe pod uaa-6bf497b5b9-8klh9 -n cf-system and let the scheduler build out a new pod and remount the secrets.
What’s Next?
You can also try out the other solution to running Cloud Foundry on Kubernetes called KubeCF. In this blog post you can read up on how to deploy KubeCF on Amazon EKS using just a bit of Terraform.
If you have any questions on KubeCF or cf-for-k8s please ask in the comments sections below.
Thanks!
PS: Thank you to Tyler Bird for pointing out some missing bits in the homebrew commands!
