Question : There are a number of Kubernetes objects created inside the omega, citadel and eden-prime namespaces. However, several suspicious/abnormal operations have been observed in these namespaces!.
For example, in the citadel namespace, the application called webapp-color is constantly changing! You can see this for yourself by clicking on the citadel-webapp link and refreshing the page every 30 seconds. Similarly there are other issues with several other objects in other namespaces.
To understand what's causing these anomalies, you would be required to configure auditing in Kubernetes and make use of the Falco tool.
Inspect the issues in detail by clicking on the icons of the interactive architecture diagram in the lab and complete the tasks to secure the cluster. Once done click on the Check button to validate your work.
Solution:
1. Lets first check the K8 cluster to proceed ahead with solution.
|
root@controlplane
~ ➜ kubectl get nodes NAME STATUS ROLES AGE VERSION controlplane Ready
control-plane,master
4m49s v1.23.0 root@controlplane ~ ➜ root@controlplane ~ ➜ kubectl get all -A NAMESPACE NAME
READY STATUS RESTARTS AGE citadel pod/webapp-color 0/1 ContainerCreating 0 3s eden-prime pod/eden-fe-77574c68cd-mmtg2 1/1 Running 0 4m15s eden-prime pod/eden-software1 1/1 Running 0 4m15s eden-prime pod/eden-software2 1/1 Running 0 4m15s eden-prime pod/eden-software3 1/1 Running 0 4m15s kube-system pod/coredns-64897985d-s55wd 1/1 Running 0 4m32s kube-system pod/coredns-64897985d-vrh62 1/1 Running 0 4m32s kube-system pod/etcd-controlplane 1/1 Running 0 4m49s kube-system pod/kube-apiserver-controlplane 1/1 Running 0 4m49s kube-system
pod/kube-controller-manager-controlplane 1/1
Running 0 4m52s kube-system pod/kube-proxy-v5jtm 1/1 Running 0 4m33s kube-system pod/kube-scheduler-controlplane 1/1 Running 0 4m49s kube-system pod/weave-net-9wkn8 2/2 Running 1 (4m29s ago) 4m33s omega pod/omega-fe-678c4ccf75-pxvft 1/1 Running 0 4m14s omega pod/omega-software4 1/1 Running 0 4m14s omega pod/omega-software5 1/1 Running 0 4m14s omega pod/omega-software6 1/1 Running 0 4m14s
NAMESPACE NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE citadel service/webapp-color NodePort
10.108.71.29 <none> 8080:32192/TCP 4m17s default service/kubernetes ClusterIP 10.96.0.1 <none> 443/TCP 4m51s kube-system service/kube-dns ClusterIP 10.96.0.10 <none> 53/UDP,53/TCP,9153/TCP 4m49s
NAMESPACE NAME DESIRED CURRENT
READY UP-TO-DATE AVAILABLE
NODE SELECTOR AGE kube-system daemonset.apps/kube-proxy 1
1 1 1 1 kubernetes.io/os=linux 4m49s kube-system daemonset.apps/weave-net 1
1 1 1 1 <none> 4m38s
NAMESPACE NAME READY UP-TO-DATE AVAILABLE
AGE eden-prime deployment.apps/eden-fe 1/1
1 1 4m15s kube-system deployment.apps/coredns 2/2
2 2 4m50s omega deployment.apps/omega-fe 1/1
1 1 4m14s
NAMESPACE NAME
DESIRED CURRENT READY
AGE eden-prime replicaset.apps/eden-fe-77574c68cd 1
1 1 4m15s kube-system replicaset.apps/coredns-64897985d 2
2 2 4m33s omega
replicaset.apps/omega-fe-678c4ccf75
1 1 1 4m14s root@controlplane ~ ➜ |
2. Let's move to /etc/kubernetes/ and create YAML file named audit-policy.yaml then create single rule policy to record events according to question.
|
root@controlplane
~ ➜ vi /etc/kubernetes/audit-policy.yaml root@controlplane ~ ➜ cat /etc/kubernetes/audit-policy.yaml apiVersion:
audit.k8s.io/v1 # This is required. kind: Policy # Don't generate
audit events for all requests in RequestReceived stage. omitStages: - "RequestReceived" rules: - level:
Metadata resources: - resources: ["pods",
"configmaps"] namespaces: ["omega",
"citadel", "eden-prime"] root@controlplane ~ ➜ |
3. Lets modify kube-apiserver YAML file to use volume called audit and mount only file.
|
root@controlplane
~ ➜ vi
/etc/kubernetes/manifests/kube-apiserver.yaml root@controlplane
~ ➜ cat
/etc/kubernetes/manifests/kube-apiserver.yaml apiVersion: v1 kind: Pod metadata: annotations:
kubeadm.kubernetes.io/kube-apiserver.advertise-address.endpoint:
192.168.121.109:6443 creationTimestamp: null labels: component: kube-apiserver tier: control-plane name: kube-apiserver namespace: kube-system spec: containers: - command: - kube-apiserver - --advertise-address=192.168.121.109 - --allow-privileged=true - --authorization-mode=Node,RBAC -
--audit-policy-file=/etc/kubernetes/audit-policy.yaml - --audit-log-path=/var/log/kubernetes/audit/audit.log -
--client-ca-file=/etc/kubernetes/pki/ca.crt -
--enable-admission-plugins=NodeRestriction - --enable-bootstrap-token-auth=true -
--etcd-cafile=/etc/kubernetes/pki/etcd/ca.crt - --etcd-certfile=/etc/kubernetes/pki/apiserver-etcd-client.crt -
--etcd-keyfile=/etc/kubernetes/pki/apiserver-etcd-client.key - --etcd-servers=https://127.0.0.1:2379 -
--kubelet-client-certificate=/etc/kubernetes/pki/apiserver-kubelet-client.crt - --kubelet-client-key=/etc/kubernetes/pki/apiserver-kubelet-client.key -
--kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname -
--proxy-client-cert-file=/etc/kubernetes/pki/front-proxy-client.crt -
--proxy-client-key-file=/etc/kubernetes/pki/front-proxy-client.key -
--requestheader-allowed-names=front-proxy-client -
--requestheader-client-ca-file=/etc/kubernetes/pki/front-proxy-ca.crt -
--requestheader-extra-headers-prefix=X-Remote-Extra- -
--requestheader-group-headers=X-Remote-Group -
--requestheader-username-headers=X-Remote-User - --secure-port=6443 -
--service-account-issuer=https://kubernetes.default.svc.cluster.local - --service-account-key-file=/etc/kubernetes/pki/sa.pub -
--service-account-signing-key-file=/etc/kubernetes/pki/sa.key - --service-cluster-ip-range=10.96.0.0/12 -
--tls-cert-file=/etc/kubernetes/pki/apiserver.crt -
--tls-private-key-file=/etc/kubernetes/pki/apiserver.key image: k8s.gcr.io/kube-apiserver:v1.23.0 imagePullPolicy: IfNotPresent livenessProbe: failureThreshold: 8 httpGet: host: 192.168.121.109 path: /livez port: 6443 scheme: HTTPS initialDelaySeconds: 10 periodSeconds: 10 timeoutSeconds: 15 name: kube-apiserver readinessProbe: failureThreshold: 3 httpGet: host: 192.168.121.109 path: /readyz port: 6443 scheme: HTTPS periodSeconds: 1 timeoutSeconds: 15 resources: requests: cpu: 250m startupProbe: failureThreshold: 24 httpGet: host: 192.168.121.109 path: /livez port: 6443 scheme: HTTPS initialDelaySeconds: 10 periodSeconds: 10 timeoutSeconds: 15 volumeMounts: - mountPath: /etc/ssl/certs name: ca-certs readOnly: true - mountPath: /etc/ca-certificates name: etc-ca-certificates readOnly: true - mountPath: /etc/kubernetes/pki name: k8s-certs readOnly: true - mountPath:
/usr/local/share/ca-certificates name: usr-local-share-ca-certificates readOnly: true - mountPath: /usr/share/ca-certificates name: usr-share-ca-certificates readOnly: true - mountPath:
/etc/kubernetes/audit-policy.yaml name: audit readOnly: true - mountPath: /var/log/kubernetes/audit/ name: audit-log readOnly: false hostNetwork: true priorityClassName: system-node-critical securityContext: seccompProfile: type: RuntimeDefault volumes: - hostPath: path: /etc/ssl/certs type: DirectoryOrCreate name: ca-certs - hostPath: path: /etc/ca-certificates type: DirectoryOrCreate name: etc-ca-certificates - hostPath: path: /etc/kubernetes/pki type: DirectoryOrCreate name: k8s-certs - hostPath: path: /usr/local/share/ca-certificates type: DirectoryOrCreate name: usr-local-share-ca-certificates - hostPath: path: /usr/share/ca-certificates type: DirectoryOrCreate name: usr-share-ca-certificates - name: audit hostPath: path: /etc/kubernetes/audit-policy.yaml type: File - name: audit-log hostPath: path: /var/log/kubernetes/audit/ type: DirectoryOrCreate status: {} root@controlplane
~ ➜ |
4. Create a YAML file with all the parameters, Kindly do the changes as per task you can copy from GitLab https://gitlab.com/nb-tech-support/devops.git
( Refer Below Video for more clarity )
5. Click on Finish & Confirm to complete the task successfully
Happy Learning!!!!
Apart from this if you need more clarity, I have made a tutorial video on this, please go through and share your comments. Like and share the knowledge
0 Comments