시스템 설치 가이드
Kubernetes 설치

Kubernetes 설치

MicroK8s 설치

Kubernetes 클러스터는 MicroK8s로 간편하게 설정할 수 있습니다. MicroK8s 설치에 대한 자세한 정보는 microk8s.io (opens in a new tab)에서 확인할 수 있습니다.

$ sudo snap install microk8s --classic
$ sudo microk8s status --wait-ready

Addon

MicroK8s에서 제공하는 다양한 Addon을 활성화해 클러스터 기능을 확장할 수 있습니다. Addon 목록과 설정 방법은 MicroK8s Addons 공식 문서 (opens in a new tab)에서 확인할 수 있습니다.

활성화된 Addons 확인 MicroK8s 상태와 활성화된 Addons 목록을 확인하려면 다음 명령어를 사용합니다.

$ microk8s status
  • 예시 출력
microk8s is running
high-availability: no
  datastore master nodes: 127.0.0.1:19001
  datastore standby nodes: none
addons:
  enabled:
        argocd               # (community) Argo CD is a declarative continuous deployment for Kubernetes.
        cert-manager         # (core) Cloud native certificate management
        community            # (core) The community addons repository
        dashboard            # (core) The Kubernetes dashboard
        dns                  # (core) CoreDNS
        ha-cluster           # (core) Configure high availability on the current node
        helm                 # (core) Helm - the package manager for Kubernetes
        helm3                # (core) Helm 3 - the package manager for Kubernetes
        hostpath-storage     # (core) Storage class; allocates storage from host directory
        ingress              # (core) Ingress controller for external access
        metallb              # (core) Loadbalancer for your Kubernetes cluster
        metrics-server       # (core) K8s Metrics Server for API access to service metrics
        observability        # (core) A lightweight observability stack for logs, traces and metrics
        storage              # (core) Alias to hostpath-storage add-on, deprecated
  disabled:
        cilium               # (community) SDN, fast with full network policy
        cloudnative-pg       # (community) PostgreSQL operator CloudNativePG
        dashboard-ingress    # (community) Ingress definition for Kubernetes dashboard
        easyhaproxy          # (community) EasyHAProxy can configure HAProxy automatically based on ingress labels
        falco                # (community) Cloud-native runtime threat detection tool for Linux and K8s
        fluentd              # (community) Elasticsearch-Fluentd-Kibana logging and monitoring
        gopaddle             # (community) DevSecOps and Multi-Cloud Kubernetes Platform
        inaccel              # (community) Simplifying FPGA management in Kubernetes
        istio                # (community) Core Istio service mesh services
        jaeger               # (community) Kubernetes Jaeger operator with its simple config
        kata                 # (community) Kata Containers is a secure runtime with lightweight VMS
        keda                 # (community) Kubernetes-based Event Driven Autoscaling
        knative              # (community) Knative Serverless and Event Driven Applications
        kubearmor            # (community) Cloud-native runtime security enforcement system for k8s
        kwasm                # (community) WebAssembly support for WasmEdge (Docker Wasm) and Spin (Azure AKS WASI)
        linkerd              # (community) Linkerd is a service mesh for Kubernetes and other frameworks
        microcks             # (community) Open source Kubernetes Native tool for API Mocking and Testing
        multus               # (community) Multus CNI enables attaching multiple network interfaces to pods
        nfs                  # (community) NFS Server Provisioner
        ngrok                # (community) ngrok Ingress Controller instantly adds connectivity, load balancing, authentication, and observability to your services
        openebs              # (community) OpenEBS is the open-source storage solution for Kubernetes
        openfaas             # (community) OpenFaaS serverless framework
        osm-edge             # (community) osm-edge is a lightweight SMI compatible service mesh for the edge-computing.
        parking              # (community) Static webserver to park a domain. Works with EasyHAProxy.
        portainer            # (community) Portainer UI for your Kubernetes cluster
        shifu                # (community) Kubernetes native IoT software development framework.
        sosivio              # (community) Kubernetes Predictive Troubleshooting, Observability, and Resource Optimization
        stunner              # (community) A Kubernetes media gateway for WebRTC
        traefik              # (community) traefik Ingress controller
        trivy                # (community) Kubernetes-native security scanner
        cis-hardening        # (core) Apply CIS K8s hardening
        gpu                  # (core) Alias to nvidia add-on
        host-access          # (core) Allow Pods connecting to Host services smoothly
        kube-ovn             # (core) An advanced network fabric for Kubernetes
        mayastor             # (core) OpenEBS MayaStor
        minio                # (core) MinIO object storage
        nvidia               # (core) NVIDIA hardware (GPU and network) support
        prometheus           # (core) Prometheus operator for monitoring and logging
        rbac                 # (core) Role-Based Access Control for authorisation
        registry             # (core) Private image registry exposed on localhost:32000
        rook-ceph            # (core) Distributed Ceph storage using Rook

운영 관리

IP 변경 시 인증서 업데이트

클러스터의 IP 변경 시 인증서를 갱신해야 합니다. 다음 명령어로 인증서를 업데이트합니다.

# 인증서는 내부 기존의 IP 정보를 포함하고 있기 때문에, IP가 변경될 경우 반드시 인증서를 갱신해 주어야 합니다.
$ sudo microk8s refresh-certs --cert

Loadbalancer 설정

Metallb Addon을 활성화하여 로드밸런서를 설정하고, IP 범위를 지정할 수 있습니다.

microk8s enable metallb
 
# IP 범위 설정 예시:
microk8s configure metallb:192.168.219.2-192.168.219.50

관리자 계정 생성

Kubernetes 클러스터의 관리 권한을 가지는 관리자 계정을 생성합니다.

apiVersion: v1
kind: ServiceAccount
metadata:
  name: admin-user
  namespace: kube-system
 
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: admin-user
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: cluster-admin
subjects:
- kind: ServiceAccount
  name: admin-user
  namespace: kube-system
 
---
apiVersion: v1
kind: Secret
metadata:
  name: admin-user-token
  namespace: kube-system
  annotations:
    kubernetes.io/service-account.name: admin-user
type: kubernetes.io/service-account-token

AWS ECR Image Pull Token Secret 생성

ECR에서 이미지를 가져올 수 있도록 Docker 레지스트리 인증 토큰을 설정합니다.

microk8s kubectl create secret docker-registry ap-northeast-2-ecr-registry \
  --docker-server=012634413971.dkr.ecr.ap-northeast-2.amazonaws.com \
  --docker-username=AWS \
  --docker-password=$(aws ecr get-login-password --region ap-northeast-2) \
  --docker-email=walter.jung@luxrobo.com -n app

ArgoCD 자동 업데이트 설정

ECR 이미지 자동 업데이트를 위한 ArgoCD 설정을 구성합니다.

apiVersion: v1
kind: Secret
stringData:
  enableOCI: "true"
  name: helm-ecr
  type: helm
  url: 012634413971.dkr.ecr.ap-northeast-2.amazonaws.com
  username: AWS
  password: ""
metadata:
  labels:
    argocd-ecr-updater: enabled
    argocd.argoproj.io/secret-type: repository
  name: helm-ecr

cert-manager 설정

인증서 발급을 위한 cert-manager의 설정을 DNS 인증(dns01) 방식과 HTTP 인증(http01) 방식으로 구분하여 구성합니다.

http01방식이 안된경우 dns01로 처리한다. 이번에는 내부 acme self check 오류로 dns01로 변경했다. 그에 따른 aws 인증서등 보안처리가 추가되어야 한다.

DNS 인증 방식 (dns01)

DNS 인증(dns01)을 위해 AWS Route53과 연동합니다.

apiVersion: cert-manager.io/v1
kind: ClusterIssuer
metadata:
  name: letsencrypt-prod
  namespace: cert-manager
spec:
  acme:
    server: https://acme-v02.api.letsencrypt.org/directory
    email: walter.jung@luxrobo.com
    privateKeySecretRef:
      name: letsencrypt-prod
    solvers:
    - dns01:
        route53:
          region: ap-northeast-2
          accessKeyID: AKIAIOSFODNN7EXAMPLE
          secretAccessKeySecretRef:
            name: route53-credentials
            key: secret-access-key

AWS Iam생성

AWS IAM에서 Route53 인증을 위한 Access Key와 Secret Access Key를 Base64로 인코딩하여 Secret으로 저장합니다.

$ echo -n 'AKIAIOSFODNN7EXAMPLE' | base64
# 출력: QUtJQUlPU0ZPRFNOQzdFWEFNUExF
 
$ echo -n 'wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY' | base64
# 출력: d0phbHJYRXJuRkVNSS9LN01ERU5HL2JPdFJmaUNZRVhBTVBMRTlLWQ==
apiVersion: v1
kind: Secret
metadata:
  name: route53-credentials
  namespace: cert-manager
type: Opaque
data:
  access-key-id: <base64-encoded-access-key-id>
  secret-access-key: <base64-encoded-secret-access-key>

HTTP 인증 방식 (http01)

HTTP 인증(http01) 방식으로 인증서를 발급할 수 있습니다.

---
apiVersion: cert-manager.io/v1
kind: ClusterIssuer
metadata:
  name: letsencrypt-prod
  namespace: kube-system
spec:
  acme:
    # The ACME server URL
    server: https://acme-v02.api.letsencrypt.org/directory
    # Email address used for ACME registration
    email: walter.jung@luxrobo.com
    # Name of a secret used to store the ACME account private key
    privateKeySecretRef:
      name: letsencrypt-prod
    # Enable the HTTP-01 challenge provider
    solvers:
      - http01:
          ingress:
            class: public
OS 설치Argo 설치