Notice
Recent Posts
Recent Comments
Link
| 일 | 월 | 화 | 수 | 목 | 금 | 토 |
|---|---|---|---|---|---|---|
| 1 | 2 | |||||
| 3 | 4 | 5 | 6 | 7 | 8 | 9 |
| 10 | 11 | 12 | 13 | 14 | 15 | 16 |
| 17 | 18 | 19 | 20 | 21 | 22 | 23 |
| 24 | 25 | 26 | 27 | 28 | 29 | 30 |
Tags
- 모듈구조
- 매직미러
- 콕스엔데버
- zwift
- 관리노드설정
- 검색엔진노출
- 모듈추가
- 라즈베리파이4B
- 원격접속설정
- k8s
- 자바스크립트
- 트림원라이트이슈
- 스토리지할당
- 라즈베리파이
- 신호드랍
- Kubernetes
- 거절메일
- 글자사라짐
- 외부에서접근설정
- magicmirro
- magicmirror
- 트림원라이트
- 볼륨할당
- 쯔위프트
- 외부볼륨
- raspberrypie4b
- 쿠버네티스
- config.js
- 신호끊김
- 즈위프트
Archives
- Today
- Total
IT를 공부하자
인터넷 접속이 불가능한 오프라인 환경의 환경에서 Kubernetes 구성하기 본문
반응형
오프라인 환경의 Ubuntu 2204 LTS 버전에서 Kubernetes 구성하기
ubuntu 서버 설치(오프라인 서버)
3개의 노드를 준비해서 설치 이미지를 이용하여 서버를 설치한다.
ubuntu 서버 설치(온라인 서버)
인터넷이 되는 환경에서 동일한 ubuntu 서버를 설치하고 이 서버에서 필요한 패키지를 다운로드 받아 인터넷이 안되는 환경의 서버로 필요한 파일을 전송하는 역할을 수행한다.
Docker 다운로드 및 설치(온라인 서버)
온라인서버에서 오프라인 서버의 패키지 설치를 위한 패키지 다운로드
오프라인 환경에 설치가 가능하도록 온라인 서버에서 아래와 같이 docker를 설치하기 위한 패키지를 다운로드 받는다.
# 도커 공식 GPG 키 추가
root@ubuntu2204:/var/cache/apt/archives# curl -fsSL https://download.docker.com/linux/ubuntu/gpg | sudo gpg --dearmor -o /usr/share/keyrings/docker-archive-keyring.gpg
# Stable 리포지토리 설정
root@ubuntu2204:/var/cache/apt/archives# echo \
"deb [arch=$(dpkg --print-architecture) signed-by=/usr/share/keyrings/docker-archive-keyring.gpg] https://download.docker.com/linux/ubuntu \
$(lsb_release -cs) stable" | sudo tee /etc/apt/sources.list.d/docker.list > /dev/null
# 업데이트
root@ubuntu2204:/var/cache/apt/archives# apt-get update
Hit:1 http://mirror.kakao.com/ubuntu jammy InRelease
Hit:2 http://mirror.kakao.com/ubuntu jammy-updates InRelease
Hit:3 http://mirror.kakao.com/ubuntu jammy-backports InRelease
Get:4 https://download.docker.com/linux/ubuntu jammy InRelease [48.8 kB]
Hit:5 http://mirror.kakao.com/ubuntu jammy-security InRelease
Get:6 https://download.docker.com/linux/ubuntu jammy/stable amd64 Packages [23.0 kB]
Fetched 71.8 kB in 1s (64.1 kB/s)
Reading package lists... Done
download-only 옵션을 통해 온라인 서버에서 필요한 패키지만 다운로드한다.
기본 다운로드 경로는 /var/cache/apt/archives 디렉토리이다.
다운로드 받을 패키지는 아래와 같이 도커와 컨테이너 패키지이다.
- docker-ce
- docker-ce-cli
- containerd.io
root@ubuntu2204:/var/cache/apt/archives# apt-get install --download-only docker-ce docker-ce-cli containerd.io
Reading package lists... Done
Building dependency tree... Done
Reading state information... Done
The following additional packages will be installed:
docker-buildx-plugin docker-ce-rootless-extras docker-compose-plugin libltdl7 libslirp0 pigz slirp4netns
Suggested packages:
aufs-tools cgroupfs-mount | cgroup-lite
The following NEW packages will be installed:
containerd.io docker-buildx-plugin docker-ce docker-ce-cli docker-ce-rootless-extras docker-compose-plugin libltdl7 libslirp0 pigz slirp4netns
0 upgraded, 10 newly installed, 0 to remove and 51 not upgraded.
Need to get 115 MB of archives.
After this operation, 411 MB of additional disk space will be used.
Do you want to continue? [Y/n]
Get:1 http://mirror.kakao.com/ubuntu jammy/universe amd64 pigz amd64 2.6-1 [63.6 kB]
Get:2 https://download.docker.com/linux/ubuntu jammy/stable amd64 containerd.io amd64 1.6.26-1 [29.5 MB]
Get:3 http://mirror.kakao.com/ubuntu jammy/main amd64 libltdl7 amd64 2.4.6-15build2 [39.6 kB]
Get:4 http://mirror.kakao.com/ubuntu jammy/main amd64 libslirp0 amd64 4.6.1-1build1 [61.5 kB]
Get:5 http://mirror.kakao.com/ubuntu jammy/universe amd64 slirp4netns amd64 1.0.1-2 [28.2 kB]
Get:6 https://download.docker.com/linux/ubuntu jammy/stable amd64 docker-buildx-plugin amd64 0.11.2-1~ubuntu.22.04~jammy [28.2 MB]
Get:7 https://download.docker.com/linux/ubuntu jammy/stable amd64 docker-ce-cli amd64 5:24.0.7-1~ubuntu.22.04~jammy [13.3 MB]
Get:8 https://download.docker.com/linux/ubuntu jammy/stable amd64 docker-ce amd64 5:24.0.7-1~ubuntu.22.04~jammy [22.6 MB]
Get:9 https://download.docker.com/linux/ubuntu jammy/stable amd64 docker-ce-rootless-extras amd64 5:24.0.7-1~ubuntu.22.04~jammy [9,030 kB]
Get:10 https://download.docker.com/linux/ubuntu jammy/stable amd64 docker-compose-plugin amd64 2.21.0-1~ubuntu.22.04~jammy [11.9 MB]
Fetched 115 MB in 4s (29.1 MB/s)
Download complete and in download only mode
# 다운로드 받은 패키지를 확인하고
root@ubuntu2204:/var/cache/apt/archives# ls -la
total 112096
drwxr-xr-x 3 root root 4096 Jan 12 00:46 .
drwxr-xr-x 3 root root 4096 Jan 12 00:43 ..
-rw-r--r-- 1 root root 29479052 Jan 9 14:26 containerd.io_1.6.26-1_amd64.deb
-rw-r--r-- 1 root root 28194650 Jul 24 19:15 docker-buildx-plugin_0.11.2-1~ubuntu.22.04~jammy_amd64.deb
-rw-r--r-- 1 root root 22627784 Nov 16 13:26 docker-ce_5%3a24.0.7-1~ubuntu.22.04~jammy_amd64.deb
-rw-r--r-- 1 root root 13285356 Nov 16 13:26 docker-ce-cli_5%3a24.0.7-1~ubuntu.22.04~jammy_amd64.deb
-rw-r--r-- 1 root root 9029678 Nov 16 13:26 docker-ce-rootless-extras_5%3a24.0.7-1~ubuntu.22.04~jammy_amd64.deb
-rw-r--r-- 1 root root 11942730 Sep 18 14:38 docker-compose-plugin_2.21.0-1~ubuntu.22.04~jammy_amd64.deb
-rw-r--r-- 1 root root 39582 Mar 25 2022 libltdl7_2.4.6-15build2_amd64.deb
-rw-r--r-- 1 root root 61494 Mar 25 2022 libslirp0_4.6.1-1build1_amd64.deb
-rw-r----- 1 root root 0 Aug 10 00:21 lock
drwx------ 2 _apt root 4096 Jan 12 00:46 partial
-rw-r--r-- 1 root root 63592 Feb 6 2021 pigz_2.6-1_amd64.deb
-rw-r--r-- 1 root root 28168 Apr 29 2021 slirp4netns_1.0.1-2_amd64.deb
# 해당 받은 파일일들을 오프라인 환경의 서버에 전송하기 위해 묶는 작업을 수행한다.
root@ubuntu2204:/var/cache/apt/archives# tar cvf u2204_docker.tar *.deb
containerd.io_1.6.26-1_amd64.deb
docker-buildx-plugin_0.11.2-1~ubuntu.22.04~jammy_amd64.deb
docker-ce_5%3a24.0.7-1~ubuntu.22.04~jammy_amd64.deb
docker-ce-cli_5%3a24.0.7-1~ubuntu.22.04~jammy_amd64.deb
docker-ce-rootless-extras_5%3a24.0.7-1~ubuntu.22.04~jammy_amd64.deb
docker-compose-plugin_2.21.0-1~ubuntu.22.04~jammy_amd64.deb
libltdl7_2.4.6-15build2_amd64.deb
libslirp0_4.6.1-1build1_amd64.deb
pigz_2.6-1_amd64.deb
slirp4netns_1.0.1-2_amd64.deb
도커 설치(온라인, 오프라인 서버 모두 작업)
온라인 서버에서 도커를 설치하기 위해 패키지를 오프라인 서버로 전송한다.
온라인 서버와 오프라인 서버 모두 도커를 설치한다.
다운로드 한 이미지를 한꺼번에 설치한다.
root@u-kuber01:/home/emc# mkdir docker
root@u-kuber01:/home/emc# mv u2204_docker.tar docker/
root@u-kuber01:/home/emc# cd docker/
root@u-kuber01:/home/emc/docker# tar xvf u2204_docker.tar
containerd.io_1.6.26-1_amd64.deb
docker-buildx-plugin_0.11.2-1~ubuntu.22.04~jammy_amd64.deb
docker-ce_5%3a24.0.7-1~ubuntu.22.04~jammy_amd64.deb
docker-ce-cli_5%3a24.0.7-1~ubuntu.22.04~jammy_amd64.deb
docker-ce-rootless-extras_5%3a24.0.7-1~ubuntu.22.04~jammy_amd64.deb
docker-compose-plugin_2.21.0-1~ubuntu.22.04~jammy_amd64.deb
libltdl7_2.4.6-15build2_amd64.deb
libslirp0_4.6.1-1build1_amd64.deb
pigz_2.6-1_amd64.deb
slirp4netns_1.0.1-2_amd64.deb
root@u-kuber01:/home/emc/docker# ls -la
total 224172
drwxr-xr-x 2 root root 4096 Jan 12 00:02 .
drwxr-xr-x 5 emc emc 4096 Jan 12 00:02 ..
-rw-r--r-- 1 root root 29479052 Jan 9 14:26 containerd.io_1.6.26-1_amd64.deb
-rw-r--r-- 1 root root 28194650 Jul 24 19:15 docker-buildx-plugin_0.11.2-1~ubuntu.22.04~jammy_amd64.deb
-rw-r--r-- 1 root root 22627784 Nov 16 13:26 docker-ce_5%3a24.0.7-1~ubuntu.22.04~jammy_amd64.deb
-rw-r--r-- 1 root root 13285356 Nov 16 13:26 docker-ce-cli_5%3a24.0.7-1~ubuntu.22.04~jammy_amd64.deb
-rw-r--r-- 1 root root 9029678 Nov 16 13:26 docker-ce-rootless-extras_5%3a24.0.7-1~ubuntu.22.04~jammy_amd64.deb
-rw-r--r-- 1 root root 11942730 Sep 18 14:38 docker-compose-plugin_2.21.0-1~ubuntu.22.04~jammy_amd64.deb
-rw-r--r-- 1 root root 39582 Mar 25 2022 libltdl7_2.4.6-15build2_amd64.deb
-rw-r--r-- 1 root root 61494 Mar 25 2022 libslirp0_4.6.1-1build1_amd64.deb
-rw-r--r-- 1 root root 63592 Feb 6 2021 pigz_2.6-1_amd64.deb
-rw-r--r-- 1 root root 28168 Apr 29 2021 slirp4netns_1.0.1-2_amd64.deb
-rw-r--r-- 1 emc emc 114769920 Jan 12 00:01 u2204_docker.tar
root@u-kuber01:/home/emc# dpkg -R --install docker/
(Reading database ... 72354 files and directories currently installed.)
Preparing to unpack .../containerd.io_1.6.26-1_amd64.deb ...
Unpacking containerd.io (1.6.26-1) over (1.6.26-1) ...
Preparing to unpack .../docker-buildx-plugin_0.11.2-1~ubuntu.22.04~jammy_amd64.deb ...
Unpacking docker-buildx-plugin (0.11.2-1~ubuntu.22.04~jammy) over (0.11.2-1~ubuntu.22.04~jammy) ...
Preparing to unpack .../docker-ce-cli_5%3a24.0.7-1~ubuntu.22.04~jammy_amd64.deb ...
Unpacking docker-ce-cli (5:24.0.7-1~ubuntu.22.04~jammy) over (5:24.0.7-1~ubuntu.22.04~jammy) ...
Preparing to unpack .../docker-ce-rootless-extras_5%3a24.0.7-1~ubuntu.22.04~jammy_amd64.deb ...
Unpacking docker-ce-rootless-extras (5:24.0.7-1~ubuntu.22.04~jammy) over (5:24.0.7-1~ubuntu.22.04~jammy) ...
Preparing to unpack .../docker-ce_5%3a24.0.7-1~ubuntu.22.04~jammy_amd64.deb ...
Unpacking docker-ce (5:24.0.7-1~ubuntu.22.04~jammy) over (5:24.0.7-1~ubuntu.22.04~jammy) ...
Preparing to unpack .../docker-compose-plugin_2.21.0-1~ubuntu.22.04~jammy_amd64.deb ...
Unpacking docker-compose-plugin (2.21.0-1~ubuntu.22.04~jammy) over (2.21.0-1~ubuntu.22.04~jammy) ...
Preparing to unpack .../libltdl7_2.4.6-15build2_amd64.deb ...
Unpacking libltdl7:amd64 (2.4.6-15build2) over (2.4.6-15build2) ...
Preparing to unpack .../libslirp0_4.6.1-1build1_amd64.deb ...
Unpacking libslirp0:amd64 (4.6.1-1build1) over (4.6.1-1build1) ...
Preparing to unpack docker//pigz_2.6-1_amd64.deb ...
Unpacking pigz (2.6-1) over (2.6-1) ...
Preparing to unpack ...//slirp4netns_1.0.1-2_amd64.deb ...
Unpacking slirp4netns (1.0.1-2) over (1.0.1-2) ...
dpkg: dependency problems prevent configuration of containerd.io:
containerd.io depends on libc6 (>= 2.34); however:
Version of libc6:amd64 on system is 2.31-0ubuntu9.9.
dpkg: error processing package containerd.io (--install):
dependency problems - leaving unconfigured
Setting up docker-buildx-plugin (0.11.2-1~ubuntu.22.04~jammy) ...
dpkg: dependency problems prevent configuration of docker-ce-cli:
docker-ce-cli depends on libc6 (>= 2.34); however:
Version of libc6:amd64 on system is 2.31-0ubuntu9.9.
dpkg: error processing package docker-ce-cli (--install):
dependency problems - leaving unconfigured
dpkg: dependency problems prevent configuration of docker-ce-rootless-extras:
docker-ce-rootless-extras depends on libc6 (>= 2.34); however:
Version of libc6:amd64 on system is 2.31-0ubuntu9.9.
dpkg: error processing package docker-ce-rootless-extras (--install):
dependency problems - leaving unconfigured
dpkg: dependency problems prevent configuration of docker-ce:
docker-ce depends on containerd.io (>= 1.6.4); however:
Package containerd.io is not configured yet.
docker-ce depends on docker-ce-cli; however:
Package docker-ce-cli is not configured yet.
docker-ce depends on libc6 (>= 2.34); however:
Version of libc6:amd64 on system is 2.31-0ubuntu9.9.
dpkg: error processing package docker-ce (--install):
dependency problems - leaving unconfigured
Setting up docker-compose-plugin (2.21.0-1~ubuntu.22.04~jammy) ...
dpkg: dependency problems prevent configuration of libltdl7:amd64:
libltdl7:amd64 depends on libc6 (>= 2.34); however:
Version of libc6:amd64 on system is 2.31-0ubuntu9.9.
dpkg: error processing package libltdl7:amd64 (--install):
dependency problems - leaving unconfigured
dpkg: dependency problems prevent configuration of libslirp0:amd64:
libslirp0:amd64 depends on libc6 (>= 2.33); however:
Version of libc6:amd64 on system is 2.31-0ubuntu9.9.
dpkg: error processing package libslirp0:amd64 (--install):
dependency problems - leaving unconfigured
dpkg: dependency problems prevent configuration of pigz:
pigz depends on libc6 (>= 2.33); however:
Version of libc6:amd64 on system is 2.31-0ubuntu9.9.
dpkg: error processing package pigz (--install):
dependency problems - leaving unconfigured
dpkg: dependency problems prevent configuration of slirp4netns:
slirp4netns depends on libslirp0 (>= 4.1.0); however:
Package libslirp0:amd64 is not configured yet.
dpkg: error processing package slirp4netns (--install):
dependency problems - leaving unconfigured
Processing triggers for man-db (2.9.1-1) ...
Processing triggers for systemd (245.4-4ubuntu3.18) ...
Processing triggers for libc-bin (2.31-0ubuntu9.9) ...
Errors were encountered while processing:
containerd.io
docker-ce-cli
docker-ce-rootless-extras
docker-ce
libltdl7:amd64
libslirp0:amd64
pigz
slirp4netns
root@u-kuber01:/home/emc# apt-get -f install
Reading package lists... Done
Building dependency tree
Reading state information... Done
Correcting dependencies... Done
The following additional packages will be installed:
libc-bin libc6 libnsl2 libnss-nis libnss-nisplus libtirpc-common libtirpc3 locales
Suggested packages:
glibc-doc
The following NEW packages will be installed:
libnsl2 libnss-nis libnss-nisplus libtirpc-common libtirpc3
The following packages will be upgraded:
libc-bin libc6 locales
3 upgraded, 5 newly installed, 0 to remove and 538 not upgraded.
8 not fully installed or removed.
Need to get 0 B/8,371 kB of archives.
After this operation, 433 kB disk space will be freed.
Do you want to continue? [Y/n]
Preconfiguring packages ...
(Reading database ... 72354 files and directories currently installed.)
Preparing to unpack .../locales_2.35-0ubuntu3_all.deb ...
Unpacking locales (2.35-0ubuntu3) over (2.31-0ubuntu9.9) ...
Preparing to unpack .../libc6_2.35-0ubuntu3_amd64.deb ...
Checking for services that may need to be restarted...
Checking init scripts...
Checking for services that may need to be restarted...
Checking init scripts...
Stopping some services possibly affected by the upgrade (will be restarted later):
cron: stopping...done.
Unpacking libc6:amd64 (2.35-0ubuntu3) over (2.31-0ubuntu9.9) ...
Setting up libc6:amd64 (2.35-0ubuntu3) ...
Checking for services that may need to be restarted...
Checking init scripts...
Restarting services possibly affected by the upgrade:
cron: restarting...done.
atd: restarting...done.
Services restarted successfully.
(Reading database ... 72336 files and directories currently installed.)
Preparing to unpack .../libc-bin_2.35-0ubuntu3_amd64.deb ...
Unpacking libc-bin (2.35-0ubuntu3) over (2.31-0ubuntu9.9) ...
Setting up libc-bin (2.35-0ubuntu3) ...
Selecting previously unselected package libtirpc-common.
(Reading database ... 72334 files and directories currently installed.)
Preparing to unpack .../libtirpc-common_1.3.2-2build1_all.deb ...
Unpacking libtirpc-common (1.3.2-2build1) ...
Selecting previously unselected package libtirpc3:amd64.
Preparing to unpack .../libtirpc3_1.3.2-2build1_amd64.deb ...
Unpacking libtirpc3:amd64 (1.3.2-2build1) ...
Selecting previously unselected package libnsl2:amd64.
Preparing to unpack .../libnsl2_1.3.0-2build2_amd64.deb ...
Unpacking libnsl2:amd64 (1.3.0-2build2) ...
Selecting previously unselected package libnss-nis:amd64.
Preparing to unpack .../libnss-nis_3.1-0ubuntu6_amd64.deb ...
Unpacking libnss-nis:amd64 (3.1-0ubuntu6) ...
Selecting previously unselected package libnss-nisplus:amd64.
Preparing to unpack .../libnss-nisplus_1.3-0ubuntu6_amd64.deb ...
Unpacking libnss-nisplus:amd64 (1.3-0ubuntu6) ...
Setting up libtirpc-common (1.3.2-2build1) ...
Setting up locales (2.35-0ubuntu3) ...
Installing new version of config file /etc/locale.alias ...
Generating locales (this might take a while)...
en_US.UTF-8... done
Generation complete.
Setting up containerd.io (1.6.26-1) ...
Created symlink /etc/systemd/system/multi-user.target.wants/containerd.service → /lib/systemd/system/containerd.service.
Setting up libltdl7:amd64 (2.4.6-15build2) ...
Setting up docker-ce-cli (5:24.0.7-1~ubuntu.22.04~jammy) ...
Setting up libslirp0:amd64 (4.6.1-1build1) ...
Setting up pigz (2.6-1) ...
Setting up docker-ce-rootless-extras (5:24.0.7-1~ubuntu.22.04~jammy) ...
Setting up libtirpc3:amd64 (1.3.2-2build1) ...
Setting up slirp4netns (1.0.1-2) ...
Setting up docker-ce (5:24.0.7-1~ubuntu.22.04~jammy) ...
Created symlink /etc/systemd/system/multi-user.target.wants/docker.service → /lib/systemd/system/docker.service.
Created symlink /etc/systemd/system/sockets.target.wants/docker.socket → /lib/systemd/system/docker.socket.
Setting up libnsl2:amd64 (1.3.0-2build2) ...
Setting up libnss-nisplus:amd64 (1.3-0ubuntu6) ...
Setting up libnss-nis:amd64 (3.1-0ubuntu6) ...
Processing triggers for man-db (2.9.1-1) ...
Processing triggers for libc-bin (2.35-0ubuntu3) ...
Processing triggers for systemd (245.4-4ubuntu3.18) ...
root@u-kuber01:/home/emc# docker version
Client: Docker Engine - Community
Version: 24.0.7
API version: 1.43
Go version: go1.20.10
Git commit: afdd53b
Built: Thu Oct 26 09:07:41 2023
OS/Arch: linux/amd64
Context: default
Server: Docker Engine - Community
Engine:
Version: 24.0.7
API version: 1.43 (minimum version 1.12)
Go version: go1.20.10
Git commit: 311b9ff
Built: Thu Oct 26 09:07:41 2023
OS/Arch: linux/amd64
Experimental: false
containerd:
Version: 1.6.26
GitCommit: 3dd1e886e55dd695541fdcd67420c2888645a495
runc:
Version: 1.1.10
GitCommit: v1.1.10-0-g18a0cb0
docker-init:
Version: 0.19.0
GitCommit: de40ad0
Kubernetes 서버 노드별 설정(오프라인 서버 작업)
기본적으로 서버별로 설정해야할 부분을 설정한다.
root@u-kuber01:/home/emc/docker# swapoff -a && sudo sed -i '/swap/s/^/#/' /etc/fstab
root@u-kuber01:/home/emc/docker# cat <<EOF | sudo tee /etc/modules-load.d/k8s.conf
> br_netfilter
> EOF
br_netfilter
root@u-kuber01:/home/emc/docker# cat <<EOF | sudo tee /etc/sysctl.d/k8s.conf
> net.bridge.bridge-nf-call-ip6tables = 1
> net.bridge.bridge-nf-call-iptables = 1
> EOF
net.bridge.bridge-nf-call-ip6tables = 1
net.bridge.bridge-nf-call-iptables = 1
root@u-kuber01:/home/emc/docker# sudo sysctl --system
* Applying /etc/sysctl.d/10-console-messages.conf ...
kernel.printk = 4 4 1 7
* Applying /etc/sysctl.d/10-ipv6-privacy.conf ...
net.ipv6.conf.all.use_tempaddr = 2
net.ipv6.conf.default.use_tempaddr = 2
* Applying /etc/sysctl.d/10-kernel-hardening.conf ...
kernel.kptr_restrict = 1
* Applying /etc/sysctl.d/10-link-restrictions.conf ...
fs.protected_hardlinks = 1
fs.protected_symlinks = 1
* Applying /etc/sysctl.d/10-magic-sysrq.conf ...
kernel.sysrq = 176
* Applying /etc/sysctl.d/10-network-security.conf ...
net.ipv4.conf.default.rp_filter = 2
net.ipv4.conf.all.rp_filter = 2
* Applying /etc/sysctl.d/10-ptrace.conf ...
kernel.yama.ptrace_scope = 1
* Applying /etc/sysctl.d/10-zeropage.conf ...
vm.mmap_min_addr = 65536
* Applying /usr/lib/sysctl.d/50-default.conf ...
net.ipv4.conf.default.promote_secondaries = 1
sysctl: setting key "net.ipv4.conf.all.promote_secondaries": Invalid argument
net.ipv4.ping_group_range = 0 2147483647
net.core.default_qdisc = fq_codel
fs.protected_regular = 1
fs.protected_fifos = 1
* Applying /usr/lib/sysctl.d/50-pid-max.conf ...
kernel.pid_max = 4194304
* Applying /etc/sysctl.d/99-sysctl.conf ...
* Applying /etc/sysctl.d/k8s.conf ...
net.bridge.bridge-nf-call-ip6tables = 1
net.bridge.bridge-nf-call-iptables = 1
* Applying /usr/lib/sysctl.d/protect-links.conf ...
fs.protected_fifos = 1
fs.protected_hardlinks = 1
fs.protected_regular = 2
fs.protected_symlinks = 1
* Applying /etc/sysctl.conf ...
root@u-kuber01:/home/emc/docker# ufw disable
Firewall stopped and disabled on system startup
오프라인 서버를 위한 우분투 리포지토리가 설정되어 있다면
오프라인 서버의 우분투 리포지토리가 설정되어 있다면 아래와 같은 방법으로 설치한다.
root@u-kuber01:/home/emc/docker# vi /etc/apt/sources.list
# See http://help.ubuntu.com/community/UpgradeNotes for how to upgrade to
# newer versions of the distribution.
deb http://10.118.99.68/ubuntu jammy main restricted
root@u-kuber01:/home/emc/docker# apt-get install -y apt-transport-https ca-certificates curl
Reading package lists... Done
Building dependency tree
Reading state information... Done
Note, selecting 'apt' instead of 'apt-transport-https'
The following additional packages will be installed:
apt-utils dpkg gcc-12-base libapt-pkg6.0 libcurl4 libgcrypt20 libgmp10 libgnutls30 libhogweed6 libldap-2.5-0 libnettle8 libsasl2-2 libsasl2-modules libsasl2-modules-db
libselinux1 libssl3 libstdc++6 libxxhash0
Suggested packages:
apt-doc aptitude | synaptic | wajig dpkg-dev debsig-verify rng-tools gnutls-bin libsasl2-modules-gssapi-mit | libsasl2-modules-gssapi-heimdal libsasl2-modules-ldap
libsasl2-modules-otp libsasl2-modules-sql
The following NEW packages will be installed:
gcc-12-base libhogweed6 libldap-2.5-0 libnettle8 libssl3 libxxhash0
The following packages will be upgraded:
apt apt-utils ca-certificates curl dpkg libapt-pkg6.0 libcurl4 libgcrypt20 libgmp10 libgnutls30 libsasl2-2 libsasl2-modules libsasl2-modules-db libselinux1 libstdc++6
15 upgraded, 6 newly installed, 0 to remove and 523 not upgraded.
Need to get 0 B/9,490 kB of archives.
After this operation, 8,101 kB of additional disk space will be used.
Preconfiguring packages ...
(Reading database ... 72363 files and directories currently installed.)
Preparing to unpack .../libselinux1_3.3-1build2_amd64.deb ...
Unpacking libselinux1:amd64 (3.3-1build2) over (3.0-1build2) ...
Setting up libselinux1:amd64 (3.3-1build2) ...
(Reading database ... 72363 files and directories currently installed.)
Preparing to unpack .../dpkg_1.21.1ubuntu2_amd64.deb ...
Unpacking dpkg (1.21.1ubuntu2) over (1.19.7ubuntu3.2) ...
Setting up dpkg (1.21.1ubuntu2) ...
Installing new version of config file /etc/cron.daily/dpkg ...
Created symlink /etc/systemd/system/timers.target.wants/dpkg-db-backup.timer → /lib/systemd/system/dpkg-db-backup.timer.
dpkg-db-backup.service is a disabled or a static unit not running, not starting it.
(Reading database ... 72390 files and directories currently installed.)
Preparing to unpack .../libgcrypt20_1.9.4-3ubuntu3_amd64.deb ...
Unpacking libgcrypt20:amd64 (1.9.4-3ubuntu3) over (1.8.5-5ubuntu1.1) ...
Setting up libgcrypt20:amd64 (1.9.4-3ubuntu3) ...
Selecting previously unselected package gcc-12-base:amd64.
(Reading database ... 72390 files and directories currently installed.)
Preparing to unpack .../gcc-12-base_12-20220319-1ubuntu1_amd64.deb ...
Unpacking gcc-12-base:amd64 (12-20220319-1ubuntu1) ...
Setting up gcc-12-base:amd64 (12-20220319-1ubuntu1) ...
(Reading database ... 72395 files and directories currently installed.)
Preparing to unpack .../libstdc++6_12-20220319-1ubuntu1_amd64.deb ...
Unpacking libstdc++6:amd64 (12-20220319-1ubuntu1) over (10.3.0-1ubuntu1~20.04) ...
Setting up libstdc++6:amd64 (12-20220319-1ubuntu1) ...
Selecting previously unselected package libxxhash0:amd64.
(Reading database ... 72395 files and directories currently installed.)
Preparing to unpack .../libxxhash0_0.8.1-1_amd64.deb ...
Unpacking libxxhash0:amd64 (0.8.1-1) ...
Setting up libxxhash0:amd64 (0.8.1-1) ...
(Reading database ... 72400 files and directories currently installed.)
Preparing to unpack .../libapt-pkg6.0_2.4.5_amd64.deb ...
Unpacking libapt-pkg6.0:amd64 (2.4.5) over (2.0.9) ...
Setting up libapt-pkg6.0:amd64 (2.4.5) ...
(Reading database ... 72400 files and directories currently installed.)
Preparing to unpack .../libgmp10_2%3a6.2.1+dfsg-3ubuntu1_amd64.deb ...
Unpacking libgmp10:amd64 (2:6.2.1+dfsg-3ubuntu1) over (2:6.2.0+dfsg-4ubuntu0.1) ...
Setting up libgmp10:amd64 (2:6.2.1+dfsg-3ubuntu1) ...
Selecting previously unselected package libnettle8:amd64.
(Reading database ... 72400 files and directories currently installed.)
Preparing to unpack .../libnettle8_3.7.3-1build2_amd64.deb ...
Unpacking libnettle8:amd64 (3.7.3-1build2) ...
Setting up libnettle8:amd64 (3.7.3-1build2) ...
Selecting previously unselected package libhogweed6:amd64.
(Reading database ... 72407 files and directories currently installed.)
Preparing to unpack .../libhogweed6_3.7.3-1build2_amd64.deb ...
Unpacking libhogweed6:amd64 (3.7.3-1build2) ...
Setting up libhogweed6:amd64 (3.7.3-1build2) ...
(Reading database ... 72412 files and directories currently installed.)
Preparing to unpack .../libgnutls30_3.7.3-4ubuntu1_amd64.deb ...
Unpacking libgnutls30:amd64 (3.7.3-4ubuntu1) over (3.6.13-2ubuntu1.7) ...
Setting up libgnutls30:amd64 (3.7.3-4ubuntu1) ...
(Reading database ... 72412 files and directories currently installed.)
Preparing to unpack .../archives/apt_2.4.5_amd64.deb ...
Unpacking apt (2.4.5) over (2.0.9) ...
Setting up apt (2.4.5) ...
Installing new version of config file /etc/cron.daily/apt-compat ...
(Reading database ... 72413 files and directories currently installed.)
Preparing to unpack .../0-apt-utils_2.4.5_amd64.deb ...
Unpacking apt-utils (2.4.5) over (2.0.9) ...
Selecting previously unselected package libssl3:amd64.
Preparing to unpack .../1-libssl3_3.0.2-0ubuntu1_amd64.deb ...
Unpacking libssl3:amd64 (3.0.2-0ubuntu1) ...
Preparing to unpack .../2-ca-certificates_20211016_all.deb ...
Unpacking ca-certificates (20211016) over (20211016~20.04.1) ...
Preparing to unpack .../3-libsasl2-modules-db_2.1.27+dfsg2-3ubuntu1_amd64.deb ...
Unpacking libsasl2-modules-db:amd64 (2.1.27+dfsg2-3ubuntu1) over (2.1.27+dfsg-2ubuntu0.1) ...
Preparing to unpack .../4-libsasl2-2_2.1.27+dfsg2-3ubuntu1_amd64.deb ...
Unpacking libsasl2-2:amd64 (2.1.27+dfsg2-3ubuntu1) over (2.1.27+dfsg-2ubuntu0.1) ...
Selecting previously unselected package libldap-2.5-0:amd64.
Preparing to unpack .../5-libldap-2.5-0_2.5.11+dfsg-1~exp1ubuntu3_amd64.deb ...
Unpacking libldap-2.5-0:amd64 (2.5.11+dfsg-1~exp1ubuntu3) ...
Preparing to unpack .../6-curl_7.81.0-1_amd64.deb ...
Unpacking curl (7.81.0-1) over (7.68.0-1ubuntu2.13) ...
Preparing to unpack .../7-libcurl4_7.81.0-1_amd64.deb ...
Unpacking libcurl4:amd64 (7.81.0-1) over (7.68.0-1ubuntu2.13) ...
Preparing to unpack .../8-libsasl2-modules_2.1.27+dfsg2-3ubuntu1_amd64.deb ...
Unpacking libsasl2-modules:amd64 (2.1.27+dfsg2-3ubuntu1) over (2.1.27+dfsg-2ubuntu0.1) ...
Setting up apt-utils (2.4.5) ...
Setting up libssl3:amd64 (3.0.2-0ubuntu1) ...
Setting up libsasl2-modules-db:amd64 (2.1.27+dfsg2-3ubuntu1) ...
Setting up ca-certificates (20211016) ...
Updating certificates in /etc/ssl/certs...
0 added, 0 removed; done.
Setting up libsasl2-2:amd64 (2.1.27+dfsg2-3ubuntu1) ...
Setting up libsasl2-modules:amd64 (2.1.27+dfsg2-3ubuntu1) ...
Installing new version of config file /etc/logcheck/ignore.d.server/libsasl2-modules ...
Setting up libldap-2.5-0:amd64 (2.5.11+dfsg-1~exp1ubuntu3) ...
Setting up libcurl4:amd64 (7.81.0-1) ...
Setting up curl (7.81.0-1) ...
Processing triggers for man-db (2.9.1-1) ...
Processing triggers for libc-bin (2.35-0ubuntu3) ...
Processing triggers for ca-certificates (20211016) ...
Updating certificates in /etc/ssl/certs...
0 added, 0 removed; done.
Running hooks in /etc/ca-certificates/update.d...
done.
쿠버네티스 패키지 다운로드(온라인 서버)
온라인 서버에서 쿠버네티스 관련 패키지를 다운로드한다.
다운로드가 완료되면 쿠버네티스 서버(오프라인 서버)에 모두 전송한다.
# 리포지토리 업데이트 후 최신 버전으로 업데이트
root@ubuntu2204:/var/cache/apt/archives# curl -fsSLo /usr/share/keyrings/kubernetes-archive-keyring.gpg https://packages.cloud.google.com/apt/doc/apt-key.gpg
root@ubuntu2204:/var/cache/apt/archives# echo "deb [signed-by=/usr/share/keyrings/kubernetes-archive-keyring.gpg] https://apt.kubernetes.io/ kubernetes-xenial main" | sudo tee /etc/apt/sources.list.d/kubernetes.list
deb [signed-by=/usr/share/keyrings/kubernetes-archive-keyring.gpg] https://apt.kubernetes.io/ kubernetes-xenial main
root@ubuntu2204:/var/cache/apt/archives# curl -fsSLo /usr/share/keyrings/kubernetes-archive-keyring.gpg https://dl.k8s.io/apt/doc/apt-key.gpg
root@ubuntu2204:/var/cache/apt/archives# echo "deb [signed-by=/usr/share/keyrings/kubernetes-archive-keyring.gpg] https://apt.kubernetes.io/ kubernetes-xenial main" | sudo tee /etc/apt/sources.list.d/kubernetes.list
deb [signed-by=/usr/share/keyrings/kubernetes-archive-keyring.gpg] https://apt.kubernetes.io/ kubernetes-xenial main
root@ubuntu2204:/var/cache/apt/archives# apt-get update
Hit:1 http://mirror.kakao.com/ubuntu jammy InRelease
Hit:2 http://mirror.kakao.com/ubuntu jammy-updates InRelease
Hit:3 http://mirror.kakao.com/ubuntu jammy-backports InRelease
Hit:4 http://mirror.kakao.com/ubuntu jammy-security InRelease
Hit:6 https://download.docker.com/linux/ubuntu jammy InRelease
Get:5 https://packages.cloud.google.com/apt kubernetes-xenial InRelease [8,993 B]
Get:7 https://packages.cloud.google.com/apt kubernetes-xenial/main amd64 Packages [69.9 kB]
Fetched 78.9 kB in 3s (28.9 kB/s)
Reading package lists... Done
# 다운로드 옵션으로 설치 하지 않고 다운로드만 함
root@ubuntu2204:/var/cache/apt/archives# apt-get install --download-only kubelet kubeadm kubectl
Reading package lists... Done
Building dependency tree... Done
Reading state information... Done
The following additional packages will be installed:
conntrack cri-tools ebtables kubernetes-cni socat
The following NEW packages will be installed:
conntrack cri-tools ebtables kubeadm kubectl kubelet kubernetes-cni socat
0 upgraded, 8 newly installed, 0 to remove and 51 not upgraded.
Need to get 87.1 MB of archives.
After this operation, 336 MB of additional disk space will be used.
Do you want to continue? [Y/n]
Get:1 http://mirror.kakao.com/ubuntu jammy/main amd64 conntrack amd64 1:1.4.6-2build2 [33.5 kB]
Get:2 http://mirror.kakao.com/ubuntu jammy/main amd64 ebtables amd64 2.0.11-4build2 [84.9 kB]
Get:3 http://mirror.kakao.com/ubuntu jammy/main amd64 socat amd64 1.7.4.1-3ubuntu4 [349 kB]
Get:4 https://packages.cloud.google.com/apt kubernetes-xenial/main amd64 cri-tools amd64 1.26.0-00 [18.9 MB]
Get:5 https://packages.cloud.google.com/apt kubernetes-xenial/main amd64 kubernetes-cni amd64 1.2.0-00 [27.6 MB]
Get:6 https://packages.cloud.google.com/apt kubernetes-xenial/main amd64 kubelet amd64 1.28.2-00 [19.5 MB]
Get:7 https://packages.cloud.google.com/apt kubernetes-xenial/main amd64 kubectl amd64 1.28.2-00 [10.3 MB]
Get:8 https://packages.cloud.google.com/apt kubernetes-xenial/main amd64 kubeadm amd64 1.28.2-00 [10.3 MB]
Fetched 87.1 MB in 19s (4,479 kB/s)
Download complete and in download only mode
# 다운로드 된 경로의 패키지 확인
root@ubuntu2204:/var/cache/apt/archives# ls -la
total 85076
drwxr-xr-x 3 root root 4096 Jan 12 01:33 .
drwxr-xr-x 3 root root 4096 Jan 12 01:32 ..
-rw-r--r-- 1 root root 33512 Mar 24 2022 conntrack_1%3a1.4.6-2build2_amd64.deb
-rw-r--r-- 1 root root 18944068 Jan 9 21:55 cri-tools_1.26.0-00_amd64.deb
-rw-r--r-- 1 root root 84856 Mar 24 2022 ebtables_2.0.11-4build2_amd64.deb
-rw-r--r-- 1 root root 10269572 Jan 11 10:46 kubeadm_1.28.2-00_amd64.deb
-rw-r--r-- 1 root root 10325168 Jan 11 10:35 kubectl_1.28.2-00_amd64.deb
-rw-r--r-- 1 root root 19494112 Jan 11 12:48 kubelet_1.28.2-00_amd64.deb
-rw-r--r-- 1 root root 27586224 Dec 22 15:33 kubernetes-cni_1.2.0-00_amd64.deb
-rw-r----- 1 root root 0 Aug 10 00:21 lock
drwx------ 2 _apt root 4096 Jan 12 01:33 partial
-rw-r--r-- 1 root root 349118 Mar 25 2022 socat_1.7.4.1-3ubuntu4_amd64.deb
# 패키지를 묶어서 오프라인 서버로 전송한다.
root@ubuntu2204:/var/cache/apt/archives# tar cvf u-k8s-1.28.2.tar *.deb
conntrack_1%3a1.4.6-2build2_amd64.deb
cri-tools_1.26.0-00_amd64.deb
ebtables_2.0.11-4build2_amd64.deb
kubeadm_1.28.2-00_amd64.deb
kubectl_1.28.2-00_amd64.deb
kubelet_1.28.2-00_amd64.deb
kubernetes-cni_1.2.0-00_amd64.deb
socat_1.7.4.1-3ubuntu4_amd64.deb
root@ubuntu2204:/var/cache/apt/archives# scp u-k8s-1.28.2.tar emc@10.118.99.114:/home/emc
emc@10.118.99.114's password:
u-k8s-1.28.2.tar 100% 83MB 8.9MB/s 00:09
쿠버네티스 패키지 설치(오프라인 서버, 모든 노드)
쿠버네티스 모든 노드에서 복사된 패키지의 압축을 풀고 패키지를 설치한다.
root@u-kuber01:/home/emc# mkdir kubernetes
root@u-kuber01:/home/emc# mv u-k8s-1.28.2.tar kubernetes/
root@u-kuber01:/home/emc# cd kubernetes/
root@u-kuber01:/home/emc/kubernetes# tar xvf u-k8s-1.28.2.tar
conntrack_1%3a1.4.6-2build2_amd64.deb
cri-tools_1.26.0-00_amd64.deb
ebtables_2.0.11-4build2_amd64.deb
kubeadm_1.28.2-00_amd64.deb
kubectl_1.28.2-00_amd64.deb
kubelet_1.28.2-00_amd64.deb
kubernetes-cni_1.2.0-00_amd64.deb
socat_1.7.4.1-3ubuntu4_amd64.deb
root@u-kuber01:/home/emc/kubernetes# dpkg -i *.deb
Selecting previously unselected package conntrack.
(Reading database ... 72433 files and directories currently installed.)
Preparing to unpack conntrack_1%3a1.4.6-2build2_amd64.deb ...
Unpacking conntrack (1:1.4.6-2build2) ...
Selecting previously unselected package cri-tools.
Preparing to unpack cri-tools_1.26.0-00_amd64.deb ...
Unpacking cri-tools (1.26.0-00) ...
Selecting previously unselected package ebtables.
Preparing to unpack ebtables_2.0.11-4build2_amd64.deb ...
Unpacking ebtables (2.0.11-4build2) ...
Selecting previously unselected package kubeadm.
Preparing to unpack kubeadm_1.28.2-00_amd64.deb ...
Unpacking kubeadm (1.28.2-00) ...
Selecting previously unselected package kubectl.
Preparing to unpack kubectl_1.28.2-00_amd64.deb ...
Unpacking kubectl (1.28.2-00) ...
Selecting previously unselected package kubelet.
Preparing to unpack kubelet_1.28.2-00_amd64.deb ...
Unpacking kubelet (1.28.2-00) ...
Selecting previously unselected package kubernetes-cni.
Preparing to unpack kubernetes-cni_1.2.0-00_amd64.deb ...
Unpacking kubernetes-cni (1.2.0-00) ...
Selecting previously unselected package socat.
Preparing to unpack socat_1.7.4.1-3ubuntu4_amd64.deb ...
Unpacking socat (1.7.4.1-3ubuntu4) ...
Setting up conntrack (1:1.4.6-2build2) ...
Setting up cri-tools (1.26.0-00) ...
Setting up ebtables (2.0.11-4build2) ...
Setting up kubectl (1.28.2-00) ...
Setting up kubernetes-cni (1.2.0-00) ...
Setting up socat (1.7.4.1-3ubuntu4) ...
Setting up kubelet (1.28.2-00) ...
Created symlink /etc/systemd/system/multi-user.target.wants/kubelet.service → /lib/systemd/system/kubelet.service.
Setting up kubeadm (1.28.2-00) ...
Processing triggers for man-db (2.9.1-1) ...
root@u-kuber01:/home/emc/kubernetes# apt-mark hold kubelet kubeadm kubectl
kubelet set on hold.
kubeadm set on hold.
kubectl set on hold.docker 및 containerd 설정 수정(오프라인 서버)
아래와 같이 기본 설정값을 수정한다.
root@u-kuber01:/home/emc/kubernetes# mkdir /etc/docker
mkdir: cannot create directory ‘/etc/docker’: File exists
root@u-kuber01:/home/emc/kubernetes# cat <<EOF | sudo tee /etc/docker/daemon.json
> {
> "exec-opts": ["native.cgroupdriver=systemd"],
> "log-driver": "json-file",
> "log-opts": {
> "max-size": "100m"
> },
> "storage-driver": "overlay2"
> }
> EOF
{
"exec-opts": ["native.cgroupdriver=systemd"],
"log-driver": "json-file",
"log-opts": {
"max-size": "100m"
},
"storage-driver": "overlay2"
root@u-kuber01:/home/emc/kubernetes# systemctl daemon-reload
root@u-kuber01:/home/emc/kubernetes# systemctl restart kubelet
root@u-kuber01:/home/emc/kubernetes# rm /etc/containerd/config.toml
root@u-kuber01:/home/emc/kubernetes# systemctl restart containerd
쿠버네티스 설치를 위한 컨테이너 이미지 다운로드(온라인 서버)
오프라인 환경이기 때문에 아래 명령을 수행해서 설치에 필요한 컨테이너 이미지를 온라인 서버에서 다운로드 후
로컬 리포지토리에 등록한다.
온라인 서버에도 동일하게 도커 환경을 설치해야한다.
다운로드 받은 이미지를 컨테이너 레지스트리 서버로 복사하여 임포트한다.
쿠버네티스 설치를 위한 컨테이너 이미지 목록 확인(오프라인 서버)
# 아래는 오프라인 서버에서 필요한 컨테이너 이미지 목록을 확인한다.
root@u-kuber01:/home/emc/kubernetes# kubeadm config images list
W0112 00:32:00.900870 33267 version.go:104] could not fetch a Kubernetes version from the internet: unable to get URL "https://dl.k8s.io/release/stable-1.txt": Get "https://dl.k8s.io/release/stable-1.txt": dial tcp: lookup dl.k8s.io on 127.0.0.53:53: no such host
W0112 00:32:00.900988 33267 version.go:105] falling back to the local client version: v1.28.2
registry.k8s.io/kube-apiserver:v1.28.2
registry.k8s.io/kube-controller-manager:v1.28.2
registry.k8s.io/kube-scheduler:v1.28.2
registry.k8s.io/kube-proxy:v1.28.2
registry.k8s.io/pause:3.9
registry.k8s.io/etcd:3.5.9-0
registry.k8s.io/coredns/coredns:v1.10.1
필요한 컨테이너 이미지 다운로드(온라인 서버)
위에서 확인한 이미지 목록을 온라인 서버에서 다운로드 받는다.
root@ubuntu2204:/apt-mirror/k8s-container-image# docker pull registry.k8s.io/kube-apiserver:v1.28.2
v1.28.2: Pulling from kube-apiserver
a7ca0d9ba68f: Pull complete
fe5ca62666f0: Pull complete
b02a7525f878: Pull complete
fcb6f6d2c998: Pull complete
e8c73c638ae9: Pull complete
1e3d9b7d1452: Pull complete
4aa0ea1413d3: Pull complete
7c881f9ab25e: Pull complete
5627a970d25e: Pull complete
167eb620404e: Pull complete
6021ef4e32c7: Pull complete
Digest: sha256:6beea2e5531a0606613594fd3ed92d71bbdcef99dd3237522049a0b32cad736c
Status: Downloaded newer image for registry.k8s.io/kube-apiserver:v1.28.2
registry.k8s.io/kube-apiserver:v1.28.2
root@ubuntu2204:/apt-mirror/k8s-container-image# docker pull registry.k8s.io/kube-controller-manager:v1.28.2
v1.28.2: Pulling from kube-controller-manager
a7ca0d9ba68f: Already exists
fe5ca62666f0: Already exists
b02a7525f878: Already exists
fcb6f6d2c998: Already exists
e8c73c638ae9: Already exists
1e3d9b7d1452: Already exists
4aa0ea1413d3: Already exists
7c881f9ab25e: Already exists
5627a970d25e: Already exists
167eb620404e: Already exists
0a7d2f84a9b9: Pull complete
Digest: sha256:6a42ce14d716205a99763f3c732c0a8f0ea041bdbbea7d2dfffcc53dafd7cac4
Status: Downloaded newer image for registry.k8s.io/kube-controller-manager:v1.28.2
registry.k8s.io/kube-controller-manager:v1.28.2
root@ubuntu2204:/apt-mirror/k8s-container-image# docker pull registry.k8s.io/kube-scheduler:v1.28.2
v1.28.2: Pulling from kube-scheduler
a7ca0d9ba68f: Already exists
fe5ca62666f0: Already exists
b02a7525f878: Already exists
fcb6f6d2c998: Already exists
e8c73c638ae9: Already exists
1e3d9b7d1452: Already exists
4aa0ea1413d3: Already exists
7c881f9ab25e: Already exists
5627a970d25e: Already exists
167eb620404e: Already exists
0227831a6c1e: Pull complete
Digest: sha256:6511193f8114a2f011790619698efe12a8119ed9a17e2e36f4c1c759ccf173ab
Status: Downloaded newer image for registry.k8s.io/kube-scheduler:v1.28.2
registry.k8s.io/kube-scheduler:v1.28.2
root@ubuntu2204:/apt-mirror/k8s-container-image# docker pull registry.k8s.io/kube-proxy:v1.28.2
v1.28.2: Pulling from kube-proxy
6d60eaba7df7: Pull complete
9c8d29e8ab26: Pull complete
Digest: sha256:41c8f92d1cd571e0e36af431f35c78379f84f5daf5b85d43014a9940d697afcf
Status: Downloaded newer image for registry.k8s.io/kube-proxy:v1.28.2
registry.k8s.io/kube-proxy:v1.28.2
root@ubuntu2204:/apt-mirror/k8s-container-image# docker pull registry.k8s.io/pause:3.9
3.9: Pulling from pause
61fec91190a0: Pull complete
Digest: sha256:7031c1b283388d2c2e09b57badb803c05ebed362dc88d84b480cc47f72a21097
Status: Downloaded newer image for registry.k8s.io/pause:3.9
registry.k8s.io/pause:3.9
root@ubuntu2204:/apt-mirror/k8s-container-image# docker pull registry.k8s.io/etcd:3.5.9-0
3.5.9-0: Pulling from etcd
a7ca0d9ba68f: Already exists
fe5ca62666f0: Already exists
b02a7525f878: Already exists
fcb6f6d2c998: Already exists
e8c73c638ae9: Already exists
1e3d9b7d1452: Already exists
4aa0ea1413d3: Already exists
7c881f9ab25e: Already exists
5627a970d25e: Already exists
3f4a72e37652: Pull complete
93182a730d98: Pull complete
5198587edd6d: Pull complete
Digest: sha256:e013d0d5e4e25d00c61a7ff839927a1f36479678f11e49502b53a5e0b14f10c3
Status: Downloaded newer image for registry.k8s.io/etcd:3.5.9-0
registry.k8s.io/etcd:3.5.9-0
root@ubuntu2204:/apt-mirror/k8s-container-image# docker pull registry.k8s.io/coredns/coredns:v1.10.1
v1.10.1: Pulling from coredns/coredns
25b7032c281a: Pull complete
3799eae1a077: Pull complete
Digest: sha256:a0ead06651cf580044aeb0a0feba63591858fb2e43ade8c9dea45a6a89ae7e5e
Status: Downloaded newer image for registry.k8s.io/coredns/coredns:v1.10.1
registry.k8s.io/coredns/coredns:v1.10.1
root@ubuntu2204:/apt-mirror/k8s-container-image# docker image list
REPOSITORY TAG IMAGE ID CREATED SIZE
registry.k8s.io/kube-apiserver v1.28.2 cdcab12b2dd1 4 months ago 126MB
registry.k8s.io/kube-proxy v1.28.2 c120fed2beb8 4 months ago 73.1MB
registry.k8s.io/kube-controller-manager v1.28.2 55f13c92defb 4 months ago 122MB
registry.k8s.io/kube-scheduler v1.28.2 7a5d9d67a13f 4 months ago 60.1MB
registry.k8s.io/etcd 3.5.9-0 73deb9a3f702 8 months ago 294MB
registry.k8s.io/coredns/coredns v1.10.1 ead0a4a53df8 11 months ago 53.6MB
registry.k8s.io/pause 3.9 e6f181688397 15 months ago 744kB다운로드 받은 이미지를 로컬 파일로 저장 후 로컬 레지스트리 서버에 전송(온라인 서버)
# 다운로드 받은 이미지를 파일로 내려 받는다.
root@ubuntu2204:/apt-mirror/k8s-container-image# docker save registry.k8s.io/kube-apiserver > kube-apiserver-v1.28.2.tar
root@ubuntu2204:/apt-mirror/k8s-container-image# docker save registry.k8s.io/kube-proxy > kube-proxy-v1.28.2.tar
root@ubuntu2204:/apt-mirror/k8s-container-image# docker save registry.k8s.io/kube-controller-manager > kube-controller-v1.28.2.tar
root@ubuntu2204:/apt-mirror/k8s-container-image# docker save registry.k8s.io/kube-scheduler > kube-scheduler-v1.28.2.tar
root@ubuntu2204:/apt-mirror/k8s-container-image# docker save registry.k8s.io/etcd > etcd-3.5.9-0.tar
root@ubuntu2204:/apt-mirror/k8s-container-image# docker save registry.k8s.io/coredns/coredns > coredns-v1.10.1.tar
root@ubuntu2204:/apt-mirror/k8s-container-image# docker save registry.k8s.io/pause > pause-3.9.tar
root@ubuntu2204:/apt-mirror/k8s-container-image# ls -la
total 719252
drwxr-xr-x 2 root root 4096 Jan 12 01:52 .
drwxr-xr-x 7 root root 4096 Jan 12 01:43 ..
-rw-r--r-- 1 root root 53628928 Jan 12 01:52 coredns-v1.10.1.tar
-rw-r--r-- 1 root root 295491072 Jan 12 01:51 etcd-3.5.9-0.tar
-rw-r--r-- 1 root root 127181312 Jan 12 01:50 kube-apiserver-v1.28.2.tar
-rw-r--r-- 1 root root 123203584 Jan 12 01:51 kube-controller-v1.28.2.tar
-rw-r--r-- 1 root root 74695680 Jan 12 01:51 kube-proxy-v1.28.2.tar
-rw-r--r-- 1 root root 61517824 Jan 12 01:51 kube-scheduler-v1.28.2.tar
-rw-r--r-- 1 root root 754176 Jan 12 01:52 pause-3.9.tar
# 내려받은 이미지 파일을 하나로 묶어서 오프라인 로컬 레지스트리 서버에 전송한다.
root@ubuntu2204:/apt-mirror# tar cvf k8s-container-image.tar k8s-container-image/
k8s-container-image/
k8s-container-image/etcd-3.5.9-0.tar
k8s-container-image/kube-apiserver-v1.28.2.tar
k8s-container-image/coredns-v1.10.1.tar
k8s-container-image/kube-proxy-v1.28.2.tar
k8s-container-image/kube-controller-v1.28.2.tar
k8s-container-image/kube-scheduler-v1.28.2.tar
k8s-container-image/pause-3.9.tar
root@ubuntu2204:/apt-mirror# scp k8s-container-image.tar 10.118.99.110:/root
The authenticity of host '10.118.99.110 (10.118.99.110)' can't be established.
ED25519 key fingerprint is SHA256:yf51pFIvmC47rhF4ZyqbISf+yFodzxOUa4i2Y4HavoE.
This key is not known by any other names
Are you sure you want to continue connecting (yes/no/[fingerprint])? yes
Warning: Permanently added '10.118.99.110' (ED25519) to the list of known hosts.
root@10.118.99.110's password:
k8s-container-image.tar 100% 702MB 6.6MB/s 01:46
root@ubuntu2204:/apt-mirror#
로컬 레지스트리 서버에 쿠버네티스 컨테이너 이미지 임포트(로컬 레지스트리 서버)
이 문서에서는 로컬 레지스트리 서버 구성에 대한 내용을 다루지 않았고 기 구축된 로컬 레지스트리 서버를 활용하여 기술하였다. 이 문서에서 로컬 레지스트리 서버는 goharbor라는 툴이고 IP는 10.118.99.110이다.
로컬 레지스트리 서버에 전송된 파일을 서버에 임포트한다.
# 전송된 이미지를 풀고
[root@registry ubuntu-k8s-container-image]# tar xvf k8s-container-image.tar
k8s-container-image/
k8s-container-image/etcd-3.5.9-0.tar
k8s-container-image/kube-apiserver-v1.28.2.tar
k8s-container-image/coredns-v1.10.1.tar
k8s-container-image/kube-proxy-v1.28.2.tar
k8s-container-image/kube-controller-v1.28.2.tar
k8s-container-image/kube-scheduler-v1.28.2.tar
k8s-container-image/pause-3.9.tar
[root@registry ubuntu-k8s-container-image]# cd k8s-container-image
[root@registry k8s-container-image]# ls -la
total 719224
drwxr-xr-x. 2 root root 211 Jan 12 10:52 .
drwxr-xr-x. 3 root root 64 Jan 12 10:58 ..
-rw-r--r--. 1 root root 53628928 Jan 12 10:52 coredns-v1.10.1.tar
-rw-r--r--. 1 root root 295491072 Jan 12 10:51 etcd-3.5.9-0.tar
-rw-r--r--. 1 root root 127181312 Jan 12 10:50 kube-apiserver-v1.28.2.tar
-rw-r--r--. 1 root root 123203584 Jan 12 10:51 kube-controller-v1.28.2.tar
-rw-r--r--. 1 root root 74695680 Jan 12 10:51 kube-proxy-v1.28.2.tar
-rw-r--r--. 1 root root 61517824 Jan 12 10:51 kube-scheduler-v1.28.2.tar
-rw-r--r--. 1 root root 754176 Jan 12 10:52 pause-3.9.tar
# 현재 로컬 레지스트리 서버에 구성되어 있는 goharbor 컨테이너
[root@registry k8s-container-image]# docker image list
REPOSITORY TAG IMAGE ID CREATED SIZE
goharbor/redis-photon v2.3.1 4a0d49a4ece0 2 years ago 191MB
goharbor/harbor-registryctl v2.3.1 91e798004920 2 years ago 132MB
goharbor/registry-photon v2.3.1 972ce19b1882 2 years ago 81.2MB
goharbor/nginx-photon v2.3.1 3b3ede1db494 2 years ago 44.3MB
goharbor/harbor-log v2.3.1 40a54594fe22 2 years ago 194MB
goharbor/harbor-jobservice v2.3.1 d6e174ae0a00 2 years ago 171MB
goharbor/harbor-core v2.3.1 f05acc3947d6 2 years ago 158MB
goharbor/harbor-portal v2.3.1 4a15c5622fda 2 years ago 57.6MB
goharbor/harbor-db v2.3.1 b16a9c81ef03 2 years ago 263MB
# docker load 명령으로 tar 파일을 로컬 도커 엔진에 먼저 로딩
# tar 파일의 이름을 리스트로 받아서 일괄적으로 로딩
[root@registry k8s-container-image]# ls -la *.tar | awk '{ print $9 }' | xargs -n1 -I{} docker load -i {}
6a4a177e62f3: Loading layer [==================================================>] 203.8kB/203.8kB
398c9baff0ce: Loading layer [==================================================>] 53.41MB/53.41MB
Loaded image: registry.k8s.io/coredns/coredns:v1.10.1
e023e0e48e6e: Loading layer [==================================================>] 327.7kB/327.7kB
6fbdf253bbc2: Loading layer [==================================================>] 51.2kB/51.2kB
7bea6b893187: Loading layer [==================================================>] 3.205MB/3.205MB
ff5700ec5418: Loading layer [==================================================>] 10.24kB/10.24kB
d52f02c6501c: Loading layer [==================================================>] 10.24kB/10.24kB
e624a5370eca: Loading layer [==================================================>] 10.24kB/10.24kB
1a73b54f556b: Loading layer [==================================================>] 10.24kB/10.24kB
d2d7ec0f6756: Loading layer [==================================================>] 10.24kB/10.24kB
4cb10dd2545b: Loading layer [==================================================>] 225.3kB/225.3kB
ba9afb2b3e0c: Loading layer [==================================================>] 2.257MB/2.257MB
22bba3da6b0d: Loading layer [==================================================>] 269.7MB/269.7MB
a4563151d59b: Loading layer [==================================================>] 19.67MB/19.67MB
Loaded image: registry.k8s.io/etcd:3.5.9-0
47c446271f8d: Loading layer [==================================================>] 1.607MB/1.607MB
c5af401a13b1: Loading layer [==================================================>] 121.7MB/121.7MB
Loaded image: registry.k8s.io/kube-apiserver:v1.28.2
5a79e3f66baa: Loading layer [==================================================>] 117.7MB/117.7MB
Loaded image: registry.k8s.io/kube-controller-manager:v1.28.2
d2a4b045f3d0: Loading layer [==================================================>] 19.63MB/19.63MB
5e8877c87d9f: Loading layer [==================================================>] 55.05MB/55.05MB
Loaded image: registry.k8s.io/kube-proxy:v1.28.2
86adbf8edefb: Loading layer [==================================================>] 56.01MB/56.01MB
Loaded image: registry.k8s.io/kube-scheduler:v1.28.2
e3e5579ddd43: Loading layer [==================================================>] 746kB/746kB
Loaded image: registry.k8s.io/pause:3.9
# 로딩된 도커 이미지 확인
[root@registry k8s-container-image]# docker image list
REPOSITORY TAG IMAGE ID CREATED SIZE
registry.k8s.io/kube-apiserver v1.28.2 cdcab12b2dd1 4 months ago 126MB
registry.k8s.io/kube-proxy v1.28.2 c120fed2beb8 4 months ago 73.1MB
registry.k8s.io/kube-scheduler v1.28.2 7a5d9d67a13f 4 months ago 60.1MB
registry.k8s.io/kube-controller-manager v1.28.2 55f13c92defb 4 months ago 122MB
registry.k8s.io/etcd 3.5.9-0 73deb9a3f702 8 months ago 294MB
registry.k8s.io/coredns/coredns v1.10.1 ead0a4a53df8 11 months ago 53.6MB
registry.k8s.io/pause 3.9 e6f181688397 15 months ago 744kB
goharbor/redis-photon v2.3.1 4a0d49a4ece0 2 years ago 191MB
goharbor/harbor-registryctl v2.3.1 91e798004920 2 years ago 132MB
goharbor/registry-photon v2.3.1 972ce19b1882 2 years ago 81.2MB
goharbor/nginx-photon v2.3.1 3b3ede1db494 2 years ago 44.3MB
goharbor/harbor-log v2.3.1 40a54594fe22 2 years ago 194MB
goharbor/harbor-jobservice v2.3.1 d6e174ae0a00 2 years ago 171MB
goharbor/harbor-core v2.3.1 f05acc3947d6 2 years ago 158MB
goharbor/harbor-portal v2.3.1 4a15c5622fda 2 years ago 57.6MB
goharbor/harbor-db v2.3.1 b16a9c81ef03 2 years ago 263MB
# 로딩된 도커 이미지의 도커 테그를 로컬 레지스트리 서버에 업로드하기 위해 변경. 여기서 로컬 레지스트리 서버의 주소는 10.118.99.110임
# 아래 명령은 첫 줄과 goharbor 이미지는 제외하고 처리하고 resgistry.k8s.io를 로컬 레지스트리 서버로 변경하여 tagging 명령을 수행한다.
[root@registry k8s-container-image]# docker image list | awk 'NR > 1 && $1 !~"goharbor" {P1=P2=$1":"$2;sub("registry.k8s.io/","",P2); system("docker tag " P1 " 10.118.99.110/library/" P2)}'
[root@registry k8s-container-image]# docker image list
REPOSITORY TAG IMAGE ID CREATED SIZE
10.118.99.110/library/kube-apiserver v1.28.2 cdcab12b2dd1 4 months ago 126MB
registry.k8s.io/kube-apiserver v1.28.2 cdcab12b2dd1 4 months ago 126MB
10.118.99.110/library/kube-proxy v1.28.2 c120fed2beb8 4 months ago 73.1MB
registry.k8s.io/kube-proxy v1.28.2 c120fed2beb8 4 months ago 73.1MB
10.118.99.110/library/kube-controller-manager v1.28.2 55f13c92defb 4 months ago 122MB
registry.k8s.io/kube-controller-manager v1.28.2 55f13c92defb 4 months ago 122MB
10.118.99.110/library/kube-scheduler v1.28.2 7a5d9d67a13f 4 months ago 60.1MB
registry.k8s.io/kube-scheduler v1.28.2 7a5d9d67a13f 4 months ago 60.1MB
10.118.99.110/library/etcd 3.5.9-0 73deb9a3f702 8 months ago 294MB
registry.k8s.io/etcd 3.5.9-0 73deb9a3f702 8 months ago 294MB
10.118.99.110/library/coredns/coredns v1.10.1 ead0a4a53df8 11 months ago 53.6MB
registry.k8s.io/coredns/coredns v1.10.1 ead0a4a53df8 11 months ago 53.6MB
10.118.99.110/library/pause 3.9 e6f181688397 15 months ago 744kB
registry.k8s.io/pause 3.9 e6f181688397 15 months ago 744kB
goharbor/redis-photon v2.3.1 4a0d49a4ece0 2 years ago 191MB
goharbor/harbor-registryctl v2.3.1 91e798004920 2 years ago 132MB
goharbor/registry-photon v2.3.1 972ce19b1882 2 years ago 81.2MB
goharbor/nginx-photon v2.3.1 3b3ede1db494 2 years ago 44.3MB
goharbor/harbor-log v2.3.1 40a54594fe22 2 years ago 194MB
goharbor/harbor-jobservice v2.3.1 d6e174ae0a00 2 years ago 171MB
goharbor/harbor-core v2.3.1 f05acc3947d6 2 years ago 158MB
goharbor/harbor-portal v2.3.1 4a15c5622fda 2 years ago 57.6MB
goharbor/harbor-db v2.3.1 b16a9c81ef03 2 years ago 263MB
# 로컬 도커 레지스트리 서버에 업로드
[root@registry k8s-container-image]# docker image list | awk 'NR > 1 && $1 ~"10.118.99.110" {P1=$1":"$2; system ("docker push " P1)}'
The push refers to repository [10.118.99.110/library/kube-apiserver]
c5af401a13b1: Layer already exists
47c446271f8d: Layer already exists
4cb10dd2545b: Layer already exists
d2d7ec0f6756: Layer already exists
1a73b54f556b: Layer already exists
e624a5370eca: Layer already exists
d52f02c6501c: Layer already exists
ff5700ec5418: Layer already exists
7bea6b893187: Layer already exists
6fbdf253bbc2: Layer already exists
e023e0e48e6e: Layer already exists
v1.28.2: digest: sha256:90dbee983fc570844dc632bfc2699319e2f59dce5000558b73e79d00e1534140 size: 2612
The push refers to repository [10.118.99.110/library/kube-scheduler]
86adbf8edefb: Pushed
47c446271f8d: Mounted from library/kube-apiserver
4cb10dd2545b: Mounted from library/sig-storage/livenessprobe
d2d7ec0f6756: Mounted from library/sig-storage/livenessprobe
1a73b54f556b: Mounted from library/sig-storage/livenessprobe
e624a5370eca: Mounted from library/sig-storage/livenessprobe
d52f02c6501c: Mounted from library/sig-storage/livenessprobe
ff5700ec5418: Mounted from library/sig-storage/livenessprobe
7bea6b893187: Mounted from library/sig-storage/livenessprobe
6fbdf253bbc2: Mounted from library/sig-storage/livenessprobe
e023e0e48e6e: Mounted from library/sig-storage/csi-node-driver-registrar
v1.28.2: digest: sha256:23317c2a2b079fc2a256a253c556034fc83b3d2b0efe7ca60fcec2e22d9f7a4c size: 2612
The push refers to repository [10.118.99.110/library/kube-controller-manager]
5a79e3f66baa: Pushed
47c446271f8d: Mounted from library/kube-scheduler
4cb10dd2545b: Mounted from library/kube-scheduler
d2d7ec0f6756: Mounted from library/kube-scheduler
1a73b54f556b: Mounted from library/kube-scheduler
e624a5370eca: Mounted from library/kube-scheduler
d52f02c6501c: Mounted from library/kube-scheduler
ff5700ec5418: Mounted from library/kube-scheduler
7bea6b893187: Mounted from library/kube-scheduler
6fbdf253bbc2: Mounted from library/kube-scheduler
e023e0e48e6e: Mounted from library/kube-scheduler
v1.28.2: digest: sha256:9fa4f918d91f61d3beb96dd054dff491aa7fea0277bb9192d51a9ba11e4bb8d2 size: 2612
The push refers to repository [10.118.99.110/library/kube-proxy]
5e8877c87d9f: Pushed
d2a4b045f3d0: Pushed
v1.28.2: digest: sha256:3330e491169be46febd3f4e487924195f60c09f284bbda38cab7cbe71a51fded size: 739
The push refers to repository [10.118.99.110/library/etcd]
a4563151d59b: Pushed
22bba3da6b0d: Pushed
ba9afb2b3e0c: Pushed
4cb10dd2545b: Mounted from library/kube-controller-manager
d2d7ec0f6756: Mounted from library/kube-controller-manager
1a73b54f556b: Mounted from library/kube-controller-manager
e624a5370eca: Mounted from library/kube-controller-manager
d52f02c6501c: Mounted from library/kube-controller-manager
ff5700ec5418: Mounted from library/kube-controller-manager
7bea6b893187: Mounted from library/kube-controller-manager
6fbdf253bbc2: Mounted from library/kube-controller-manager
e023e0e48e6e: Mounted from library/kube-controller-manager
3.5.9-0: digest: sha256:7769e9b8ad25dbb831ea0c92794eb188e2ef6d865e154edcc73b64d90d913250 size: 2824
The push refers to repository [10.118.99.110/library/coredns/coredns]
398c9baff0ce: Pushed
6a4a177e62f3: Pushed
v1.10.1: digest: sha256:be7652ce0b43b1339f3d14d9b14af9f588578011092c1f7893bd55432d83a378 size: 739
The push refers to repository [10.118.99.110/library/pause]
e3e5579ddd43: Pushed
3.9: digest: sha256:0fc1f3b764be56f7c881a69cbd553ae25a2b5523c6901fbacb8270307c29d0c4 size: 526
# coredns만 경로가 틀려서 수정해서 다시 업로드
[root@registry k8s-container-image]# docker tag registry.k8s.io/coredns/coredns:v1.10.1 10.118.99.110/library/coredns:v1.10.1
[root@registry k8s-container-image]# docker push 10.118.99.110/library/coredns:v1.10.1
The push refers to repository [10.118.99.110/library/coredns]
398c9baff0ce: Mounted from library/coredns/coredns
6a4a177e62f3: Mounted from library/coredns/coredns
v1.10.1: digest: sha256:be7652ce0b43b1339f3d14d9b14af9f588578011092c1f7893bd55432d83a378 size: 739
쿠버네티스 docker, containerd가 로컬 레지스트리 서버를 참조하도록 변경(오프라인 서버)
모든 쿠버네티스 노드에서 아래 설정을 진행
containerd 설정 파일에 pause 컨테이너 이미지를 참조하도록 이미지 경로가 들어가 있는데 이 부분을 로컬에서 참조하도록 변경해야 한다.
root@u-kuber01:/home/emc/kubernetes# vi /etc/docker/daemon.json
{
"exec-opts": ["native.cgroupdriver=systemd"],
"log-driver": "json-file",
"log-opts": {
"max-size": "100m"
},
"storage-driver": "overlay2",
"insecure-registries" : ["10.118.99.110"]
}
~
root@u-kuber01:/home/emc/kubernetes# systemctl reload docker
root@u-kuber01:/home/emc/kubernetes# systemctl restart docker
root@u-kuber01:/home/emc/kubernetes# cd /etc/containerd/
root@u-kuber01:/etc/containerd# containerd config default > /etc/containerd/config.toml
root@u-kuber01:/etc/containerd# ls -la
total 16
drwxr-xr-x 2 root root 4096 Jan 12 01:12 .
drwxr-xr-x 102 root root 4096 Jan 12 00:30 ..
-rw-r--r-- 1 root root 7041 Jan 12 01:12 config.toml
# config.toml 파일의 61행의 컨테이너 이미지 참조를 수정하고, 149행 아래에 설정을 추가한다.
root@u-kuber01:/etc/containerd# vi config.toml
max_concurrent_downloads = 3
max_container_log_line_size = 16384
netns_mounts_under_state_dir = false
restrict_oom_score_adj = false
sandbox_image = "10.118.99.110/library/pause:3.6" # 자신의 레지스트리 주소로 변경, 버전을 3.9로
selinux_category_range = 1024
stats_collect_period = 10
stream_idle_timeout = "4h0m0s"
[plugins."io.containerd.grpc.v1.cri".registry.configs] # 149행 여기 아래에 추가
[plugins."io.containerd.grpc.v1.cri".registry.configs."10.118.99.110"]
[plugins."io.containerd.grpc.v1.cri".registry.configs."10.118.99.110".tls]
ca_file = ""
cert_file = ""
insecure_skip_verify = true
key_file = ""
[plugins."io.containerd.grpc.v1.cri".registry.headers]
[plugins."io.containerd.grpc.v1.cri".registry.mirrors] # 여기 아래에 추가 2
[plugins."io.containerd.grpc.v1.cri".registry.mirrors."10.118.99.110"]
endpoint = ["http://10.118.99.110"]
root@u-kuber01:/etc/containerd# systemctl restart containerd
쿠버네티스 마스터 노드 설정(오프라인 서버)
kubeadm init 명령을 통해 마스터 노드를 초기화한다.
--image-repository라는 옵션으로 로컬 이미지 리포지토리 주소를 전달할 수 있다.
root@u-kuber01:/home/emc# kubeadm init --pod-network-cidr=10.244.0.0/16 --image-repository=10.118.99.110/library
W0112 03:15:09.588394 2859 version.go:104] could not fetch a Kubernetes version from the internet: unable to get URL "https://dl.k8s.io/release/stable-1.txt": Get "https://dl.k8s.io/release/stable-1.txt": dial tcp: lookup dl.k8s.io on 127.0.0.53:53: no such host
W0112 03:15:09.588505 2859 version.go:105] falling back to the local client version: v1.28.2
[init] Using Kubernetes version: v1.28.2
[preflight] Running pre-flight checks
[preflight] Pulling images required for setting up a Kubernetes cluster
[preflight] This might take a minute or two, depending on the speed of your internet connection
[preflight] You can also perform this action in beforehand using 'kubeadm config images pull'
W0112 03:15:10.056195 2859 checks.go:835] detected that the sandbox image "10.118.99.110/library/pause:3.6" of the container runtime is inconsistent with that used by kubeadm. It is recommended that using "10.118.99.110/library/pause:3.9" as the CRI sandbox image.
[certs] Using certificateDir folder "/etc/kubernetes/pki"
[certs] Generating "ca" certificate and key
[certs] Generating "apiserver" certificate and key
[certs] apiserver serving cert is signed for DNS names [kubernetes kubernetes.default kubernetes.default.svc kubernetes.default.svc.cluster.local u-kuber01] and IPs [10.96.0.1 10.118.99.114]
[certs] Generating "apiserver-kubelet-client" certificate and key
[certs] Generating "front-proxy-ca" certificate and key
[certs] Generating "front-proxy-client" certificate and key
[certs] Generating "etcd/ca" certificate and key
[certs] Generating "etcd/server" certificate and key
[certs] etcd/server serving cert is signed for DNS names [localhost u-kuber01] and IPs [10.118.99.114 127.0.0.1 ::1]
[certs] Generating "etcd/peer" certificate and key
[certs] etcd/peer serving cert is signed for DNS names [localhost u-kuber01] and IPs [10.118.99.114 127.0.0.1 ::1]
[certs] Generating "etcd/healthcheck-client" certificate and key
[certs] Generating "apiserver-etcd-client" certificate and key
[certs] Generating "sa" key and public key
[kubeconfig] Using kubeconfig folder "/etc/kubernetes"
[kubeconfig] Writing "admin.conf" kubeconfig file
[kubeconfig] Writing "kubelet.conf" kubeconfig file
[kubeconfig] Writing "controller-manager.conf" kubeconfig file
[kubeconfig] Writing "scheduler.conf" kubeconfig file
[etcd] Creating static Pod manifest for local etcd in "/etc/kubernetes/manifests"
[control-plane] Using manifest folder "/etc/kubernetes/manifests"
[control-plane] Creating static Pod manifest for "kube-apiserver"
[control-plane] Creating static Pod manifest for "kube-controller-manager"
[control-plane] Creating static Pod manifest for "kube-scheduler"
[kubelet-start] Writing kubelet environment file with flags to file "/var/lib/kubelet/kubeadm-flags.env"
[kubelet-start] Writing kubelet configuration to file "/var/lib/kubelet/config.yaml"
[kubelet-start] Starting the kubelet
[wait-control-plane] Waiting for the kubelet to boot up the control plane as static Pods from directory "/etc/kubernetes/manifests". This can take up to 4m0s
[apiclient] All control plane components are healthy after 15.020271 seconds
[upload-config] Storing the configuration used in ConfigMap "kubeadm-config" in the "kube-system" Namespace
[kubelet] Creating a ConfigMap "kubelet-config" in namespace kube-system with the configuration for the kubelets in the cluster
[upload-certs] Skipping phase. Please see --upload-certs
[mark-control-plane] Marking the node u-kuber01 as control-plane by adding the labels: [node-role.kubernetes.io/control-plane node.kubernetes.io/exclude-from-external-load-balancers]
[mark-control-plane] Marking the node u-kuber01 as control-plane by adding the taints [node-role.kubernetes.io/control-plane:NoSchedule]
[bootstrap-token] Using token: 128rsm.zul0sqz7lcfxo7xc
[bootstrap-token] Configuring bootstrap tokens, cluster-info ConfigMap, RBAC Roles
[bootstrap-token] Configured RBAC rules to allow Node Bootstrap tokens to get nodes
[bootstrap-token] Configured RBAC rules to allow Node Bootstrap tokens to post CSRs in order for nodes to get long term certificate credentials
[bootstrap-token] Configured RBAC rules to allow the csrapprover controller automatically approve CSRs from a Node Bootstrap Token
[bootstrap-token] Configured RBAC rules to allow certificate rotation for all node client certificates in the cluster
[bootstrap-token] Creating the "cluster-info" ConfigMap in the "kube-public" namespace
[kubelet-finalize] Updating "/etc/kubernetes/kubelet.conf" to point to a rotatable kubelet client certificate and key
[addons] Applied essential addon: CoreDNS
[addons] Applied essential addon: kube-proxy
Your Kubernetes control-plane has initialized successfully!
To start using your cluster, you need to run the following as a regular user:
mkdir -p $HOME/.kube
sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
sudo chown $(id -u):$(id -g) $HOME/.kube/config
Alternatively, if you are the root user, you can run:
export KUBECONFIG=/etc/kubernetes/admin.conf
You should now deploy a pod network to the cluster.
Run "kubectl apply -f [podnetwork].yaml" with one of the options listed at:
https://kubernetes.io/docs/concepts/cluster-administration/addons/
Then you can join any number of worker nodes by running the following on each as root:
kubeadm join 10.118.99.114:6443 --token 128rsm.zul0sqz7lcfxo7xc \
--discovery-token-ca-cert-hash sha256:1bd12a6a445f79c81ce73792a9caac1ba6b5da6bc0d8f684c1e95638b7a08ae1
# 아래 명령을 마스터에서 수행
mkdir -p $HOME/.kube
sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
sudo chown $(id -u):$(id -g) $HOME/.kube/config
export KUBECONFIG=/etc/kubernetes/admin.conf
쿠버네티스 워커 노드 작업(오프라인 서버)
마스터노드에 워커노드를 추가하기 위해 아래와 같이 마스터 노드 결과로 나온 join 구문을 복사해서 수행한다.
# join 명령을 노드별로 수행
root@u-kuber02:/home/emc# kubeadm join 10.118.99.114:6443 --token 128rsm.zul0sqz7lcfxo7xc \
> --discovery-token-ca-cert-hash sha256:1bd12a6a445f79c81ce73792a9caac1ba6b5da6bc0d8f684c1e95638b7a08ae1
[preflight] Running pre-flight checks
[preflight] Reading configuration from the cluster...
[preflight] FYI: You can look at this config file with 'kubectl -n kube-system get cm kubeadm-config -o yaml'
[kubelet-start] Writing kubelet configuration to file "/var/lib/kubelet/config.yaml"
[kubelet-start] Writing kubelet environment file with flags to file "/var/lib/kubelet/kubeadm-flags.env"
[kubelet-start] Starting the kubelet
[kubelet-start] Waiting for the kubelet to perform the TLS Bootstrap...
This node has joined the cluster:
* Certificate signing request was sent to apiserver and a response was received.
* The Kubelet was informed of the new secure connection details.
Run 'kubectl get nodes' on the control-plane to see this node join the cluster.
root@u-kuber03:/home/emc# kubeadm join 10.118.99.114:6443 --token 128rsm.zul0sqz7lcfxo7xc \
> --discovery-token-ca-cert-hash sha256:1bd12a6a445f79c81ce73792a9caac1ba6b5da6bc0d8f684c1e95638b7a08ae1
[preflight] Running pre-flight checks
[preflight] Reading configuration from the cluster...
[preflight] FYI: You can look at this config file with 'kubectl -n kube-system get cm kubeadm-config -o yaml'
[kubelet-start] Writing kubelet configuration to file "/var/lib/kubelet/config.yaml"
[kubelet-start] Writing kubelet environment file with flags to file "/var/lib/kubelet/kubeadm-flags.env"
[kubelet-start] Starting the kubelet
[kubelet-start] Waiting for the kubelet to perform the TLS Bootstrap...
This node has joined the cluster:
* Certificate signing request was sent to apiserver and a response was received.
* The Kubelet was informed of the new secure connection details.
Run 'kubectl get nodes' on the control-plane to see this node join the cluster.
root@u-kuber01:/home/emc# kubectl get nodes
NAME STATUS ROLES AGE VERSION
u-kuber01 NotReady control-plane 110s v1.28.2
u-kuber02 NotReady <none> 24s v1.28.2
u-kuber03 NotReady <none> 6s v1.28.2
root@u-kuber01:/home/emc# kubectl get pods -A -o wide
NAMESPACE NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
kube-system coredns-994d56f45-6n5z9 0/1 Pending 0 8m14s <none> <none> <none> <none>
kube-system coredns-994d56f45-jkc7n 0/1 Pending 0 8m14s <none> <none> <none> <none>
kube-system etcd-u-kuber01 1/1 Running 0 8m27s 10.118.99.114 u-kuber01 <none> <none>
kube-system kube-apiserver-u-kuber01 1/1 Running 0 8m27s 10.118.99.114 u-kuber01 <none> <none>
kube-system kube-controller-manager-u-kuber01 1/1 Running 0 8m30s 10.118.99.114 u-kuber01 <none> <none>
kube-system kube-proxy-4jmr4 1/1 Running 0 6m47s 10.118.99.116 u-kuber03 <none> <none>
kube-system kube-proxy-bgzmn 1/1 Running 0 8m14s 10.118.99.114 u-kuber01 <none> <none>
kube-system kube-proxy-q9rjx 1/1 Running 0 7m5s 10.118.99.115 u-kuber02 <none> <none>
kube-system kube-scheduler-u-kuber01 1/1 Running 0 8m27s 10.118.99.114 u-kuber01 <none> <none>
쿠버네티스 네트웍 플러그인 설치
쿠버네티스 네티웍 플러그인 설치를 위해서 온라인 서버에서 관련 yaml 파일을 다운로드 받고 Yaml 파일에서 참조하고 있는 이미지 정보를 확인하여 해당 컨테이너 이미지를 다운로드 받는다.
root@ubuntu2204:/apt-mirror# mkdir calico
root@ubuntu2204:/apt-mirror# mv calico.yaml calico
root@ubuntu2204:/apt-mirror# cd calico/
root@ubuntu2204:/apt-mirror/calico# wget https://raw.githubusercontent.com/projectcalico/calico/v3.26.1/manifests/calico.yaml
--2024-01-12 04:37:41-- https://raw.githubusercontent.com/projectcalico/calico/v3.26.1/manifests/calico.yaml
Resolving raw.githubusercontent.com (raw.githubusercontent.com)... 185.199.108.133, 185.199.111.133, 185.199.110.133, ...
Connecting to raw.githubusercontent.com (raw.githubusercontent.com)|185.199.108.133|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 244734 (239K) [text/plain]
Saving to: ‘calico.yaml’
calico.yaml 100%[=====================================================================================>] 239.00K --.-KB/s in 0.07s
2024-01-12 04:37:41 (3.17 MB/s) - ‘calico.yaml’ saved [244734/244734]
root@ubuntu2204:/apt-mirror/calico# ls -la
total 248
drwxr-xr-x 2 root root 4096 Jan 12 04:37 .
drwxr-xr-x 8 root root 4096 Jan 12 04:36 ..
-rw-r--r-- 1 root root 244734 Jan 12 04:37 calico.yaml
위에서 다운로드 받은 yaml파일을 열어보면 image: 로 되어 있는 부분이 있는데 해당 이미지 이름을 추출하여 별도로 다운로드 받는다.
총 3개의 이미지를 참조한다.
docker.io/calico/kube-controllers:v3.26.1
docker.io/calico/cni:v3.26.1
docker.io/calico/node:v3.26.1
root@ubuntu2204:/apt-mirror/calico# docker pull docker.io/calico/kube-controllers:v3.26.1
v3.26.1: Pulling from calico/kube-controllers
fe2ccd270eb7: Pull complete
5fcafcc9e60e: Pull complete
8152bb18c4b9: Pull complete
c383daff5dcb: Pull complete
6e777fc9ae27: Pull complete
566290fdc43c: Pull complete
807b54c42901: Pull complete
26bd6f1dcca4: Pull complete
b9b2f2cb965a: Pull complete
96510a036d14: Pull complete
94d15eeac54c: Pull complete
70e5fa918535: Pull complete
58bd207ae111: Pull complete
5cd0c04b7525: Pull complete
Digest: sha256:01ce29ea8f2b34b6cef904f526baed98db4c0581102f194e36f2cd97943f77aa
Status: Downloaded newer image for calico/kube-controllers:v3.26.1
docker.io/calico/kube-controllers:v3.26.1
root@ubuntu2204:/apt-mirror/calico# docker pull docker.io/calico/cni:v3.26.1
v3.26.1: Pulling from calico/cni
ead1d93126e2: Pull complete
263e6c47f9f4: Pull complete
3d817138ad22: Pull complete
81ce5fcda971: Pull complete
31f2a5683ad7: Pull complete
a13d92692813: Pull complete
85790a7a5aee: Pull complete
a0ff5af14493: Pull complete
cec6d7c8ed45: Pull complete
93a53306e2e3: Pull complete
32305daf6772: Pull complete
4f4fb700ef54: Pull complete
Digest: sha256:3be3c67ddba17004c292eafec98cc49368ac273b40b27c8a6621be4471d348d6
Status: Downloaded newer image for calico/cni:v3.26.1
docker.io/calico/cni:v3.26.1
root@ubuntu2204:/apt-mirror/calico# docker pull docker.io/calico/node:v3.26.1
v3.26.1: Pulling from calico/node
c998069b4a7c: Pull complete
6cf92fa5251d: Pull complete
Digest: sha256:8e34517775f319917a0be516ed3a373dbfca650d1ee8e72158087c24356f47fb
Status: Downloaded newer image for calico/node:v3.26.1
docker.io/calico/node:v3.26.1
root@ubuntu2204:/apt-mirror/calico# docker image list
REPOSITORY TAG IMAGE ID CREATED SIZE
registry.k8s.io/kube-apiserver v1.28.2 cdcab12b2dd1 4 months ago 126MB
registry.k8s.io/kube-controller-manager v1.28.2 55f13c92defb 4 months ago 122MB
registry.k8s.io/kube-scheduler v1.28.2 7a5d9d67a13f 4 months ago 60.1MB
registry.k8s.io/kube-proxy v1.28.2 c120fed2beb8 4 months ago 73.1MB
calico/kube-controllers v3.26.1 1919f2787fa7 6 months ago 75.1MB
calico/cni v3.26.1 9dee260ef7f5 6 months ago 210MB
calico/node v3.26.1 8065b798a4d6 6 months ago 246MB
registry.k8s.io/etcd 3.5.9-0 73deb9a3f702 8 months ago 294MB
registry.k8s.io/coredns/coredns v1.10.1 ead0a4a53df8 11 months ago 53.6MB
registry.k8s.io/pause 3.9 e6f181688397 15 months ago 744kB
root@ubuntu2204:/apt-mirror/calico# docker save calico/kube-controllers:v3.26.1 > calico-kube-controllers-v3.26.1.tar
root@ubuntu2204:/apt-mirror/calico# docker save calico/cni:v3.26.1 > calico-cni-v3.26.1.tar
root@ubuntu2204:/apt-mirror/calico# docker save calico/node:v3.26.1 > calico-node-v3.26.1.tar
root@ubuntu2204:/apt-mirror/calico# ls -la
total 521080
drwxr-xr-x 2 root root 4096 Jan 12 04:45 .
drwxr-xr-x 8 root root 4096 Jan 12 04:36 ..
-rw-r--r-- 1 root root 210457600 Jan 12 04:45 calico-cni-v3.26.1.tar
-rw-r--r-- 1 root root 75176448 Jan 12 04:44 calico-kube-controllers-v3.26.1.tar
-rw-r--r-- 1 root root 247693312 Jan 12 04:45 calico-node-v3.26.1.tar
-rw-r--r-- 1 root root 244734 Jan 12 04:37 calico.yaml네트웍 플러그인 파일 전송 및 리포지토리 임포트(온라인)
# yaml을 마스터 노드에 복사하고 이미지는 묶어서 로컬 레지스트리 서버에 보내고 임포트한다.
root@ubuntu2204:/apt-mirror/calico# scp calico.yaml emc@10.118.99.114:/home/emc
emc@10.118.99.114's password:
calico.yaml 100% 239KB 300.7KB/s 00:00
root@ubuntu2204:/apt-mirror/calico# cd ..
root@ubuntu2204:/apt-mirror# tar cvf calico.tar calico/
calico/
calico/calico-node-v3.26.1.tar
calico/calico-kube-controllers-v3.26.1.tar
calico/calico-cni-v3.26.1.tar
calico/calico.yaml
root@ubuntu2204:/apt-mirror# scp calico.tar 10.118.99.110:/root
root@10.118.99.110's password:
calico.tar 100% 509MB 8.3MB/s 01:01
root@ubuntu2204:/apt-mirror#
# 레지스트리 서버에 업로드
[root@registry ~]# tar xvf calico.tar
calico/
calico/calico-node-v3.26.1.tar
calico/calico-kube-controllers-v3.26.1.tar
calico/calico-cni-v3.26.1.tar
calico/calico.yaml
[root@registry ~]# cd calico
[root@registry calico]# ls
calico-cni-v3.26.1.tar calico-kube-controllers-v3.26.1.tar calico-node-v3.26.1.tar calico.yaml
[root@registry calico]# docker load -i calico-cni-v3.26.1.tar
e2fc87ba0086: Loading layer [==================================================>] 88.58kB/88.58kB
cfe5b7d5aece: Loading layer [==================================================>] 13.82kB/13.82kB
ce81308843c9: Loading layer [==================================================>] 1.064MB/1.064MB
cd1ec5cd0b0b: Loading layer [==================================================>] 152.1kB/152.1kB
e304784934ce: Loading layer [==================================================>] 2.092MB/2.092MB
29bac9e3886c: Loading layer [==================================================>] 31.74kB/31.74kB
2f763bfb2190: Loading layer [==================================================>] 56.83kB/56.83kB
fdd10df223c3: Loading layer [==================================================>] 95.23kB/95.23kB
80b8b7216b75: Loading layer [==================================================>] 2.56kB/2.56kB
4bc051404070: Loading layer [==================================================>] 4.608kB/4.608kB
e6e351e5d632: Loading layer [==================================================>] 206.8MB/206.8MB
5f70bf18a086: Loading layer [==================================================>] 1.024kB/1.024kB
Loaded image: calico/cni:v3.26.1
[root@registry calico]# docker load -i calico-kube-controllers-v3.26.1.tar
af3f237ad601: Loading layer [==================================================>] 13.82kB/13.82kB
67399101bcb0: Loading layer [==================================================>] 2.56kB/2.56kB
94e265c94a6e: Loading layer [==================================================>] 2.048kB/2.048kB
264c62d097b0: Loading layer [==================================================>] 2.048kB/2.048kB
4c2f5abdad5e: Loading layer [==================================================>] 152.1kB/152.1kB
c45d1f3c1ea1: Loading layer [==================================================>] 2.092MB/2.092MB
fa43ba7fe668: Loading layer [==================================================>] 1.064MB/1.064MB
0c643e9161ee: Loading layer [==================================================>] 31.74kB/31.74kB
e9eb2cc3ba24: Loading layer [==================================================>] 56.83kB/56.83kB
cc35e000725a: Loading layer [==================================================>] 95.23kB/95.23kB
e20e1a0e381d: Loading layer [==================================================>] 2.56kB/2.56kB
b6d2049961ea: Loading layer [==================================================>] 4.608kB/4.608kB
bd3c9aa3d2c7: Loading layer [==================================================>] 68.72MB/68.72MB
4fa10cc85829: Loading layer [==================================================>] 2.881MB/2.881MB
Loaded image: calico/kube-controllers:v3.26.1
[root@registry calico]# docker load -i calico-node-v3.26.1.tar
5ed6bb5a3642: Loading layer [==================================================>] 247.7MB/247.7MB
4ae93345f1e4: Loading layer [==================================================>] 13.82kB/13.82kB
Loaded image: calico/node:v3.26.1
[root@registry calico]# docker image list
REPOSITORY TAG IMAGE ID CREATED SIZE
calico/kube-controllers v3.26.1 1919f2787fa7 6 months ago 75.1MB
calico/cni v3.26.1 9dee260ef7f5 6 months ago 210MB
calico/node v3.26.1 8065b798a4d6 6 months ago 246MB
goharbor/redis-photon v2.3.1 4a0d49a4ece0 2 years ago 191MB
goharbor/harbor-registryctl v2.3.1 91e798004920 2 years ago 132MB
goharbor/registry-photon v2.3.1 972ce19b1882 2 years ago 81.2MB
goharbor/nginx-photon v2.3.1 3b3ede1db494 2 years ago 44.3MB
goharbor/harbor-log v2.3.1 40a54594fe22 2 years ago 194MB
goharbor/harbor-jobservice v2.3.1 d6e174ae0a00 2 years ago 171MB
goharbor/harbor-core v2.3.1 f05acc3947d6 2 years ago 158MB
goharbor/harbor-portal v2.3.1 4a15c5622fda 2 years ago 57.6MB
goharbor/harbor-db v2.3.1 b16a9c81ef03 2 years ago 263MB
[root@registry calico]# docker image list | awk 'NR > 1 && $1 !~"goharbor" {P1=P2=$1":"$2;system("docker tag " P1 " 10.118.99.110/library/" P2)}'
[root@registry calico]# docker image list
REPOSITORY TAG IMAGE ID CREATED SIZE
10.118.99.110/library/calico/kube-controllers v3.26.1 1919f2787fa7 6 months ago 75.1MB
calico/kube-controllers v3.26.1 1919f2787fa7 6 months ago 75.1MB
calico/cni v3.26.1 9dee260ef7f5 6 months ago 210MB
10.118.99.110/library/calico/cni v3.26.1 9dee260ef7f5 6 months ago 210MB
10.118.99.110/library/calico/node v3.26.1 8065b798a4d6 6 months ago 246MB
calico/node v3.26.1 8065b798a4d6 6 months ago 246MB
goharbor/redis-photon v2.3.1 4a0d49a4ece0 2 years ago 191MB
goharbor/harbor-registryctl v2.3.1 91e798004920 2 years ago 132MB
goharbor/registry-photon v2.3.1 972ce19b1882 2 years ago 81.2MB
goharbor/nginx-photon v2.3.1 3b3ede1db494 2 years ago 44.3MB
goharbor/harbor-log v2.3.1 40a54594fe22 2 years ago 194MB
goharbor/harbor-jobservice v2.3.1 d6e174ae0a00 2 years ago 171MB
goharbor/harbor-core v2.3.1 f05acc3947d6 2 years ago 158MB
goharbor/harbor-portal v2.3.1 4a15c5622fda 2 years ago 57.6MB
goharbor/harbor-db v2.3.1 b16a9c81ef03 2 years ago 263MB
[root@registry calico]# docker image list | awk 'NR > 1 && $1 ~"10.118.99.110" {P1=$1":"$2; system ("docker push " P1)}'
The push refers to repository [10.118.99.110/library/calico/kube-controllers]
4fa10cc85829: Layer already exists
bd3c9aa3d2c7: Layer already exists
b6d2049961ea: Layer already exists
e20e1a0e381d: Layer already exists
cc35e000725a: Layer already exists
e9eb2cc3ba24: Layer already exists
0c643e9161ee: Layer already exists
fa43ba7fe668: Layer already exists
c45d1f3c1ea1: Layer already exists
4c2f5abdad5e: Layer already exists
264c62d097b0: Layer already exists
94e265c94a6e: Layer already exists
67399101bcb0: Layer already exists
af3f237ad601: Layer already exists
v3.26.1: digest: sha256:2c5526ad8cd69740448207b90f4077fd68a5d2e922014e32141b38a529295c55 size: 3240
The push refers to repository [10.118.99.110/library/calico/cni]
5f70bf18a086: Layer already exists
e6e351e5d632: Layer already exists
4bc051404070: Layer already exists
80b8b7216b75: Layer already exists
fdd10df223c3: Layer already exists
2f763bfb2190: Layer already exists
29bac9e3886c: Layer already exists
e304784934ce: Layer already exists
cd1ec5cd0b0b: Layer already exists
ce81308843c9: Layer already exists
cfe5b7d5aece: Layer already exists
e2fc87ba0086: Layer already exists
v3.26.1: digest: sha256:86779fab56f3c0c51abcae6d5c5d712f54ed86b50eebf83e54b8c80fdcb4a76e size: 2823
The push refers to repository [10.118.99.110/library/calico/node]
4ae93345f1e4: Layer already exists
5ed6bb5a3642: Layer already exists
v3.26.1: digest: sha256:9459d1b2831955120fdf0037e6816b21e5d88dd11110d6d89398e5ef53cdf54c size: 737
네트웍 플러그인 설치(오프라인 마스터 노드)
마스터 노드에 복사된 yaml 파일을 열고 해당 파일에서 참조하고 있는 이미지를 로컬 레지스트리 서버로 변경한다.
# 파일을 열어서 이미지가 참조하고 있는 docker.io 주소를 로컬 레지스트리 주소로 변경
root@u-kuber01:/home/emc# vi calico.yaml
matchLabels:
k8s-app: calico-kube-controllers
strategy:
type: Recreate
template:
metadata:
name: calico-kube-controllers
:1,$s/docker.io/10.118.99.110\/library/g
# 설치 진행
root@u-kuber01:/home/emc# kubectl create -f calico.yaml
poddisruptionbudget.policy/calico-kube-controllers created
serviceaccount/calico-kube-controllers created
serviceaccount/calico-node created
serviceaccount/calico-cni-plugin created
configmap/calico-config created
customresourcedefinition.apiextensions.k8s.io/bgpconfigurations.crd.projectcalico.org created
customresourcedefinition.apiextensions.k8s.io/bgpfilters.crd.projectcalico.org created
customresourcedefinition.apiextensions.k8s.io/bgppeers.crd.projectcalico.org created
customresourcedefinition.apiextensions.k8s.io/blockaffinities.crd.projectcalico.org created
customresourcedefinition.apiextensions.k8s.io/caliconodestatuses.crd.projectcalico.org created
customresourcedefinition.apiextensions.k8s.io/clusterinformations.crd.projectcalico.org created
customresourcedefinition.apiextensions.k8s.io/felixconfigurations.crd.projectcalico.org created
customresourcedefinition.apiextensions.k8s.io/globalnetworkpolicies.crd.projectcalico.org created
customresourcedefinition.apiextensions.k8s.io/globalnetworksets.crd.projectcalico.org created
customresourcedefinition.apiextensions.k8s.io/hostendpoints.crd.projectcalico.org created
customresourcedefinition.apiextensions.k8s.io/ipamblocks.crd.projectcalico.org created
customresourcedefinition.apiextensions.k8s.io/ipamconfigs.crd.projectcalico.org created
customresourcedefinition.apiextensions.k8s.io/ipamhandles.crd.projectcalico.org created
customresourcedefinition.apiextensions.k8s.io/ippools.crd.projectcalico.org created
customresourcedefinition.apiextensions.k8s.io/ipreservations.crd.projectcalico.org created
customresourcedefinition.apiextensions.k8s.io/kubecontrollersconfigurations.crd.projectcalico.org created
customresourcedefinition.apiextensions.k8s.io/networkpolicies.crd.projectcalico.org created
customresourcedefinition.apiextensions.k8s.io/networksets.crd.projectcalico.org created
clusterrole.rbac.authorization.k8s.io/calico-kube-controllers created
clusterrole.rbac.authorization.k8s.io/calico-node created
clusterrole.rbac.authorization.k8s.io/calico-cni-plugin created
clusterrolebinding.rbac.authorization.k8s.io/calico-kube-controllers created
clusterrolebinding.rbac.authorization.k8s.io/calico-node created
clusterrolebinding.rbac.authorization.k8s.io/calico-cni-plugin created
daemonset.apps/calico-node created
deployment.apps/calico-kube-controllers created
root@u-kuber01:/home/emc#
# 설치 후 POD 정보를 조회하면 설치 과정을 볼 수 있다.
root@u-kuber01:/home/emc# kubectl get pods -A
NAMESPACE NAME READY STATUS RESTARTS AGE
kube-system calico-kube-controllers-6c66c7f6bd-6plc4 0/1 Pending 0 15s
kube-system calico-node-d8klp 0/1 Init:0/3 0 15s
kube-system calico-node-tp8zz 0/1 Init:0/3 0 15s
kube-system calico-node-xvl8h 0/1 Init:0/3 0 15s
kube-system coredns-994d56f45-6n5z9 0/1 Pending 0 33m
kube-system coredns-994d56f45-jkc7n 0/1 Pending 0 33m
kube-system etcd-u-kuber01 1/1 Running 0 33m
kube-system kube-apiserver-u-kuber01 1/1 Running 0 33m
kube-system kube-controller-manager-u-kuber01 1/1 Running 0 33m
kube-system kube-proxy-4jmr4 1/1 Running 0 32m
kube-system kube-proxy-bgzmn 1/1 Running 0 33m
kube-system kube-proxy-q9rjx 1/1 Running 0 32m
kube-system kube-scheduler-u-kuber01 1/1 Running 0 33m
root@u-kuber01:/home/emc# kubectl get pods -A
NAMESPACE NAME READY STATUS RESTARTS AGE
kube-system calico-kube-controllers-6c66c7f6bd-6plc4 0/1 ContainerCreating 0 29s
kube-system calico-node-d8klp 0/1 Init:2/3 0 29s
kube-system calico-node-tp8zz 0/1 Init:2/3 0 29s
kube-system calico-node-xvl8h 0/1 Init:2/3 0 29s
kube-system coredns-994d56f45-6n5z9 0/1 ContainerCreating 0 33m
kube-system coredns-994d56f45-jkc7n 0/1 ContainerCreating 0 33m
kube-system etcd-u-kuber01 1/1 Running 0 34m
kube-system kube-apiserver-u-kuber01 1/1 Running 0 34m
kube-system kube-controller-manager-u-kuber01 1/1 Running 0 34m
kube-system kube-proxy-4jmr4 1/1 Running 0 32m
kube-system kube-proxy-bgzmn 1/1 Running 0 33m
kube-system kube-proxy-q9rjx 1/1 Running 0 32m
kube-system kube-scheduler-u-kuber01 1/1 Running 0 34m
# 최종적으로 구성이 완료되면 모든 POD들이 Running 상태로 변경된다.
root@u-kuber01:/home/emc# kubectl get pods -A
NAMESPACE NAME READY STATUS RESTARTS AGE
kube-system calico-kube-controllers-6c66c7f6bd-6plc4 1/1 Running 0 62s
kube-system calico-node-d8klp 1/1 Running 0 62s
kube-system calico-node-tp8zz 1/1 Running 0 62s
kube-system calico-node-xvl8h 1/1 Running 0 62s
kube-system coredns-994d56f45-6n5z9 1/1 Running 0 34m
kube-system coredns-994d56f45-jkc7n 1/1 Running 0 34m
kube-system etcd-u-kuber01 1/1 Running 0 34m
kube-system kube-apiserver-u-kuber01 1/1 Running 0 34m
kube-system kube-controller-manager-u-kuber01 1/1 Running 0 34m
kube-system kube-proxy-4jmr4 1/1 Running 0 32m
kube-system kube-proxy-bgzmn 1/1 Running 0 34m
kube-system kube-proxy-q9rjx 1/1 Running 0 33m
kube-system kube-scheduler-u-kuber01 1/1 Running 0 34m
'kubernetes' 카테고리의 다른 글
| Visual Studio Code를 활용하여 Kubernetes Yaml 파일을 오류 없이 작성해 보자 (0) | 2023.01.06 |
|---|---|
| 외부 시스템에서 kubernetes 클러스터를 관리할 수 있도록 설정하자 (0) | 2023.01.05 |
| Kubernetes NFS CSI 드라이버 설치 (0) | 2023.01.02 |
'kubernetes' Related Articles
more
Comments