CI/CD是近年來正夯的自動化軟體開發與部署方法，透過可程式腳本化的流程，自動的持續整合code、自動的持續交付版本、自動的將結果持續部署至測試或正式區，用以加速軟體開發的週期。
過往並不是沒有土炮的方法可以去實現這個流程，但終究有著開發環境、測試環境與正式部署環境的可能不同，導致了最後結果的可能偏差，直至Docker的問世後，拼齊了這最後一塊拼圖、成功召喚神龍！讓CI/CD這個想法成為普遍現實，從此，開發人員再也不用煩惱系統環境的建置，運維人員再也不用擔心新版本程式碼因套件依賴、開發環境的不同而導致生產環境的運作問題。
由於每間公司既有的軟體開發流程與包袱不同，Gitlab為CI/CD提供了Shell、SSH、Parallels、Docker、Kubernetes、Custom…等數種不同的自動化方法（在gitlab稱為executor），本文我們將透過自建Gitlab運行簡易的nodejs軟體開發專案、通過docker環境建構依賴套件、docker環境下的程式碼測試、將CI的成果打包至docker映像檔、最後將映像檔部署至生產環境，完成一個CI/CD的pipeline。
- 本次實驗需要2台虛擬機器
- 作業系統皆為RockyLinux 8.8
- 閱讀本文您需要有以下技術的知識儲備：docker、docker registry、Dockerfile、docker-compose、git、nodejs、nginx、DNS、Linux shell
A. 虛擬機器環境配置
對象：test-docker、test-deploy
關閉selinux
[root@host ~]# sed -i 's/SELINUX=enforcing/SELINUX=disabled/g' /etc/selinux/config [root@host ~]# setenforce 0
安裝通用服務
[root@host ~]# dnf install -y nano wget git
安裝docker服務
[root@host ~]# wget https://download.docker.com/linux/centos/docker-ce.repo [root@host ~]# mv docker-ce.repo /etc/yum.repos.d/ [root@host ~]# dnf install -y docker-ce-20.10.24 [root@host ~]# systemctl enable docker && systemctl start docker
安裝docker-compose
[root@host ~]# curl -L "https://github.com/docker/compose/releases/download/1.29.2/docker-compose-$(uname -s)-$(uname -m)" -o /usr/local/bin/docker-compose [root@host ~]# chmod +x /usr/local/bin/docker-compose [root@host ~]# docker-compose --version docker-compose version 1.29.2, build 5becea4c
B. test-docker機器上安裝Gitlab
對象：test-docker
透過docker-compose安裝gitlab
[root@test-docker ~]# mkdir -p /opt/docker_home/gitlab [root@test-docker ~]# cd /opt/docker_home/gitlab [root@test-docker gitlab]# nano docker-compose.yml
<docker-compose.yml>
version: '3.6'
services:
web:
image: 'gitlab/gitlab-ee:15.8.2-ee.0'
restart: always
hostname: 'test-gitlab.tomy168.com'
environment:
GITLAB_OMNIBUS_CONFIG: |
external_url 'http://test-gitlab.tomy168.com'
gitlab_rails['gitlab_shell_ssh_port'] = 2222
# Add any other gitlab.rb configuration here, each on its own line
ports:
- '8080:80'
- '2222:22'
volumes:
- ./config:/etc/gitlab
- ./logs:/var/log/gitlab
- ./data:/var/opt/gitlab
shm_size: '256m'確認狀態為healthy後代表容器已經啟動完畢，約莫等待1~2分鐘即可
[root@test-docker gitlab]# docker-compose up -d [root@test-docker gitlab]# docker-compose ps Name Command State Ports ----------------------------------------------------------------------------------------------------------------------------- gitlab /assets/wrapper Up (healthy) 0.0.0.0:2222->22/tcp,:::2222->22/tcp, 443/tcp, 0.0.0.0:8080->80/tcp,:::8080->80/tcp
此處我們預設你已配置nginx upstream反向代理至gitlab容器的TCP 8080，並已於DNS設定好了a record的指向。
查看root用戶初始密碼
[root@test-docker gitlab]# docker exec -it gitlab /bin/grep 'Password:' /etc/gitlab/initial_root_password Password: 0zf6mL8UIEaTsjCmJwgQ4PgghNIQeRF/8Aojg9nGvHA=
輸入帳號root與初始密碼登入gitlab
依照指示Deactive 不讓任何人可以隨意註冊我們私有的gitlab
取消勾選Sign-up enabled選項並Save changes
root初始密碼檔案會在系統設定異動後的24小時內被移除，建議先更改為自己要使用的root密碼
C. test-docker機器上安裝Gitlab Runner
對象：test-docker
用途：分擔gitlab主站壓力，僅作為CI/CD相關任務執行使用
透過docker安裝gitlab runner
[root@test-docker ~]# docker run -d --name gitlab-runner --restart always \ -v /srv/gitlab-runner/config:/etc/gitlab-runner \ -v /var/run/docker.sock:/var/run/docker.sock \ gitlab/gitlab-runner:v15.8.2
查閱gitlab runner的註冊token
註冊gitlab runner至gitlab主站
[root@test-docker ~]# docker run --rm -it -v /srv/gitlab-runner/config:/etc/gitlab-runner gitlab/gitlab-runner:v15.8.2 register \
--docker-volumes /var/run/docker.sock:/var/run/docker.sock
Runtime platform arch=amd64 os=linux pid=7 revision=4d1ca121 version=15.8.2
Running in system-mode.
Enter the GitLab instance URL (for example, https://gitlab.com/):
http://test-gitlab.tomy168.com/
Enter the registration token:
GiwWf6-Msyv29HMe8kDA
Enter a description for the runner:
[588264fe558a]: docker executor
Enter tags for the runner (comma-separated):
docker-runner
Enter optional maintenance note for the runner:
WARNING: Support for registration tokens and runner parameters in the 'register' command has been deprecated in GitLab Runner 15.6 and will be replaced with support for authentication tokens. For more information, see https://gitlab.com/gitlab-org/gitlab/-/issues/380872
Registering runner... succeeded runner=GiwWf6
Enter an executor: docker+machine, docker-ssh+machine, custom, docker-ssh, parallels, shell, ssh, docker, virtualbox, instance, kubernetes:
docker
Enter the default Docker image (for example, ruby:2.7):
docker:20.10.24
Runner registered successfully. Feel free to start it, but if it's running already the config should be automatically reloaded!
Configuration (with the authentication token) was saved in "/etc/gitlab-runner/config.toml"其中executor指的是該runner用什麼方式執行CI/CD所分配的任務，default Docker image指的是pipeline預設使用哪一個映像檔去跑任務，這邊隨便設定就可以了，因為通常我們會在.gitlab-ci.yml中指定映像檔而非用預設的。
可以在admin後台看到已註冊的runner，預設是shared runner、可供所有專案的CI/CD使用
D. test-docker機器上安裝harbor映像檔倉庫
對象：test-docker
用途：組織內能提供對不同帳號成員做權限管理的docker registry
至github下載所需版本的tgz壓縮檔，本例選擇online線上安裝
[root@test-docker ~]# cd /opt/docker_home [root@test-docker docker_home]# wget https://github.com/goharbor/harbor/releases/download/v2.8.2/harbor-online-installer-v2.8.2.tgz [root@test-docker docker_home]# tar zxvf ./harbor-online-installer-v2.8.2.tgz>
配置harbor並對設定檔中的相關列變數進行修改
[root@test-docker docker_home]# cd harbor [root@test-docker harbor]# cp harbor.yml.tmpl harbor.yml [root@test-docker harbor]# nano harbor.yml
<harbor.yml>
hostname: test-harbor.tomy168.com # 請依照需求自行設定
http:
port:8088 # 請依照需求自行設定
#https:
#port: 443
#certificate: /your/certificate/path
#private_key: /your/private/key/path
external_url: http://test-harbor.tomy168.com # 請依照需求自行設定
harbor_admin_password: Harbor12345 # 請依照需求自行設定
data_volume: /opt/docker_home/harbor/data # 請依照需求自行設定掛載路徑以避免污染主機安裝harbor
[root@test-docker harbor]# ./install.sh [root@test-docker harbor]# docker-compose ps Name Command State Ports ------------------------------------------------------------------------------------------------------------ harbor-core /harbor/harbor_core Up (healthy) harbor-db /docker-entrypoint.sh 96 13 Up (healthy) 5432/tcp harbor-jobservice /harbor/harbor_jobservice ... Up (healthy) harbor-log /bin/sh -c /usr/local/bin/ ... Up (healthy) 127.0.0.1:1514->10514/tcp harbor-portal nginx -g daemon off; Up (healthy) 8080/tcp nginx nginx -g daemon off; Up (healthy) 0.0.0.0:8088->8080/tcp,:::8088->8080/tcp redis redis-server /etc/redis.conf Up (healthy) 6379/tcp registry /home/harbor/entrypoint.sh Up (healthy) 5000/tcp registryctl /home/harbor/start.sh Up (healthy)
此處我們預設你已配置nginx upstream反向代理http和https至harbor容器的TCP 8088，並已於DNS設定好了a record的指向
特別注意nginx upstream中需要配置參數client_max_body_size避免拉傳image時發生「413 Request Entity Too Large」的狀況
輸入帳號admin與自訂密碼登入harbor
系統已經預設了一個library倉庫給我們使用，這也是後面的實驗會用到的倉庫
因應docker registry交互默認使用的是https、修改私有倉庫地址參數
[root@test-docker harbor]# nano /etc/docker/daemon.json
<daemon.json>
{
"registry-mirrors": ["https://mirror.gcr.io"],
"insecure-registries": ["test-harbor.tomy168.com"]
}重啟服務
[root@test-docker harbor]# systemctl restart docker
確認CLI部分可以順利登入harbor registry
[root@test-docker harbor]# docker login -u admin test-harbor.tomy168.com Password: WARNING! Your password will be stored unencrypted in /root/.docker/config.json. Configure a credential helper to remove this warning. See https://docs.docker.com/engine/reference/commandline/login/#credentials-store Login Succeeded>
測試harbor拉傳映像檔狀況
[root@test-docker harbor]# docker pull alpine:3.18.2 3.18.2: Pulling from library/alpine 31e352740f53: Pull complete Digest: sha256:82d1e9d7ed48a7523bdebc18cf6290bdb97b82302a8a9c27d4fe885949ea94d1 Status: Downloaded newer image for alpine:3.18.2 docker.io/library/alpine:3.18.2 [root@test-docker harbor]# docker image ls alpine 3.18.2 c1aabb73d233 6 weeks ago 7.33MB gitlab/gitlab-ee 15.8.2-ee.0 198a1f4dae4c 5 months ago 3.04GB gitlab/gitlab-runner v15.8.2 33f13ba88111 5 months ago 735MB [root@test-docker harbor]# docker tag alpine:3.18.2 test-harbor.tomy168.com/library/alpine:3.18.2 [root@test-docker harbor]# docker push test-harbor.tomy168.com/library/alpine:3.18.2 The push refers to repository [test-harbor.tomy168.com/library/alpine] 78a822fe2a2d: Pushed 3.18.2: digest: sha256:25fad2a32ad1f6f510e528448ae1ec69a28ef81916a004d3629874104f8a7f70 size: 528
確認alpine映像檔已上傳至harbor的library倉庫裡
E. 替gitlab CI/CD與正式部署節點建立信任
對象：test-deploy
隨便找一台機器創建密鑰對，將公鑰寫入test-deploy機器中的autorized_keys
[root@test-deploy ~]# ssh-keygen -t rsa -C "docker-runner" Generating public/private rsa key pair. Enter file in which to save the key (/root/.ssh/id_rsa): /tmp/id_rsa Enter passphrase (empty for no passphrase): Enter same passphrase again: Your identification has been saved in /tmp/id_rsa. Your public key has been saved in /tmp/id_rsa.pub. The key fingerprint is: SHA256:wL8aM+7HzaA87UOGvQMYzjRr0sD1TYniTLzD/8Ubs1w docker-runner The key's randomart image is: +---[RSA 3072]----+ | . +.. | | .+.+ o | | *o+o. | | %. . | | O *S . | | + @o= = B | | O=o = o * | | .Ooo + + | | o+= | +----[SHA256]-----+ [root@test-deploy ~]# cat /tmp/id_rsa.pub >> ~/.ssh/authorized_keys
私鑰的部分檢視後複製，我們會在F章節將他放入gitlab的專案裡作為變數「TESTDEPLOY_PRIVATE_KEY」，讓gitlab在CI/CD時可以透過SSH控制test-deploy機器
[root@test-deploy ~]# cat /tmp/id.rsa
F. 建立並驗證CI/CD所需的nodejs測試專案
對象：test-docker、gitlab網站
CI/CD流程說明：
- 當專案被commit至main branch時，將觸發.gitlab-ci.yml的CI/CD pipeline
- gitlab驅動gitlab runner（docker executor）創建一個node臨時容器，於容器之中配置package.json內容、並對code進行單元測試
- pipeline測試無誤後接著gitlab runner創建一個docker臨時容器透過Dockerfile用來build包含test-nodejs專案所需的docker image、並推送至docker registry
- pipeline最後階段透過docker registry於遠端正式機器部署我們剛建立好的docker image
創建一個git倉庫供接下來的測試使用
這裡我們選擇用nodejs作為後續CI/CD的示範，專案名稱為「test-nodejs」
從右下方複製git倉庫位置，供稍後於內部開發環境機器上使用
配置開發環境的開發目錄並將test-nodejs專案的git倉庫給複製下來
[root@test-docker ~]# git config --global user.email "admin@example.com" [root@test-docker ~]# git config --global user.name "root" [root@test-docker ~]# mkdir -p /opt/git_home && cd /opt/git_home [root@test-docker git_home]# git clone http://test-gitlab.tomy168.com/gitlab-instance-0e2f120d/test-nodejs.git Cloning into 'test-nodejs'... Username for 'http://test-gitlab.tomy168.com': root Password for 'http://root@test-gitlab.tomy168.com': remote: Enumerating objects: 3, done. remote: Counting objects: 100% (3/3), done. remote: Compressing objects: 100% (2/2), done. remote: Total 3 (delta 0), reused 0 (delta 0), pack-reused 0 Receiving objects: 100% (3/3), done.
進入專案目錄，配置test-nodejs專案所需相關套件與環境設定檔
[root@test-docker git_home]# cd test-nodejs [root@test-docker test-nodejs]# nano package.json
<package.json>
{
"name": "nodejs_example",
"version": "1.0.0",
"description": "",
"main": "app.js",
"scripts": {
"start": "node app.js",
"test": "./node_modules/nyc/bin/nyc.js --reporter=text-summary ./node_modules/mocha/bin/mocha 'test/**/*test.js' --exit"
},
"author": "",
"license": "ISC",
"dependencies": {
"express": "^4.18.2"
},
"devDependencies": {
"mocha": "^8.0.0",
"nyc": "^15.1.0",
"supertest": "^6.3.3"
}
}編寫main功能，這邊利用express框架提供一個簡易的TCP 3000服務，當收到請求時返回「Hello World」字串
[root@test-docker test-nodejs]# nano app.js
<app.js>
'use strict'
const request = require('supertest')
const app = require('../app')
describe('GET /', () => {
it('should respond "Hello World"', (done) => {
request(app)
.get('/')
.expect(200, 'Hello World', done)
})
})編寫單元測試代碼
[root@test-docker test-nodejs]# mkdir -p test [root@test-docker test-nodejs]# nano ./test/app.test.js
<app.test.js>
'use strict'
const request = require('supertest')
const app = require('../app')
describe('GET /', () => {
it('should respond "Hello World"', (done) => {
request(app)
.get('/')
.expect(200, 'Hello World', done)
})
})編寫Dockerfile，作為日後CI/CD產出應用程式的docker image
[root@test-docker test-nodejs]# nano Dockerfile
<Dockerfile>
FROM node:10.24.0 WORKDIR /app # Copy nodejs to the container COPY . /app/ # Install dependency RUN npm install # Run App CMD ["npm", "run", "start"]
編寫.gitlab-ci.yml，注意這邊測試Stage用的映像檔需要和Dockerfile裡的base image一致，才能保證應用程式在測試階段與部署到正式環境的結果是一致的
[root@test-docker test-nodejs]# nano .gitlab-ci.yml
<.gitlab-ci.yml>
stages:
- test
- build
- deploy
test-job:
tags:
- docker-runner
image: node:10.24.0
stage: test
before_script:
- npm install
script:
- npm test
build-job:
tags:
- docker-runner
image: docker:20.10.24
stage: build
script:
- echo "Start building docker image"
- docker login -u $REGISTRY_USER -p $REGISTRY_PASSWD $REGISTRY_URL
- docker build -t $REGISTRY_URL/$REPOSITORY_NAME/$CI_PROJECT_NAME:$CI_COMMIT_SHORT_SHA .
- docker push $REGISTRY_URL/$REPOSITORY_NAME/$CI_PROJECT_NAME:$CI_COMMIT_SHORT_SHA
- echo "Building docker image successfully!"
deploy-job:
tags:
- docker-runner
image: ubuntu:20.04
stage: deploy
before_script:
- "which ssh-agent || ( apt-get update -y && apt-get install openssh-client -y )"
- eval $(ssh-agent -s)
- echo "${TESTDEPLOY_PRIVATE_KEY}" | tr -d '\r' > deploy.pem
- chmod 400 deploy.pem
- ssh-add deploy.pem
- mkdir -p ~/.ssh
- chmod 700 ~/.ssh
- 'echo -e "Host *\n\tStrictHostKeyChecking no\n\n" > ~/.ssh/config'
script:
- ssh root@192.168.88.123 "docker images && docker ps -a"
- ssh root@192.168.88.123 "docker login -u $REGISTRY_USER -p $REGISTRY_PASSWD $REGISTRY_URL"
- ssh root@192.168.88.123 "docker pull $REGISTRY_URL/$REPOSITORY_NAME/$CI_PROJECT_NAME:$CI_COMMIT_SHORT_SHA"
- ssh root@192.168.88.123 "docker run -d --name ${CI_PROJECT_NAME}_${CI_COMMIT_SHORT_SHA} -p 3000:3000 $REGISTRY_URL/$REPOSITORY_NAME/$CI_PROJECT_NAME:$CI_COMMIT_SHORT_SHA"
- echo "deploy complete!"其中.gitlab-ci.yml使用的變數可以是系統預設變數如「$CI_PROJECT_NAME」或是專案自定義變數，以本例來說沒有掛CI前綴的都是我設定的專案自定義變數，一般用意是可以保護token等相關敏感資料，可在gitlab如下處配置
- REGISTRY_PASSWD請設定為harbor的admin密碼
- REGISTRY_URL設定為harbor的域名（本例為test-harbor.tomy168.com）
- REGISTRY_USER設定為admin
- REPOSITORY_NAME設定為library（harbor預設提供的公用倉庫）
- TESTDEPLOY_PRIVATE_KEY設定參考E章節將id_rsa私鑰複製貼上來
將寫好的專案commit至gitlab，只要含有.gitlab-ci.yml的專案被commit並push到main branch，gitlab就會自動觸發CI/CD
[root@test-docker test-nodejs]# git add . [root@test-docker test-nodejs]# git commit -m "first commit" [root@test-docker test-nodejs]# git push origin main
此時再回到gitlab觀看CI/CD的狀況，狀態顯示綠色的passed！
pipeline中的三個stage都成功打勾
分別點進去每個stage裡面，可以看到每個job運行的過程，比如在test-job的最後，執行了npm test，確認了程式碼的功能無誤
來到正式機上驗證nodejs應用果然被成功部署！
[root@test-deploy ~]# docker images REPOSITORY TAG IMAGE ID CREATED SIZE test-harbor.tomy168.com/library/test-nodejs 0078b2f4 8935914c1ff0 About a minute ago 956MB [root@test-deploy ~]# docker ps CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES ca128133ae52 test-harbor.tomy168.com/library/test-nodejs:0078b2f4 "docker-entrypoint.s…" 55 seconds ago Up 54 seconds 0.0.0.0:3000->3000/tcp test-nodejs_0078b2f4 [root@test-deploy ~]# curl localhost:3000 Hello World
來自台灣的系統工程師，一直熱衷於 Open source 相關技術的學習、建置、應用與分享。
0 Comments:
張貼留言