DELAEMON BLOG

Live as if you were to die tomorrow. Learn as if you were to live forever.

CoreOS fleetで分散デプロイ・フェイルオーバー

CoreOS + Vargrant でクラスタ構成を作る - DELAEMON BLOG
前回からの続き

ローカルPCにfleetをセットアップ

% go version
go version go1.4.2 darwin/amd64
% git clone git@github.com:coreos/fleet.git
% cd fleet
% ./build

クラスタを確認

% bin/fleetctl --endpoint=http://172.17.8.101:4001 list-machines -l
MACHINE					IP		METADATA
7911e093c0fd4883891ab966aa7feec3	172.17.8.101	-
f3ea4deb0bf2404db1bd6a9199daa511	172.17.8.102	-
fb059b20960d4119995c23cc22ca178e	172.17.8.103	-

稼働中のunitを確認、この時点では何もない

% bin/fleetctl --endpoint=http://172.17.8.101:4001 list-units
UNIT	MACHINE	ACTIVE	SUB

systemdに読み込まれたunitを確認、これも何もなし

% bin/fleetctl --endpoint=http://172.17.8.101:4001 list-unit-files
UNIT	HASH	DSTATE	STATE	TARGET

unitを登録、確認。systemdに読み込まれたけど、起動はしてない状態

% bin/fleetctl --endpoint=http://172.17.8.101:4001 submit examples/hello.service
% bin/fleetctl --endpoint=http://172.17.8.101:4001 list-unit-files
UNIT		HASH	DSTATE		STATE		TARGET
hello.service	e55c0ae	inactive	inactive	-
% bin/fleetctl --endpoint=http://172.17.8.101:4001 list-units
UNIT	MACHINE	ACTIVE	SUB

unitを起動、確認

% bin/fleetctl --endpoint=http://172.17.8.101:4001 start hello
Unit hello.service launched on 7911e093.../172.17.8.101
% bin/fleetctl --endpoint=http://172.17.8.101:4001 list-unit-files
UNIT		HASH	DSTATE		STATE		TARGET
hello.service	e55c0ae	launched	launched	7911e093.../172.17.8.101
% bin/fleetctl --endpoint=http://172.17.8.101:4001 list-units
UNIT		MACHINE				ACTIVE	SUB
hello.service	7911e093.../172.17.8.101	active	running

現在稼働中のサーバが停止したら、他所に移動するか試すためにVM止める

% vagrant status
Current machine states:

core-01                   running (virtualbox)
core-02                   running (virtualbox)
core-03                   running (virtualbox)

% vagrant suspend core-01
==> core-01: Saving VM state and suspending execution...
% vagrant status
Current machine states:

core-01                   saved (virtualbox)
core-02                   running (virtualbox)
core-03                   running (virtualbox)

unitが別のサーバ上で動いてるのを確認

% bin/fleetctl --endpoint=http://172.17.8.103:4001 list-units
UNIT		MACHINE				ACTIVE	SUB
hello.service	f3ea4deb.../172.17.8.102	active	running
% bin/fleetctl --endpoint=http://172.17.8.103:4001 list-unit-files
UNIT		HASH	DSTATE		STATE		TARGET
hello.service	e55c0ae	launched	launched	f3ea4deb.../172.17.8.102

core-01をvagrant resumeで復活させてもunitは移動しない

依存するunitを同一のサーバ上で動かすサンプルの設定

% cat examples/ping.service
[Unit]
Description=PING

[Service]
ExecStart=/bin/bash -c "while true; do echo \"ping\"; sleep 1; done"

% cat examples/pong.service
[Unit]
Description=PONG

[Service]
ExecStart=/bin/bash -c "while true; do echo \"pong\"; sleep 1; done"

[X-Fleet]
MachineOf=ping.service

X-FleetのMachineOf=ping.serviceのように書く

unit起動、submitは端折れる

% bin/fleetctl --endpoint=http://172.17.8.103:4001 start examples/ping.service
Unit ping.service launched on 7911e093.../172.17.8.101
% bin/fleetctl --endpoint=http://172.17.8.103:4001 start examples/pong.service
Unit pong.service launched on 7911e093.../172.17.8.101
% bin/fleetctl --endpoint=http://172.17.8.103:4001 list-units
UNIT		MACHINE				ACTIVE	SUB
hello.service	f3ea4deb.../172.17.8.102	active	running
ping.service	7911e093.../172.17.8.101	active	running
pong.service	7911e093.../172.17.8.101	active	running
% bin/fleetctl --endpoint=http://172.17.8.103:4001 list-unit-files
UNIT		HASH	DSTATE		STATE		TARGET
hello.service	e55c0ae	launched	launched	f3ea4deb.../172.17.8.102
ping.service	d9de011	launched	launched	7911e093.../172.17.8.101
pong.service	667e8de	launched	launched	7911e093.../172.17.8.101

unitの稼働状態の詳細を確認

% bin/fleetctl --endpoint=http://172.17.8.103:4001 status ping
● ping.service - PING
   Loaded: loaded (/run/fleet/units/ping.service; linked-runtime; vendor preset: disabled)
   Active: active (running) since Sun 2015-08-16 13:09:21 UTC; 1min 38s ago
 Main PID: 2879 (bash)
   Memory: 328.0K
      CPU: 84ms
   CGroup: /system.slice/ping.service
           ├─2879 /bin/bash -c while true; do echo "ping"; sleep 1; done
           └─3101 sleep 1

Aug 16 13:10:51 core-01 bash[2879]: ping
Aug 16 13:10:52 core-01 bash[2879]: ping
Aug 16 13:10:53 core-01 bash[2879]: ping
Aug 16 13:10:54 core-01 bash[2879]: ping
Aug 16 13:10:55 core-01 bash[2879]: ping
Aug 16 13:10:56 core-01 bash[2879]: ping
Aug 16 13:10:57 core-01 bash[2879]: ping
Aug 16 13:10:58 core-01 bash[2879]: ping
Aug 16 13:10:59 core-01 bash[2879]: ping
Aug 16 13:11:00 core-01 bash[2879]: ping
% bin/fleetctl --endpoint=http://172.17.8.103:4001 status pong
● pong.service - PONG
   Loaded: loaded (/run/fleet/units/pong.service; linked-runtime; vendor preset: disabled)
   Active: active (running) since Sun 2015-08-16 13:09:31 UTC; 1min 38s ago
 Main PID: 2911 (bash)
   Memory: 324.0K
      CPU: 80ms
   CGroup: /system.slice/pong.service
           ├─2911 /bin/bash -c while true; do echo "pong"; sleep 1; done
           └─3125 sleep 1

Aug 16 13:11:00 core-01 bash[2911]: pong
Aug 16 13:11:01 core-01 bash[2911]: pong
Aug 16 13:11:02 core-01 bash[2911]: pong
Aug 16 13:11:03 core-01 bash[2911]: pong
Aug 16 13:11:04 core-01 bash[2911]: pong
Aug 16 13:11:05 core-01 bash[2911]: pong
Aug 16 13:11:06 core-01 bash[2911]: pong
Aug 16 13:11:07 core-01 bash[2911]: pong
Aug 16 13:11:08 core-01 bash[2911]: pong
Aug 16 13:11:09 core-01 bash[2911]: pong

unitのログを確認

% bin/fleetctl --endpoint=http://172.17.8.102:4001 journal -f ping
-- Logs begin at Wed 2015-08-12 01:08:16 UTC. --
Aug 16 13:15:49 core-01 bash[2879]: ping
Aug 16 13:15:50 core-01 bash[2879]: ping
Aug 16 13:15:51 core-01 bash[2879]: ping

unitを停止

% bin/fleetctl --endpoint=http://172.17.8.102:4001 stop ping
Unit ping.service loaded on 7911e093.../172.17.8.101
% bin/fleetctl --endpoint=http://172.17.8.102:4001 list-units
UNIT		MACHINE				ACTIVE		SUB
hello.service	f3ea4deb.../172.17.8.102	active		running
ping.service	7911e093.../172.17.8.101	inactive	dead
pong.service	7911e093.../172.17.8.101	active		running
% bin/fleetctl --endpoint=http://172.17.8.102:4001 list-unit-files
UNIT		HASH	DSTATE		STATE		TARGET
hello.service	e55c0ae	launched	launched	f3ea4deb.../172.17.8.102
ping.service	d9de011	loaded		loaded		7911e093.../172.17.8.101
pong.service	667e8de	launched	launched	7911e093.../172.17.8.101

pingを止めてもpongは止まらない

unitを削除

% bin/fleetctl --endpoint=http://172.17.8.102:4001 destroy ping
Destroyed ping.service
% bin/fleetctl --endpoint=http://172.17.8.102:4001 list-units
UNIT		MACHINE				ACTIVE	SUB
hello.service	f3ea4deb.../172.17.8.102	active	running
% bin/fleetctl --endpoint=http://172.17.8.102:4001 list-unit-files
UNIT		HASH	DSTATE		STATE		TARGET
hello.service	e55c0ae	launched	launched	f3ea4deb.../172.17.8.102
pong.service	667e8de	launched	inactive	-

pingをsystemd上の登録から消すと、pongは止まる、systemdには登録されたまま
pingをunloadしてもpongは止まる

依存するunitのサーバが停止したらunitは別サーバに移動されない

% bin/fleetctl --endpoint=http://172.17.8.102:4001 list-unit-files
UNIT		HASH	DSTATE		STATE		TARGET
hello.service	e55c0ae	launched	launched	f3ea4deb.../172.17.8.102
ping.service	d9de011	launched	launched	7911e093.../172.17.8.101
pong.service	667e8de	launched	launched	7911e093.../172.17.8.101

% vagrant suspend core-01
==> core-01: Saving VM state and suspending execution...
% vagrant status
Current machine states:

core-01                   saved (virtualbox)
core-02                   running (virtualbox)
core-03                   running (virtualbox)

% bin/fleetctl --endpoint=http://172.17.8.102:4001 list-unit-files
UNIT		HASH	DSTATE		STATE		TARGET
hello.service	e55c0ae	launched	launched	f3ea4deb.../172.17.8.102
ping.service	d9de011	launched	launched	7911e093.../172.17.8.101
pong.service	667e8de	launched	launched	7911e093.../172.17.8.101
% bin/fleetctl --endpoint=http://172.17.8.102:4001 list-units
UNIT		MACHINE				ACTIVE	SUB
hello.service	f3ea4deb.../172.17.8.102	active	running
ping.service	7911e093.../172.17.8.101	active	running
pong.service	7911e093.../172.17.8.101	active	running
% bin/fleetctl --endpoint=http://172.17.8.102:4001 journal -f ping

Error running remote command: timed out while initiating SSH connection