DELAEMON BLOG

Live as if you were to die tomorrow. Learn as if you were to live forever.

Chainer2.0.0@OSX 10.12.5(Python 3.6.1)

環境

$ sw_vers
ProductName:	Mac OS X
ProductVersion:	10.12.5
BuildVersion:	16F73
$ pyenv versions
  system
* 3.6.1 (set by /Users/dela/.pyenv/version)
$ pip freeze | grep chainer
chainer==2.0.0

セットアップ

pip install sklearn
pip install chainer

コード

#!/usr/bin/env python
# -*- encoding: utf-8 -*-

from sklearn.datasets import fetch_mldata
from sklearn.cross_validation import train_test_split
from sklearn.svm import LinearSVC as Classifier
from sklearn.metrics import confusion_matrix
import numpy as np

mnist = fetch_mldata("MNIST original", data_home=".")
data = np.asarray(mnist.data, np.float32)
data_train, data_test, label_train, label_test = train_test_split(data, mnist.target, test_size=0.2)

classifier = Classifier()
classifier.fit(data_train, label_train)

result = classifier.predict(data_test)

cmat = confusion_matrix(label_test, result)
print(cmat)

実行ログ

/Users/dela/.pyenv/versions/3.6.1/lib/python3.6/site-packages/sklearn/cross_validation.py:44: DeprecationWarning: This module was deprecated in version 0.18 in favor of the model_selection module into which all the refactored classes and functions are moved. Also note that the interface of the new CV iterators are different from that of this module. This module will be removed in 0.20.
  "This module will be removed in 0.20.", DeprecationWarning)
[[1306    0    7   20    2    4    2    1   15    1]
 [   0 1511    6   22    2    0    0    5   62    2]
 [  13   24 1068  112   21    4    5   33  149    3]
 [   4    2   21 1360    2    5    2   16   48    9]
 [   5    2    8   14 1228    0    7   13   64   28]
 [  27    2    5  302   21  672    5   15  209   22]
 [  17    5   31   45    8   19 1101    0   68    0]
 [   4    2    4   36   21    0    0 1311   22   21]
 [   9   14    3  102    9    5    2   10 1192   12]
 [  12    2    7   63   62    3    0  158  102 1000]]

Keras 2.0.4@OSX 10.12.5(Python 3.6.1)

環境

$ sw_vers
ProductName:	Mac OS X
ProductVersion:	10.12.5
BuildVersion:	16F73
$ pyenv versions
  system
* 3.6.1 (set by /Users/dela/.pyenv/version)
$ pip freeze | grep Keras
Keras==2.0.4

セットアップ

pip install -U tensorflow
pip install -U keras
pip install h5py
pip install matplotlib

コード

from keras.datasets import mnist
from keras.models import Sequential
from keras.layers.core import Dense, Activation
from keras.utils import np_utils
import matplotlib.pyplot as plt

#ValueError: Error when checking target: expected activation_2 to have shape (None, 10) but got array with shape (60000, 1)
(x_train, y_train), (x_test, y_test) = mnist.load_data()

x_train = x_train.reshape(60000, 784) / 255
x_test = x_test.reshape(10000, 784) / 255

model = Sequential([
    Dense(512, input_shape=(784,)),
    Activation('sigmoid'),
    Dense(10),
    Activation('softmax')
])

model.compile(loss='sparse_categorical_crossentropy', optimizer='sgd', metrics=['accuracy'])

hist = model.fit(x_train, y_train, batch_size=200, verbose=1, epochs=20, validation_split=0.1)

score = model.evaluate(x_test, y_test, verbose=1)
print("\ntest accuracy : ", score[1])

loss = hist.history['loss']
val_loss = hist.history['val_loss']
plt.plot(range(20), loss, marker = '.', label = 'loss')
plt.plot(range(20), val_loss, marker = '.', label = 'val_loss')
plt.legend(loc  = 'best', fontsize = 10)
plt.grid()
plt.xlabel('epoch')
plt.ylabel('loss')
plt.show()

acc = hist.history['acc']
val_acc = hist.history['val_acc']
plt.plot(range(20), acc, marker = '.', label = 'acc')
plt.plot(range(20), val_acc, marker = '.', label = 'val_acc')
plt.legend(loc  = 'best', fontsize = 10)
plt.grid()
plt.xlabel('epoch')
plt.ylabel('acc')
plt.show()

実行ログ

python multi_layer_perceptron.py
Using TensorFlow backend.
Train on 54000 samples, validate on 6000 samples
Epoch 1/20
2017-06-11 09:45:37.849024: W tensorflow/core/platform/cpu_feature_guard.cc:45] The TensorFlow library wasn't compiled to use SSE4.1 instructions, but these are available on your machine and could speed up CPU computations.
2017-06-11 09:45:37.849085: W tensorflow/core/platform/cpu_feature_guard.cc:45] The TensorFlow library wasn't compiled to use SSE4.2 instructions, but these are available on your machine and could speed up CPU computations.
2017-06-11 09:45:37.849100: W tensorflow/core/platform/cpu_feature_guard.cc:45] The TensorFlow library wasn't compiled to use AVX instructions, but these are available on your machine and could speed up CPU computations.
54000/54000 [==============================] - 5s - loss: 2.0821 - acc: 0.4838 - val_loss: 1.8598 - val_acc: 0.6797
Epoch 2/20
54000/54000 [==============================] - 4s - loss: 1.7041 - acc: 0.7005 - val_loss: 1.5052 - val_acc: 0.7832
Epoch 3/20
54000/54000 [==============================] - 4s - loss: 1.4035 - acc: 0.7591 - val_loss: 1.2284 - val_acc: 0.8223
Epoch 4/20
54000/54000 [==============================] - 4s - loss: 1.1764 - acc: 0.7901 - val_loss: 1.0275 - val_acc: 0.8498
Epoch 5/20
54000/54000 [==============================] - 4s - loss: 1.0121 - acc: 0.8095 - val_loss: 0.8814 - val_acc: 0.8563
Epoch 6/20
54000/54000 [==============================] - 4s - loss: 0.8931 - acc: 0.8239 - val_loss: 0.7762 - val_acc: 0.8685
Epoch 7/20
54000/54000 [==============================] - 4s - loss: 0.8052 - acc: 0.8338 - val_loss: 0.6971 - val_acc: 0.8773
Epoch 8/20
54000/54000 [==============================] - 4s - loss: 0.7386 - acc: 0.8405 - val_loss: 0.6370 - val_acc: 0.8825
Epoch 9/20
54000/54000 [==============================] - 4s - loss: 0.6867 - acc: 0.8474 - val_loss: 0.5895 - val_acc: 0.8853
Epoch 10/20
54000/54000 [==============================] - 4s - loss: 0.6451 - acc: 0.8526 - val_loss: 0.5518 - val_acc: 0.8870
Epoch 11/20
54000/54000 [==============================] - 5s - loss: 0.6113 - acc: 0.8565 - val_loss: 0.5204 - val_acc: 0.8915
Epoch 12/20
54000/54000 [==============================] - 4s - loss: 0.5832 - acc: 0.8602 - val_loss: 0.4947 - val_acc: 0.8942
Epoch 13/20
54000/54000 [==============================] - 4s - loss: 0.5595 - acc: 0.8637 - val_loss: 0.4726 - val_acc: 0.8958
Epoch 14/20
54000/54000 [==============================] - 5s - loss: 0.5393 - acc: 0.8668 - val_loss: 0.4542 - val_acc: 0.8985
Epoch 15/20
54000/54000 [==============================] - 4s - loss: 0.5218 - acc: 0.8697 - val_loss: 0.4383 - val_acc: 0.8995
Epoch 16/20
54000/54000 [==============================] - 4s - loss: 0.5066 - acc: 0.8720 - val_loss: 0.4245 - val_acc: 0.8998
Epoch 17/20
54000/54000 [==============================] - 5s - loss: 0.4931 - acc: 0.8737 - val_loss: 0.4119 - val_acc: 0.9013
Epoch 18/20
54000/54000 [==============================] - 4s - loss: 0.4813 - acc: 0.8748 - val_loss: 0.4010 - val_acc: 0.9028
Epoch 19/20
54000/54000 [==============================] - 4s - loss: 0.4705 - acc: 0.8772 - val_loss: 0.3921 - val_acc: 0.9038
Epoch 20/20
54000/54000 [==============================] - 6s - loss: 0.4610 - acc: 0.8788 - val_loss: 0.3831 - val_acc: 0.9052
 9984/10000 [============================>.] - ETA: 0s
test accuracy :  0.8905

グラフ描画

以下のエラーが発生するので、対応が必要

エラー

Fontconfig warning: line 146: blank doesn't take any effect anymore. please remove it from your fonts.conf
/Users/dela/.pyenv/versions/3.6.1/lib/python3.6/site-packages/matplotlib/font_manager.py:280: UserWarning: Matplotlib is building the font cache using fc-list. This may take a moment.
  'Matplotlib is building the font cache using fc-list. '
Traceback (most recent call last):
  File "multi_layer_perceptron.py", line 5, in <module>
    import matplotlib.pyplot as plt
  File "/Users/dela/.pyenv/versions/3.6.1/lib/python3.6/site-packages/matplotlib/pyplot.py", line 115, in <module>
    _backend_mod, new_figure_manager, draw_if_interactive, _show = pylab_setup()
  File "/Users/dela/.pyenv/versions/3.6.1/lib/python3.6/site-packages/matplotlib/backends/__init__.py", line 32, in pylab_setup
    globals(),locals(),[backend_name],0)
  File "/Users/dela/.pyenv/versions/3.6.1/lib/python3.6/site-packages/matplotlib/backends/backend_macosx.py", line 19, in <module>
    from matplotlib.backends import _macosx
RuntimeError: Python is not installed as a framework. The Mac OS X backend will not be able to function correctly if Python is not installed as a framework. See the Python documentation for more information on installing Python as a framework on Mac OS X. Please either reinstall Python as a framework, or try one of the other backends. If you are using (Ana)Conda please install python.app and replace the use of 'python' with 'pythonw'. See 'Working with Matplotlib on OSX' in the Matplotlib FAQ for more information.

対応

$ python -c "import matplotlib;print(matplotlib.matplotlib_fname())"
$ vim /Users/dela/.pyenv/versions/3.6.1/lib/python3.6/site-packages/matplotlib/mpl-data/matplotlibrc
line:38 #backend      : macosx
line:39 backend      : Tkagg

グラフ

f:id:delaemon:20170611104848p:plain

Nodejs Jest CiecleCI

課題

CricleCIでReactjs + Jest でテストを実行していたら、最近になってメモリの制限にひっかかって失敗するようになった。

原因

CircleCI の ビルドに割り当てられる最大メモリ容量は4GB
Your build hit the 4G memory limit - CircleCI

テストフレームワーク Jest は 見えてるCPU数 - 1Coreを使う
https://github.com/facebook/jest/blob/ec782b567bf343d9d278cd79265a1af2bfc3e83c/packages/jest-runtime/src/cli/index.js#L72

nodejsはデフォルトのメモリ割り当ては1.5GBまで。v8起因。オプションつければ上げ下げできる(GCの実行判定調整)
--max_old_space_sizeで調整可能。

CircleCIはホストが8Core。コンテナへのCPU割り当てがどうなってるかは不明だけど、メモリの最大利用サイズは以下
8Core × 1.5GB = 12GB
4Core × 1.5GB = 6GB
2Core × 1.5GB = 3GB

対策

JestのWorker数を制限すればいい。
maxWorkersオプションで指定できる。

% jest --maxWorkers=2

テストの中身によるかもだけど、--maxWorkers=1にしても実行時間に大差はなかった。

GCするメモリ量も決めておくと、コマンド見ただけでイメージつきやすくて、Worker数変更したときにケアしないといけないことに気付き易くて、安心

% node --max_old_space_size=2048 jest --maxWorkers=2

無事CIはSuccess状態に戻った

おまけ

v8のオプションには他にも細かくメモリ関連のオプションが付いており、nodejsからも指定できる。

% node -v
v6.4.0
% node --v8-options | grep size
  --optimize_for_size (Enables optimizations which favor memory size over execution speed)
  --max_inlined_source_size (maximum source size in bytes considered for a single inlining)
  --typed_array_max_size_in_heap (threshold for in-heap typed array)
  --stack_size (default size of stack region v8 is allowed to use (in kBytes))
  --min_semi_space_size (min size of a semi-space (in MBytes), the new space consists of twosemi-spaces)
  --max_semi_space_size (max size of a semi-space (in MBytes), the new space consists of twosemi-spaces)
  --max_old_space_size (max size of the old space (in Mbytes))
  --initial_old_space_size (initial old space size (in Mbytes))
  --max_executable_size (max size of executable memory (in Mbytes))
  --heap_profiler_trace_objects (Dump heap object allocations/movements/size_updates)
  --sim_stack_size (Stack size of the ARM64, MIPS64 and PPC64 simulator in kBytes (default is 2 MB))
  --force_marking_deque_overflows (force overflows of marking deque by reducing it's size to 64 words)

redis3.2@Fedora22

環境

OS: Fedora22

概要

dnsコマンドではredis2.8がinstallされるので、redis3.2を手動でビルド

手順

1. ソースをダウンロード

% wget http://download.redis.io/releases/redis-3.2.5.tar.gz
% tar xvzf redis-3.2.5.tar.gz
% cd redis-3.2.5/

README.mdよんで手順を確認

2. ビルド

% make

....

    CC geo.o
    LINK redis-server
    INSTALL redis-sentinel
    CC redis-cli.o
    LINK redis-cli
    CC redis-benchmark.o
    LINK redis-benchmark
    INSTALL redis-check-rdb
    CC redis-check-aof.o
    LINK redis-check-aof

Hint: It's a good idea to run 'make test' ;)

make[1]: Leaving directory '/home/dela/source/redis-3.2.5/src'

geo.oがみえる。3.2からGEO APIが追加された。

3. ビルドのテスト

% make test

....

\o/ All tests passed without errors!

Cleanup: may take some time... OK
make[1]: Leaving directory '/home/dela/source/redis-3.2.5/src'

4. 起動

% ./src/redis-server

5. 疎通確認

% (echo "PING\r\nPING\r\nPING\r\n"; sleep 1) | nc localhost 6379
+PONG
+PONG
+PONG

その他

ヘルプ見てみる

./src/redis-server -h
Usage: ./redis-server [/path/to/redis.conf] [options]
       ./redis-server - (read config from stdin)
       ./redis-server -v or --version
       ./redis-server -h or --help
       ./redis-server --test-memory <megabytes>

Examples:
       ./redis-server (run the server with default conf)
       ./redis-server /etc/redis/6379.conf
       ./redis-server --port 7777
       ./redis-server --port 7777 --slaveof 127.0.0.1 8888
       ./redis-server /etc/myredis.conf --loglevel verbose

Sentinel mode:
       ./redis-server /etc/sentinel.conf --sentinel

"--test-memory"とは。
Problems with Redis? This is a good starting point. – Redis

% ./src/redis-server --test-memory 1000

# AAA..., CCC..., RRR..., SSS....,で画面が埋まるけど放置

Your memory passed this test.
Please if you are still in doubt use the following two tools:
1) memtest86: http://www.memtest86.com/
2) memtester: http://pyropus.ca/software/memtester/

latency problems troubleshootingのページも役立ちそう。
Redis latency problems troubleshooting – Redis

Hugo SSL/TLS(Let's Encrypt) Renew

Systemdで動かしている Hugo サイトのSSL/TLS(Let's Encrypt)を更新した。

更新しようとしたらエラー

$ ./certbot-auto renew
Requesting root privileges to run certbot...
  /home/dela/.local/share/letsencrypt/bin/letsencrypt renew

-------------------------------------------------------------------------------
Processing <domain>.conf
-------------------------------------------------------------------------------

-------------------------------------------------------------------------------
The program hugo (process ID 786) is already listening on TCP port 80. This will
prevent us from binding to that port. Please stop the hugo program temporarily
and then try again. For automated renewal, you may want to use a script that
stops and starts your webserver. You can find an example at
https://letsencrypt.org/howitworks/#writing-your-own-renewal-script.
Alternatively you can use the webroot plugin to renew without needing to stop
and start your webserver.
-------------------------------------------------------------------------------
2016-09-28 22:22:53,632:WARNING:certbot.renewal:Attempting to renew cert from <domain>.conf produced an unexpected error: At least one of the (possibly) required ports is already taken.. Skipping.

All renewal attempts failed. The following certs could not be renewed:
 <domain>/fullchain.pem (failure)
1 renew failure(s), 0 parse failure(s)

80portを使ってるサービスを停止

$ sudo systemctl stop hugo

再度、更新しようとしたらエラー

$ ./certbot-auto renew
Requesting root privileges to run certbot...
  /home/dela/.local/share/letsencrypt/bin/letsencrypt renew

-------------------------------------------------------------------------------
Processing <domain>.conf
-------------------------------------------------------------------------------

-------------------------------------------------------------------------------
The program hugo-https (process ID 14806) is already listening on TCP port 443.
This will prevent us from binding to that port. Please stop the hugo-https
program temporarily and then try again. For automated renewal, you may want to
use a script that stops and starts your webserver. You can find an example at
https://letsencrypt.org/howitworks/#writing-your-own-renewal-script.
Alternatively you can use the webroot plugin to renew without needing to stop
and start your webserver.
-------------------------------------------------------------------------------
2016-09-28 22:26:37,520:WARNING:certbot.renewal:Attempting to renew cert from <domain>.conf produced an unexpected error: At least one of the (possibly) required ports is already taken.. Skipping.

All renewal attempts failed. The following certs could not be renewed:
  <domain>/fullchain.pem (failure)
1 renew failure(s), 0 parse failure(s)

443portを使ってるサービスを停止

$ sudo systemctl stop hugos

再々更新

$ ./certbot-auto renew
Requesting root privileges to run certbot...
  /home/dela/.local/share/letsencrypt/bin/letsencrypt renew

-------------------------------------------------------------------------------
Processing <domain>.conf
-------------------------------------------------------------------------------

-------------------------------------------------------------------------------
new certificate deployed without reload, fullchain is
<domain>/fullchain.pem
-------------------------------------------------------------------------------

Congratulations, all renewals succeeded. The following certs have been renewed:
  <domain>/fullchain.pem (success)

停止したサービスを起動

$ sudo systemctl start hugos
$ sudo systemctl start hugo

簡単だけど、サービス無停止で更新する方法を考える

AWS Certified Solution Architect Associate 対策

1軍サービス

制限、実践的な構成、トラブルシーティングなど詳細に把握しておく必要あり

  • VPC (Multi-AZ, Internet Gateway, Route Table, NetworkACL, Security Group)
  • IAM
  • EC2 (+ Auto Scaling)
  • EBS
  • S3
  • CloudFront
  • Route53

2軍サービス

サービス概要・基本的な使い方まで把握しておけばOK

  • CloudFormation
  • RDS
  • ElastiCache
  • DynamoDB
  • DirectConnect
  • Trusted Advisor
  • CloudTrail
  • CloudWatch
  • SNS
  • SQS
  • OpsWorks
  • KMS / STS

※1軍7割、2軍3割くらいの力配分

役立つドキュメント

※1軍2軍共にブラックベルトの資料を読むよりは、マニュアルを読む方が役立つ印象

その他

  • AWS用語覚える
  • 模擬試験・サンプル問題などで問題文の文章になれる

※ 最初の感覚を掴むためにEC2+RDS+ElasCticache+S3+VPC+EIPでWebサイトを立てて、AWS ConsoleからCloudwatchLogs, TrustedAdvisorみたり、IAM Role, SecurityGroupいじるくらいやっておくと良さそう

http/2@Hugo & Let's Encrypt

Let's Encryptで発行した証明書を使って、GolangのHugoで動いてるWebサイトをhttp/2にする

環境

OS: Fedora22
Go: 1.6.2

準備

firewalldのhttps/443番ポートを有効にしておく

$ sudo firewall-cmd --list-all
$ sudo firewall-cmd --permanent --add-service=https
$ sudo firewall-cmd --permanent --add-port=443/tcp
$ sudo systemctl restart firewalld
$ sudo firewall-cmd --list-all

80番 ポートで動いてるサービスは止めておく
Certbotクライアントが疎通確認をする場合に既に80番ポートが使われていると、メッセージが表示されて、先に進めない

SSL/TLS証明書発行

ここ読めば簡単
Let's Encrypt の使い方 - Let's Encrypt 総合ポータル

Let's Encryptで証明書を発行するツールのcertbotをダウンロード

$ git clone https://github.com/certbot/certbot

Let's Encryptで証明証を発行する

./certbot-auto certonly --standalone -d <domain>

Certbotクライアントの問いに答えてく。連絡先のメアドとか、同意しますかとか

生成されたファイルの保存場所と期限。期限は3ヶ月と短い

IMPORTANT NOTES:
 - Congratulations! Your certificate and chain have been saved at
   /etc/letsencrypt/live/<domain>/fullchain.pem. Your cert
   will expire on <有効期限>.

でも更新はこれだけで良いので簡単。毎日更新すると怒られる

$ ./certbot-auto renew

Hugo

自分のレポジトリにcloneしてコード変更

commands/server.go

-       err = http.ListenAndServe(endpoint, nil)
+       err = http.ListenAndServeTLS(endpoint, "/etc/letsencrypt/live/<domain>/fullchain.pem", "/etc/letsencrypt/live/<domain>/privkey.pem

main.go

-       "github.com/spf13/hugo/commands"
+       "github.com/delaemon/hugo-https/commands"

go build して $GOPATH/bin/ に hugo-httpsとして置いておく

Systemd

新しいサービスとして設定ファイルを追加
web socket使ってるところをwssにするのは面倒だったのでExecStart の hugo-https serverコマンドに

    • disableLiveReloadを付けた

付けえない場合は以下のエラーがブラウザで吐かれていた

livereload.js:1 Mixed Content: The page at 'https://<domain>' was loaded over HTTPS, but attempted to connect to the insecure WebSocket endpoint 'ws://<domain>:443/livereload'. This request has been blocked; this endpoint must be available over WSS.n.Connector.e.connect @ livereload.js:1
livereload.js:1 Uncaught SecurityError: Failed to construct 'WebSocket': An insecure WebSocket connection may not be initiated from a page loaded over HTTPS.

Go

当初サーバーにインストールされてたGoはdnf installしたgo1.5.3 linux/amd64だったが、
これではhttps通信になっても、http/2にはならなかった。1.6.2をダウンロードして配置したら、http/2になった