DELAEMON BLOG

Live as if you were to die tomorrow. Learn as if you were to live forever.

Tensorflow Conda Apple Silicon M1

M1 Mac (macOS Monterey 12.1) にTensor Flow環境を開発用に構築する手順。Xcodeのセットアップは終わってる前提。

Conda

Conda ダウンロード元 GitHub - conda-forge/miniforge: A conda-forge distribution.

% bash Miniforge3-MacOSX-arm64.sh # M1用インストールファイル
% conda config --set auto_activate_base false # シェル起動時のConda環境起動を無効化
% conda create --name tensorflow26 python=3.9 # Condaのデフォルト(2022年1月15日時点)Python 3.10では以降出てくるtensorflow-*がサポート外。バージョン進み過ぎのため明示的にサポート内Pythonバージョンを指定
% conda activate tensorflow26

Tensor Flow

% conda install numpy opencv matplotlib
% conda install -c apple tensorflow-deps
% python -m pip install tensorflow-metal
% python -m pip install tensorflow-macos

Jupyter

% conda install -y jupyter

Tensor Flow 実行時、無視してるエラーメッセージ (今はそれでいいらしい。結果には影響なく正常に動いている)

2022-01-15 14:59:26.033384: I tensorflow/core/common_runtime/pluggable_device/pluggable_device_factory.cc:305] Could not identify NUMA node of platform GPU ID 0, defaulting to 0. Your kernel may not have been built with NUMA support.
Metal device set to: Apple M1
2022-01-15 14:59:26.033582: I tensorflow/core/common_runtime/pluggable_device/pluggable_device_factory.cc:271] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 0 MB memory) -> physical PluggableDevice (device: 0, name: METAL, pci bus id: <undefined>)

Jupyter NotebookからConda仮想環境を使う設定

% conda install ipykernel
% ipython kernel install --user --name tensorflow26
% jupyter kernelspec list

Jupyter Notebookのデフォルトブラウザを変更

jupyter notebook --generate-config
vim ~/.jupyter/jupyter_notebook_config.py

#コメントアウトを外して追記
#c.NotebookApp.browser = '/Applications/Google\ Chrome.app/Contents/MacOS/Google\ Chrome %s'

Jupypter Server for iPad Pro

iPad Proを購入したものの、やっぱり軽くでもいいからコード書いて、あわよくば実行できる環境が欲しい。
そこでJupter Serverを既存レンタルサーバーへインストールした。

インストール

$ pip install jupyter

パスワード生成

$ python -c "import IPython;print(IPython.lib.passwd())"
Enter password:
Verify password:
sha1:xxxxxxxxxxxxxxxxxxxxxxxxxx

**コンフィグ生成
>|sh|
$ jupyter notebook --generate-config
$ ls ~/.jupyter/jupyter_notebook_config.py

コンフィグ編集箇所

c.NotebookApp.ip = '<server ip address>'
c.NotebookApp.notebook_dir = u'/var/jupyter'
c.NotebookApp.open_browser = False
c.NotebookApp.password = u'sha1:xxxxxxxxxxxxxxxxxxxxxxxxxx'
c.NotebookApp.port = 5656

Jupyter server起動

$ jupyter notebook

※rootで動かしちゃう場合は --allow-root オプションが必要

常時使うわけじゃないので、必要な時だけ立ち上げてます。

WebAssembly with Rust

環境

必要物のバージョン確認

$ python --version; node -v; git --version; java -version; gcc --version; g++; cmake
Python 3.6.1
v8.9.1
git version 2.10.1
Picked up JAVA_TOOL_OPTIONS: -Dfile.encoding=UTF-8
java version "1.8.0_65"
Java(TM) SE Runtime Environment (build 1.8.0_65-b17)
Java HotSpot(TM) 64-Bit Server VM (build 25.65-b01, mixed mode)
Configured with: --prefix=/Library/Developer/CommandLineTools/usr --with-gxx-include-dir=/usr/include/c++/4.2.1
Apple LLVM version 9.0.0 (clang-900.0.38)
Target: x86_64-apple-darwin17.3.0
Thread model: posix
InstalledDir: /Library/Developer/CommandLineTools/usr/bin
clang: error: no input files
Usage

  cmake [options] <path-to-source>
  cmake [options] <path-to-existing-build>

Specify a source directory to (re-)generate a build system for it in the
current working directory.  Specify an existing build directory to
re-generate its build system.

Run 'cmake --help' for more information.

ecmascriptenインストール

$ git clone https://github.com/juj/emsdk.git
$ cd emsdk
$ source ~/emsdk/emsdk_env.sh
$ ./emsdk install sdk-incoming-64bit binaryen-master-64bit
$ ./emsdk activate sdk-incoming-64bit binaryen-master-64bit
  • emsdkコマンドはどうでもいいところがうまく動かなくてテキトウにいじった。ただのpython

とりあえずRustなしでWebAssemblyをお試し

Hello World
$ cat hello.c 
#include <stdio.h>

int main() {
  printf("Hello world!");
  return 0;
}
コンパイル / Webサーバーを起動して確認
$ emcc -o hello.html -s WASM=1 hello.c -s NO_EXIT_RUNTIME=0

$ ls 
hello.c hello.html hello.js hello.wasm

$ emrun hello.html
cppでフィボナッチ
$ cat fib.cpp
#include <iostream>
#include <iomanip>
using namespace std;

int fib(int n) {
    if (n < 2) return n;
    return fib(n - 1) + fib(n - 2);
}

int main(int argc, const char* argv[]) {
    int n = 10000;
    int i = fib(n);
    std::cout << "n: " << n << " i: " << i << endl;
    return 0;
}

$ emcc -o fib.html -s WASM=1 fib.cpp -s NO_EXIT_RUNTIME=0
$ emrun fib.html
  • n を10000000とか大きくするとコンパイル通るけど、ブラウザで Maximum call stack size exceeded

RustでWebAssembly

古いRustを消す
$ /usr/local/lib/rustlib/uninstall.sh
Rustインストール
$ curl https://sh.rustup.rs -sSf | sh
Rust で Hello World
$ cargo new --bin hello
$ cd hello
$ cat src/main.rs
fn main() {
    println!("Hello, world!");
}

$ cargo run
Hello, world!
Rust で フィボナッチ
$ cargo new --bin fib
$ cat src/main.rs
pub fn fibonacci_reccursive(n: i32) -> u64 {
    if n < 0 {
        panic!("{} is negative!", n);
    }
    match n {
        0     => panic!("zero is not a right argument to fibonacci_reccursive()!"),
        1 | 2 => 1,
        3     => 2,
        /*
         * 50 => 12586269025,
         */
        _     => fibonacci_reccursive(n - 1) + fibonacci_reccursive(n - 2)
    }
}

fn main() {
    let found = fibonacci_reccursive(10);
    println!("{}", found);
}

$ cargo run
55
Rustでasmjsコンパイル
$ rustup target add asmjs-unknown-emscripten
$ cargo build --target asmjs-unknown-emscripten
$ ls fib/target/asmjs-unknown-emscripten/debug/ 
build       deps        examples    fib.d       fib.js      incremental native
ブラウザから確認
$ cat fib_rust.html
<!DOCTYPE html>
<html>
    <head>
        <meta charset="utf-8"/>
        <title>Rust to asm.js example</title>
    </head>
    <body>
        <script src="fib/target/asmjs-unknown-emscripten/debug/fib.js"></script>
    </body>
</html>

$ php -S localhost:8080

Chainer2.0.0@OSX 10.12.5(Python 3.6.1)

環境

$ sw_vers
ProductName:	Mac OS X
ProductVersion:	10.12.5
BuildVersion:	16F73
$ pyenv versions
  system
* 3.6.1 (set by /Users/dela/.pyenv/version)
$ pip freeze | grep chainer
chainer==2.0.0

セットアップ

pip install sklearn
pip install chainer

コード

#!/usr/bin/env python
# -*- encoding: utf-8 -*-

from sklearn.datasets import fetch_mldata
from sklearn.cross_validation import train_test_split
from sklearn.svm import LinearSVC as Classifier
from sklearn.metrics import confusion_matrix
import numpy as np

mnist = fetch_mldata("MNIST original", data_home=".")
data = np.asarray(mnist.data, np.float32)
data_train, data_test, label_train, label_test = train_test_split(data, mnist.target, test_size=0.2)

classifier = Classifier()
classifier.fit(data_train, label_train)

result = classifier.predict(data_test)

cmat = confusion_matrix(label_test, result)
print(cmat)

実行ログ

/Users/dela/.pyenv/versions/3.6.1/lib/python3.6/site-packages/sklearn/cross_validation.py:44: DeprecationWarning: This module was deprecated in version 0.18 in favor of the model_selection module into which all the refactored classes and functions are moved. Also note that the interface of the new CV iterators are different from that of this module. This module will be removed in 0.20.
  "This module will be removed in 0.20.", DeprecationWarning)
[[1306    0    7   20    2    4    2    1   15    1]
 [   0 1511    6   22    2    0    0    5   62    2]
 [  13   24 1068  112   21    4    5   33  149    3]
 [   4    2   21 1360    2    5    2   16   48    9]
 [   5    2    8   14 1228    0    7   13   64   28]
 [  27    2    5  302   21  672    5   15  209   22]
 [  17    5   31   45    8   19 1101    0   68    0]
 [   4    2    4   36   21    0    0 1311   22   21]
 [   9   14    3  102    9    5    2   10 1192   12]
 [  12    2    7   63   62    3    0  158  102 1000]]

Keras 2.0.4@OSX 10.12.5(Python 3.6.1)

環境

$ sw_vers
ProductName:	Mac OS X
ProductVersion:	10.12.5
BuildVersion:	16F73
$ pyenv versions
  system
* 3.6.1 (set by /Users/dela/.pyenv/version)
$ pip freeze | grep Keras
Keras==2.0.4

セットアップ

pip install -U tensorflow
pip install -U keras
pip install h5py
pip install matplotlib

コード

from keras.datasets import mnist
from keras.models import Sequential
from keras.layers.core import Dense, Activation
from keras.utils import np_utils
import matplotlib.pyplot as plt

#ValueError: Error when checking target: expected activation_2 to have shape (None, 10) but got array with shape (60000, 1)
(x_train, y_train), (x_test, y_test) = mnist.load_data()

x_train = x_train.reshape(60000, 784) / 255
x_test = x_test.reshape(10000, 784) / 255

model = Sequential([
    Dense(512, input_shape=(784,)),
    Activation('sigmoid'),
    Dense(10),
    Activation('softmax')
])

model.compile(loss='sparse_categorical_crossentropy', optimizer='sgd', metrics=['accuracy'])

hist = model.fit(x_train, y_train, batch_size=200, verbose=1, epochs=20, validation_split=0.1)

score = model.evaluate(x_test, y_test, verbose=1)
print("\ntest accuracy : ", score[1])

loss = hist.history['loss']
val_loss = hist.history['val_loss']
plt.plot(range(20), loss, marker = '.', label = 'loss')
plt.plot(range(20), val_loss, marker = '.', label = 'val_loss')
plt.legend(loc  = 'best', fontsize = 10)
plt.grid()
plt.xlabel('epoch')
plt.ylabel('loss')
plt.show()

acc = hist.history['acc']
val_acc = hist.history['val_acc']
plt.plot(range(20), acc, marker = '.', label = 'acc')
plt.plot(range(20), val_acc, marker = '.', label = 'val_acc')
plt.legend(loc  = 'best', fontsize = 10)
plt.grid()
plt.xlabel('epoch')
plt.ylabel('acc')
plt.show()

実行ログ

python multi_layer_perceptron.py
Using TensorFlow backend.
Train on 54000 samples, validate on 6000 samples
Epoch 1/20
2017-06-11 09:45:37.849024: W tensorflow/core/platform/cpu_feature_guard.cc:45] The TensorFlow library wasn't compiled to use SSE4.1 instructions, but these are available on your machine and could speed up CPU computations.
2017-06-11 09:45:37.849085: W tensorflow/core/platform/cpu_feature_guard.cc:45] The TensorFlow library wasn't compiled to use SSE4.2 instructions, but these are available on your machine and could speed up CPU computations.
2017-06-11 09:45:37.849100: W tensorflow/core/platform/cpu_feature_guard.cc:45] The TensorFlow library wasn't compiled to use AVX instructions, but these are available on your machine and could speed up CPU computations.
54000/54000 [==============================] - 5s - loss: 2.0821 - acc: 0.4838 - val_loss: 1.8598 - val_acc: 0.6797
Epoch 2/20
54000/54000 [==============================] - 4s - loss: 1.7041 - acc: 0.7005 - val_loss: 1.5052 - val_acc: 0.7832
Epoch 3/20
54000/54000 [==============================] - 4s - loss: 1.4035 - acc: 0.7591 - val_loss: 1.2284 - val_acc: 0.8223
Epoch 4/20
54000/54000 [==============================] - 4s - loss: 1.1764 - acc: 0.7901 - val_loss: 1.0275 - val_acc: 0.8498
Epoch 5/20
54000/54000 [==============================] - 4s - loss: 1.0121 - acc: 0.8095 - val_loss: 0.8814 - val_acc: 0.8563
Epoch 6/20
54000/54000 [==============================] - 4s - loss: 0.8931 - acc: 0.8239 - val_loss: 0.7762 - val_acc: 0.8685
Epoch 7/20
54000/54000 [==============================] - 4s - loss: 0.8052 - acc: 0.8338 - val_loss: 0.6971 - val_acc: 0.8773
Epoch 8/20
54000/54000 [==============================] - 4s - loss: 0.7386 - acc: 0.8405 - val_loss: 0.6370 - val_acc: 0.8825
Epoch 9/20
54000/54000 [==============================] - 4s - loss: 0.6867 - acc: 0.8474 - val_loss: 0.5895 - val_acc: 0.8853
Epoch 10/20
54000/54000 [==============================] - 4s - loss: 0.6451 - acc: 0.8526 - val_loss: 0.5518 - val_acc: 0.8870
Epoch 11/20
54000/54000 [==============================] - 5s - loss: 0.6113 - acc: 0.8565 - val_loss: 0.5204 - val_acc: 0.8915
Epoch 12/20
54000/54000 [==============================] - 4s - loss: 0.5832 - acc: 0.8602 - val_loss: 0.4947 - val_acc: 0.8942
Epoch 13/20
54000/54000 [==============================] - 4s - loss: 0.5595 - acc: 0.8637 - val_loss: 0.4726 - val_acc: 0.8958
Epoch 14/20
54000/54000 [==============================] - 5s - loss: 0.5393 - acc: 0.8668 - val_loss: 0.4542 - val_acc: 0.8985
Epoch 15/20
54000/54000 [==============================] - 4s - loss: 0.5218 - acc: 0.8697 - val_loss: 0.4383 - val_acc: 0.8995
Epoch 16/20
54000/54000 [==============================] - 4s - loss: 0.5066 - acc: 0.8720 - val_loss: 0.4245 - val_acc: 0.8998
Epoch 17/20
54000/54000 [==============================] - 5s - loss: 0.4931 - acc: 0.8737 - val_loss: 0.4119 - val_acc: 0.9013
Epoch 18/20
54000/54000 [==============================] - 4s - loss: 0.4813 - acc: 0.8748 - val_loss: 0.4010 - val_acc: 0.9028
Epoch 19/20
54000/54000 [==============================] - 4s - loss: 0.4705 - acc: 0.8772 - val_loss: 0.3921 - val_acc: 0.9038
Epoch 20/20
54000/54000 [==============================] - 6s - loss: 0.4610 - acc: 0.8788 - val_loss: 0.3831 - val_acc: 0.9052
 9984/10000 [============================>.] - ETA: 0s
test accuracy :  0.8905

グラフ描画

以下のエラーが発生するので、対応が必要

エラー

Fontconfig warning: line 146: blank doesn't take any effect anymore. please remove it from your fonts.conf
/Users/dela/.pyenv/versions/3.6.1/lib/python3.6/site-packages/matplotlib/font_manager.py:280: UserWarning: Matplotlib is building the font cache using fc-list. This may take a moment.
  'Matplotlib is building the font cache using fc-list. '
Traceback (most recent call last):
  File "multi_layer_perceptron.py", line 5, in <module>
    import matplotlib.pyplot as plt
  File "/Users/dela/.pyenv/versions/3.6.1/lib/python3.6/site-packages/matplotlib/pyplot.py", line 115, in <module>
    _backend_mod, new_figure_manager, draw_if_interactive, _show = pylab_setup()
  File "/Users/dela/.pyenv/versions/3.6.1/lib/python3.6/site-packages/matplotlib/backends/__init__.py", line 32, in pylab_setup
    globals(),locals(),[backend_name],0)
  File "/Users/dela/.pyenv/versions/3.6.1/lib/python3.6/site-packages/matplotlib/backends/backend_macosx.py", line 19, in <module>
    from matplotlib.backends import _macosx
RuntimeError: Python is not installed as a framework. The Mac OS X backend will not be able to function correctly if Python is not installed as a framework. See the Python documentation for more information on installing Python as a framework on Mac OS X. Please either reinstall Python as a framework, or try one of the other backends. If you are using (Ana)Conda please install python.app and replace the use of 'python' with 'pythonw'. See 'Working with Matplotlib on OSX' in the Matplotlib FAQ for more information.

対応

$ python -c "import matplotlib;print(matplotlib.matplotlib_fname())"
$ vim /Users/dela/.pyenv/versions/3.6.1/lib/python3.6/site-packages/matplotlib/mpl-data/matplotlibrc
line:38 #backend      : macosx
line:39 backend      : Tkagg

グラフ

f:id:delaemon:20170611104848p:plain

Nodejs Jest CiecleCI

課題

CricleCIでReactjs + Jest でテストを実行していたら、最近になってメモリの制限にひっかかって失敗するようになった。

原因

CircleCI の ビルドに割り当てられる最大メモリ容量は4GB
Your build hit the 4G memory limit - CircleCI

テストフレームワーク Jest は 見えてるCPU数 - 1Coreを使う
https://github.com/facebook/jest/blob/ec782b567bf343d9d278cd79265a1af2bfc3e83c/packages/jest-runtime/src/cli/index.js#L72

nodejsはデフォルトのメモリ割り当ては1.5GBまで。v8起因。オプションつければ上げ下げできる(GCの実行判定調整)
--max_old_space_sizeで調整可能。

CircleCIはホストが8Core。コンテナへのCPU割り当てがどうなってるかは不明だけど、メモリの最大利用サイズは以下
8Core × 1.5GB = 12GB
4Core × 1.5GB = 6GB
2Core × 1.5GB = 3GB

対策

JestのWorker数を制限すればいい。
maxWorkersオプションで指定できる。

% jest --maxWorkers=2

テストの中身によるかもだけど、--maxWorkers=1にしても実行時間に大差はなかった。

GCするメモリ量も決めておくと、コマンド見ただけでイメージつきやすくて、Worker数変更したときにケアしないといけないことに気付き易くて、安心

% node --max_old_space_size=2048 jest --maxWorkers=2

無事CIはSuccess状態に戻った

おまけ

v8のオプションには他にも細かくメモリ関連のオプションが付いており、nodejsからも指定できる。

% node -v
v6.4.0
% node --v8-options | grep size
  --optimize_for_size (Enables optimizations which favor memory size over execution speed)
  --max_inlined_source_size (maximum source size in bytes considered for a single inlining)
  --typed_array_max_size_in_heap (threshold for in-heap typed array)
  --stack_size (default size of stack region v8 is allowed to use (in kBytes))
  --min_semi_space_size (min size of a semi-space (in MBytes), the new space consists of twosemi-spaces)
  --max_semi_space_size (max size of a semi-space (in MBytes), the new space consists of twosemi-spaces)
  --max_old_space_size (max size of the old space (in Mbytes))
  --initial_old_space_size (initial old space size (in Mbytes))
  --max_executable_size (max size of executable memory (in Mbytes))
  --heap_profiler_trace_objects (Dump heap object allocations/movements/size_updates)
  --sim_stack_size (Stack size of the ARM64, MIPS64 and PPC64 simulator in kBytes (default is 2 MB))
  --force_marking_deque_overflows (force overflows of marking deque by reducing it's size to 64 words)

redis3.2@Fedora22

環境

OS: Fedora22

概要

dnsコマンドではredis2.8がinstallされるので、redis3.2を手動でビルド

手順

1. ソースをダウンロード

% wget http://download.redis.io/releases/redis-3.2.5.tar.gz
% tar xvzf redis-3.2.5.tar.gz
% cd redis-3.2.5/

README.mdよんで手順を確認

2. ビルド

% make

....

    CC geo.o
    LINK redis-server
    INSTALL redis-sentinel
    CC redis-cli.o
    LINK redis-cli
    CC redis-benchmark.o
    LINK redis-benchmark
    INSTALL redis-check-rdb
    CC redis-check-aof.o
    LINK redis-check-aof

Hint: It's a good idea to run 'make test' ;)

make[1]: Leaving directory '/home/dela/source/redis-3.2.5/src'

geo.oがみえる。3.2からGEO APIが追加された。

3. ビルドのテスト

% make test

....

\o/ All tests passed without errors!

Cleanup: may take some time... OK
make[1]: Leaving directory '/home/dela/source/redis-3.2.5/src'

4. 起動

% ./src/redis-server

5. 疎通確認

% (echo "PING\r\nPING\r\nPING\r\n"; sleep 1) | nc localhost 6379
+PONG
+PONG
+PONG

その他

ヘルプ見てみる

./src/redis-server -h
Usage: ./redis-server [/path/to/redis.conf] [options]
       ./redis-server - (read config from stdin)
       ./redis-server -v or --version
       ./redis-server -h or --help
       ./redis-server --test-memory <megabytes>

Examples:
       ./redis-server (run the server with default conf)
       ./redis-server /etc/redis/6379.conf
       ./redis-server --port 7777
       ./redis-server --port 7777 --slaveof 127.0.0.1 8888
       ./redis-server /etc/myredis.conf --loglevel verbose

Sentinel mode:
       ./redis-server /etc/sentinel.conf --sentinel

"--test-memory"とは。
Problems with Redis? This is a good starting point. – Redis

% ./src/redis-server --test-memory 1000

# AAA..., CCC..., RRR..., SSS....,で画面が埋まるけど放置

Your memory passed this test.
Please if you are still in doubt use the following two tools:
1) memtest86: http://www.memtest86.com/
2) memtester: http://pyropus.ca/software/memtester/

latency problems troubleshootingのページも役立ちそう。
Redis latency problems troubleshooting – Redis