Tensorflow Conda Apple Silicon M1
M1 Mac (macOS Monterey 12.1) にTensor Flow環境を開発用に構築する手順。Xcodeのセットアップは終わってる前提。
Conda
Conda ダウンロード元 GitHub - conda-forge/miniforge: A conda-forge distribution.
% bash Miniforge3-MacOSX-arm64.sh # M1用インストールファイル % conda config --set auto_activate_base false # シェル起動時のConda環境起動を無効化 % conda create --name tensorflow26 python=3.9 # Condaのデフォルト(2022年1月15日時点)Python 3.10では以降出てくるtensorflow-*がサポート外。バージョン進み過ぎのため明示的にサポート内Pythonバージョンを指定 % conda activate tensorflow26
Tensor Flow
% conda install numpy opencv matplotlib % conda install -c apple tensorflow-deps % python -m pip install tensorflow-metal % python -m pip install tensorflow-macos
Jupyter
% conda install -y jupyter
Tensor Flow 実行時、無視してるエラーメッセージ (今はそれでいいらしい。結果には影響なく正常に動いている)
2022-01-15 14:59:26.033384: I tensorflow/core/common_runtime/pluggable_device/pluggable_device_factory.cc:305] Could not identify NUMA node of platform GPU ID 0, defaulting to 0. Your kernel may not have been built with NUMA support. Metal device set to: Apple M1 2022-01-15 14:59:26.033582: I tensorflow/core/common_runtime/pluggable_device/pluggable_device_factory.cc:271] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 0 MB memory) -> physical PluggableDevice (device: 0, name: METAL, pci bus id: <undefined>)
Jupyter NotebookからConda仮想環境を使う設定
% conda install ipykernel % ipython kernel install --user --name tensorflow26 % jupyter kernelspec list
Jupyter Notebookのデフォルトブラウザを変更
jupyter notebook --generate-config vim ~/.jupyter/jupyter_notebook_config.py #コメントアウトを外して追記 #c.NotebookApp.browser = '/Applications/Google\ Chrome.app/Contents/MacOS/Google\ Chrome %s'
Jupypter Server for iPad Pro
iPad Proを購入したものの、やっぱり軽くでもいいからコード書いて、あわよくば実行できる環境が欲しい。
そこでJupter Serverを既存レンタルサーバーへインストールした。
インストール
$ pip install jupyter
パスワード生成
$ python -c "import IPython;print(IPython.lib.passwd())" Enter password: Verify password: sha1:xxxxxxxxxxxxxxxxxxxxxxxxxx **コンフィグ生成 >|sh| $ jupyter notebook --generate-config $ ls ~/.jupyter/jupyter_notebook_config.py
コンフィグ編集箇所
c.NotebookApp.ip = '<server ip address>' c.NotebookApp.notebook_dir = u'/var/jupyter' c.NotebookApp.open_browser = False c.NotebookApp.password = u'sha1:xxxxxxxxxxxxxxxxxxxxxxxxxx' c.NotebookApp.port = 5656
Jupyter server起動
$ jupyter notebook
※rootで動かしちゃう場合は --allow-root オプションが必要
常時使うわけじゃないので、必要な時だけ立ち上げてます。
WebAssembly with Rust
必要物のバージョン確認
$ python --version; node -v; git --version; java -version; gcc --version; g++; cmake Python 3.6.1 v8.9.1 git version 2.10.1 Picked up JAVA_TOOL_OPTIONS: -Dfile.encoding=UTF-8 java version "1.8.0_65" Java(TM) SE Runtime Environment (build 1.8.0_65-b17) Java HotSpot(TM) 64-Bit Server VM (build 25.65-b01, mixed mode) Configured with: --prefix=/Library/Developer/CommandLineTools/usr --with-gxx-include-dir=/usr/include/c++/4.2.1 Apple LLVM version 9.0.0 (clang-900.0.38) Target: x86_64-apple-darwin17.3.0 Thread model: posix InstalledDir: /Library/Developer/CommandLineTools/usr/bin clang: error: no input files Usage cmake [options] <path-to-source> cmake [options] <path-to-existing-build> Specify a source directory to (re-)generate a build system for it in the current working directory. Specify an existing build directory to re-generate its build system. Run 'cmake --help' for more information.
ecmascriptenインストール
$ git clone https://github.com/juj/emsdk.git $ cd emsdk $ source ~/emsdk/emsdk_env.sh $ ./emsdk install sdk-incoming-64bit binaryen-master-64bit $ ./emsdk activate sdk-incoming-64bit binaryen-master-64bit
- emsdkコマンドはどうでもいいところがうまく動かなくてテキトウにいじった。ただのpython
とりあえずRustなしでWebAssemblyをお試し
Hello World
$ cat hello.c #include <stdio.h> int main() { printf("Hello world!"); return 0; }
コンパイル / Webサーバーを起動して確認
$ emcc -o hello.html -s WASM=1 hello.c -s NO_EXIT_RUNTIME=0 $ ls hello.c hello.html hello.js hello.wasm $ emrun hello.html
cppでフィボナッチ
$ cat fib.cpp #include <iostream> #include <iomanip> using namespace std; int fib(int n) { if (n < 2) return n; return fib(n - 1) + fib(n - 2); } int main(int argc, const char* argv[]) { int n = 10000; int i = fib(n); std::cout << "n: " << n << " i: " << i << endl; return 0; } $ emcc -o fib.html -s WASM=1 fib.cpp -s NO_EXIT_RUNTIME=0 $ emrun fib.html
- n を10000000とか大きくするとコンパイル通るけど、ブラウザで Maximum call stack size exceeded
RustでWebAssembly
古いRustを消す
$ /usr/local/lib/rustlib/uninstall.sh
Rustインストール
$ curl https://sh.rustup.rs -sSf | sh
Rust で Hello World
$ cargo new --bin hello $ cd hello $ cat src/main.rs fn main() { println!("Hello, world!"); } $ cargo run Hello, world!
Rust で フィボナッチ
$ cargo new --bin fib $ cat src/main.rs pub fn fibonacci_reccursive(n: i32) -> u64 { if n < 0 { panic!("{} is negative!", n); } match n { 0 => panic!("zero is not a right argument to fibonacci_reccursive()!"), 1 | 2 => 1, 3 => 2, /* * 50 => 12586269025, */ _ => fibonacci_reccursive(n - 1) + fibonacci_reccursive(n - 2) } } fn main() { let found = fibonacci_reccursive(10); println!("{}", found); } $ cargo run 55
Rustでasmjsコンパイル
$ rustup target add asmjs-unknown-emscripten $ cargo build --target asmjs-unknown-emscripten $ ls fib/target/asmjs-unknown-emscripten/debug/ build deps examples fib.d fib.js incremental native
ブラウザから確認
$ cat fib_rust.html <!DOCTYPE html> <html> <head> <meta charset="utf-8"/> <title>Rust to asm.js example</title> </head> <body> <script src="fib/target/asmjs-unknown-emscripten/debug/fib.js"></script> </body> </html> $ php -S localhost:8080
Chainer2.0.0@OSX 10.12.5(Python 3.6.1)
環境
$ sw_vers ProductName: Mac OS X ProductVersion: 10.12.5 BuildVersion: 16F73 $ pyenv versions system * 3.6.1 (set by /Users/dela/.pyenv/version) $ pip freeze | grep chainer chainer==2.0.0
セットアップ
pip install sklearn pip install chainer
コード
#!/usr/bin/env python # -*- encoding: utf-8 -*- from sklearn.datasets import fetch_mldata from sklearn.cross_validation import train_test_split from sklearn.svm import LinearSVC as Classifier from sklearn.metrics import confusion_matrix import numpy as np mnist = fetch_mldata("MNIST original", data_home=".") data = np.asarray(mnist.data, np.float32) data_train, data_test, label_train, label_test = train_test_split(data, mnist.target, test_size=0.2) classifier = Classifier() classifier.fit(data_train, label_train) result = classifier.predict(data_test) cmat = confusion_matrix(label_test, result) print(cmat)
実行ログ
/Users/dela/.pyenv/versions/3.6.1/lib/python3.6/site-packages/sklearn/cross_validation.py:44: DeprecationWarning: This module was deprecated in version 0.18 in favor of the model_selection module into which all the refactored classes and functions are moved. Also note that the interface of the new CV iterators are different from that of this module. This module will be removed in 0.20. "This module will be removed in 0.20.", DeprecationWarning) [[1306 0 7 20 2 4 2 1 15 1] [ 0 1511 6 22 2 0 0 5 62 2] [ 13 24 1068 112 21 4 5 33 149 3] [ 4 2 21 1360 2 5 2 16 48 9] [ 5 2 8 14 1228 0 7 13 64 28] [ 27 2 5 302 21 672 5 15 209 22] [ 17 5 31 45 8 19 1101 0 68 0] [ 4 2 4 36 21 0 0 1311 22 21] [ 9 14 3 102 9 5 2 10 1192 12] [ 12 2 7 63 62 3 0 158 102 1000]]
Keras 2.0.4@OSX 10.12.5(Python 3.6.1)
環境
$ sw_vers ProductName: Mac OS X ProductVersion: 10.12.5 BuildVersion: 16F73 $ pyenv versions system * 3.6.1 (set by /Users/dela/.pyenv/version) $ pip freeze | grep Keras Keras==2.0.4
セットアップ
pip install -U tensorflow pip install -U keras pip install h5py pip install matplotlib
コード
from keras.datasets import mnist from keras.models import Sequential from keras.layers.core import Dense, Activation from keras.utils import np_utils import matplotlib.pyplot as plt #ValueError: Error when checking target: expected activation_2 to have shape (None, 10) but got array with shape (60000, 1) (x_train, y_train), (x_test, y_test) = mnist.load_data() x_train = x_train.reshape(60000, 784) / 255 x_test = x_test.reshape(10000, 784) / 255 model = Sequential([ Dense(512, input_shape=(784,)), Activation('sigmoid'), Dense(10), Activation('softmax') ]) model.compile(loss='sparse_categorical_crossentropy', optimizer='sgd', metrics=['accuracy']) hist = model.fit(x_train, y_train, batch_size=200, verbose=1, epochs=20, validation_split=0.1) score = model.evaluate(x_test, y_test, verbose=1) print("\ntest accuracy : ", score[1]) loss = hist.history['loss'] val_loss = hist.history['val_loss'] plt.plot(range(20), loss, marker = '.', label = 'loss') plt.plot(range(20), val_loss, marker = '.', label = 'val_loss') plt.legend(loc = 'best', fontsize = 10) plt.grid() plt.xlabel('epoch') plt.ylabel('loss') plt.show() acc = hist.history['acc'] val_acc = hist.history['val_acc'] plt.plot(range(20), acc, marker = '.', label = 'acc') plt.plot(range(20), val_acc, marker = '.', label = 'val_acc') plt.legend(loc = 'best', fontsize = 10) plt.grid() plt.xlabel('epoch') plt.ylabel('acc') plt.show()
実行ログ
python multi_layer_perceptron.py Using TensorFlow backend. Train on 54000 samples, validate on 6000 samples Epoch 1/20 2017-06-11 09:45:37.849024: W tensorflow/core/platform/cpu_feature_guard.cc:45] The TensorFlow library wasn't compiled to use SSE4.1 instructions, but these are available on your machine and could speed up CPU computations. 2017-06-11 09:45:37.849085: W tensorflow/core/platform/cpu_feature_guard.cc:45] The TensorFlow library wasn't compiled to use SSE4.2 instructions, but these are available on your machine and could speed up CPU computations. 2017-06-11 09:45:37.849100: W tensorflow/core/platform/cpu_feature_guard.cc:45] The TensorFlow library wasn't compiled to use AVX instructions, but these are available on your machine and could speed up CPU computations. 54000/54000 [==============================] - 5s - loss: 2.0821 - acc: 0.4838 - val_loss: 1.8598 - val_acc: 0.6797 Epoch 2/20 54000/54000 [==============================] - 4s - loss: 1.7041 - acc: 0.7005 - val_loss: 1.5052 - val_acc: 0.7832 Epoch 3/20 54000/54000 [==============================] - 4s - loss: 1.4035 - acc: 0.7591 - val_loss: 1.2284 - val_acc: 0.8223 Epoch 4/20 54000/54000 [==============================] - 4s - loss: 1.1764 - acc: 0.7901 - val_loss: 1.0275 - val_acc: 0.8498 Epoch 5/20 54000/54000 [==============================] - 4s - loss: 1.0121 - acc: 0.8095 - val_loss: 0.8814 - val_acc: 0.8563 Epoch 6/20 54000/54000 [==============================] - 4s - loss: 0.8931 - acc: 0.8239 - val_loss: 0.7762 - val_acc: 0.8685 Epoch 7/20 54000/54000 [==============================] - 4s - loss: 0.8052 - acc: 0.8338 - val_loss: 0.6971 - val_acc: 0.8773 Epoch 8/20 54000/54000 [==============================] - 4s - loss: 0.7386 - acc: 0.8405 - val_loss: 0.6370 - val_acc: 0.8825 Epoch 9/20 54000/54000 [==============================] - 4s - loss: 0.6867 - acc: 0.8474 - val_loss: 0.5895 - val_acc: 0.8853 Epoch 10/20 54000/54000 [==============================] - 4s - loss: 0.6451 - acc: 0.8526 - val_loss: 0.5518 - val_acc: 0.8870 Epoch 11/20 54000/54000 [==============================] - 5s - loss: 0.6113 - acc: 0.8565 - val_loss: 0.5204 - val_acc: 0.8915 Epoch 12/20 54000/54000 [==============================] - 4s - loss: 0.5832 - acc: 0.8602 - val_loss: 0.4947 - val_acc: 0.8942 Epoch 13/20 54000/54000 [==============================] - 4s - loss: 0.5595 - acc: 0.8637 - val_loss: 0.4726 - val_acc: 0.8958 Epoch 14/20 54000/54000 [==============================] - 5s - loss: 0.5393 - acc: 0.8668 - val_loss: 0.4542 - val_acc: 0.8985 Epoch 15/20 54000/54000 [==============================] - 4s - loss: 0.5218 - acc: 0.8697 - val_loss: 0.4383 - val_acc: 0.8995 Epoch 16/20 54000/54000 [==============================] - 4s - loss: 0.5066 - acc: 0.8720 - val_loss: 0.4245 - val_acc: 0.8998 Epoch 17/20 54000/54000 [==============================] - 5s - loss: 0.4931 - acc: 0.8737 - val_loss: 0.4119 - val_acc: 0.9013 Epoch 18/20 54000/54000 [==============================] - 4s - loss: 0.4813 - acc: 0.8748 - val_loss: 0.4010 - val_acc: 0.9028 Epoch 19/20 54000/54000 [==============================] - 4s - loss: 0.4705 - acc: 0.8772 - val_loss: 0.3921 - val_acc: 0.9038 Epoch 20/20 54000/54000 [==============================] - 6s - loss: 0.4610 - acc: 0.8788 - val_loss: 0.3831 - val_acc: 0.9052 9984/10000 [============================>.] - ETA: 0s test accuracy : 0.8905
グラフ描画
以下のエラーが発生するので、対応が必要
エラー
Fontconfig warning: line 146: blank doesn't take any effect anymore. please remove it from your fonts.conf /Users/dela/.pyenv/versions/3.6.1/lib/python3.6/site-packages/matplotlib/font_manager.py:280: UserWarning: Matplotlib is building the font cache using fc-list. This may take a moment. 'Matplotlib is building the font cache using fc-list. ' Traceback (most recent call last): File "multi_layer_perceptron.py", line 5, in <module> import matplotlib.pyplot as plt File "/Users/dela/.pyenv/versions/3.6.1/lib/python3.6/site-packages/matplotlib/pyplot.py", line 115, in <module> _backend_mod, new_figure_manager, draw_if_interactive, _show = pylab_setup() File "/Users/dela/.pyenv/versions/3.6.1/lib/python3.6/site-packages/matplotlib/backends/__init__.py", line 32, in pylab_setup globals(),locals(),[backend_name],0) File "/Users/dela/.pyenv/versions/3.6.1/lib/python3.6/site-packages/matplotlib/backends/backend_macosx.py", line 19, in <module> from matplotlib.backends import _macosx RuntimeError: Python is not installed as a framework. The Mac OS X backend will not be able to function correctly if Python is not installed as a framework. See the Python documentation for more information on installing Python as a framework on Mac OS X. Please either reinstall Python as a framework, or try one of the other backends. If you are using (Ana)Conda please install python.app and replace the use of 'python' with 'pythonw'. See 'Working with Matplotlib on OSX' in the Matplotlib FAQ for more information.
対応
$ python -c "import matplotlib;print(matplotlib.matplotlib_fname())" $ vim /Users/dela/.pyenv/versions/3.6.1/lib/python3.6/site-packages/matplotlib/mpl-data/matplotlibrc line:38 #backend : macosx line:39 backend : Tkagg
グラフ
Nodejs Jest CiecleCI
課題
CricleCIでReactjs + Jest でテストを実行していたら、最近になってメモリの制限にひっかかって失敗するようになった。
原因
CircleCI の ビルドに割り当てられる最大メモリ容量は4GB
Your build hit the 4G memory limit - CircleCI
テストフレームワーク Jest は 見えてるCPU数 - 1Coreを使う
https://github.com/facebook/jest/blob/ec782b567bf343d9d278cd79265a1af2bfc3e83c/packages/jest-runtime/src/cli/index.js#L72
nodejsはデフォルトのメモリ割り当ては1.5GBまで。v8起因。オプションつければ上げ下げできる(GCの実行判定調整)
--max_old_space_sizeで調整可能。
CircleCIはホストが8Core。コンテナへのCPU割り当てがどうなってるかは不明だけど、メモリの最大利用サイズは以下
8Core × 1.5GB = 12GB
4Core × 1.5GB = 6GB
2Core × 1.5GB = 3GB
対策
JestのWorker数を制限すればいい。
maxWorkersオプションで指定できる。
% jest --maxWorkers=2
テストの中身によるかもだけど、--maxWorkers=1にしても実行時間に大差はなかった。
GCするメモリ量も決めておくと、コマンド見ただけでイメージつきやすくて、Worker数変更したときにケアしないといけないことに気付き易くて、安心
% node --max_old_space_size=2048 jest --maxWorkers=2
無事CIはSuccess状態に戻った
おまけ
v8のオプションには他にも細かくメモリ関連のオプションが付いており、nodejsからも指定できる。
% node -v v6.4.0 % node --v8-options | grep size --optimize_for_size (Enables optimizations which favor memory size over execution speed) --max_inlined_source_size (maximum source size in bytes considered for a single inlining) --typed_array_max_size_in_heap (threshold for in-heap typed array) --stack_size (default size of stack region v8 is allowed to use (in kBytes)) --min_semi_space_size (min size of a semi-space (in MBytes), the new space consists of twosemi-spaces) --max_semi_space_size (max size of a semi-space (in MBytes), the new space consists of twosemi-spaces) --max_old_space_size (max size of the old space (in Mbytes)) --initial_old_space_size (initial old space size (in Mbytes)) --max_executable_size (max size of executable memory (in Mbytes)) --heap_profiler_trace_objects (Dump heap object allocations/movements/size_updates) --sim_stack_size (Stack size of the ARM64, MIPS64 and PPC64 simulator in kBytes (default is 2 MB)) --force_marking_deque_overflows (force overflows of marking deque by reducing it's size to 64 words)
redis3.2@Fedora22
環境
OS: Fedora22
概要
dnsコマンドではredis2.8がinstallされるので、redis3.2を手動でビルド
手順
1. ソースをダウンロード
% wget http://download.redis.io/releases/redis-3.2.5.tar.gz
% tar xvzf redis-3.2.5.tar.gz
% cd redis-3.2.5/
README.mdよんで手順を確認
2. ビルド
% make .... CC geo.o LINK redis-server INSTALL redis-sentinel CC redis-cli.o LINK redis-cli CC redis-benchmark.o LINK redis-benchmark INSTALL redis-check-rdb CC redis-check-aof.o LINK redis-check-aof Hint: It's a good idea to run 'make test' ;) make[1]: Leaving directory '/home/dela/source/redis-3.2.5/src'
geo.oがみえる。3.2からGEO APIが追加された。
3. ビルドのテスト
% make test .... \o/ All tests passed without errors! Cleanup: may take some time... OK make[1]: Leaving directory '/home/dela/source/redis-3.2.5/src'
4. 起動
% ./src/redis-server
5. 疎通確認
% (echo "PING\r\nPING\r\nPING\r\n"; sleep 1) | nc localhost 6379 +PONG +PONG +PONG
その他
ヘルプ見てみる
./src/redis-server -h Usage: ./redis-server [/path/to/redis.conf] [options] ./redis-server - (read config from stdin) ./redis-server -v or --version ./redis-server -h or --help ./redis-server --test-memory <megabytes> Examples: ./redis-server (run the server with default conf) ./redis-server /etc/redis/6379.conf ./redis-server --port 7777 ./redis-server --port 7777 --slaveof 127.0.0.1 8888 ./redis-server /etc/myredis.conf --loglevel verbose Sentinel mode: ./redis-server /etc/sentinel.conf --sentinel
"--test-memory"とは。
Problems with Redis? This is a good starting point. – Redis
% ./src/redis-server --test-memory 1000 # AAA..., CCC..., RRR..., SSS....,で画面が埋まるけど放置 Your memory passed this test. Please if you are still in doubt use the following two tools: 1) memtest86: http://www.memtest86.com/ 2) memtester: http://pyropus.ca/software/memtester/
latency problems troubleshootingのページも役立ちそう。
Redis latency problems troubleshooting – Redis