chibi@1604:~$ sudo nvidia-docker run --rm -ti nvcr.io/nvidia/tensorflow:18.04-py 3 Unable to find image 'nvcr.io/nvidia/tensorflow:18.04-py3' locally 18.04-py3: Pulling from nvidia/tensorflow f2233041f557: Pulling fs layer f321bcc6a76c: Pulling fs layer 2f25d8d1d058: Pulling fs layer 87bfe0d2f0e8: Waiting 145c1bf7947a: Waiting b146afd09c39: Waiting daadc0376ab4: Waiting b35ad3405786: Pulling fs layer a33366ff728e: Waiting 9384272f4e0f: Waiting 7537bf7c96ee: Waiting 91e40537a771: Waiting fc54fc28c6ac: Pulling fs layer 8916482f2511: Waiting a839dd4b292e: Pulling fs layer cbe311e8f219: Pull complete d0d0e0570d0b: Pull complete de7441a98787: Pull complete ade9aa97247a: Pull complete 0e7e4a241af1: Pull complete 5cd0f4fcfcdb: Pull complete 24e05bc1c0e7: Pull complete d035a88e9703: Pull complete 4455b37f1ded: Pull complete 26869b529dd7: Pull complete 140115f8aca4: Pull complete 26c9801d4f61: Pull complete 2ed2d812e59f: Pull complete 6c8c432fc81a: Pull complete 7a4180c5e47b: Pull complete 6dcfd394e7e9: Pull complete e4f43f8223d0: Pull complete e60c3de5820e: Pull complete 00afe61cb1f7: Pull complete d0fc9beab7e4: Pull complete beaa6e8c724e: Pull complete 821c7edbef93: Pull complete b3cfb51d2ac6: Pull complete Digest: sha256:9fc19d659a37d443e5320a20e86582e1e167323ac839d43fa85db9e744e0de71 Status: Downloaded newer image for nvcr.io/nvidia/tensorflow:18.04-py3 ================ == TensorFlow == ================ NVIDIA Release 18.04 (build 377127) Container image Copyright (c) 2018, NVIDIA CORPORATION. All rights reserved. Copyright 2017 The TensorFlow Authors. All rights reserved. Various files include modifications (c) NVIDIA CORPORATION. All rights reserved. NVIDIA modifications are covered by the license terms that apply to the underlying project or file. WARNING: Detected NVIDIA TITAN RTX GPU, which is not yet supported in this version of the container WARNING: Detected NVIDIA TITAN RTX GPU, which is not yet supported in this version of the container WARNING: Detected NVIDIA GeForce RTX 2080 Ti GPU, which is not yet supported in this version of the container WARNING: Detected NVIDIA GeForce RTX 2080 Ti GPU, which is not yet supported in this version of the container ERROR: No NVIDIA supported GPU(s) detected to run this container NOTE: The SHMEM allocation limit is set to the default of 64MB. This may be insufficient for TensorFlow. NVIDIA recommends the use of the following flags: nvidia-docker run --shm-size=1g --ulimit memlock=-1 --ulimit stack=67108864 ... root@2083bb347905:/workspace# ls README.md docker-examples nvidia-examples root@2083bb347905:/workspace# cd nvidia-examples/cnn root@2083bb347905:/workspace/nvidia-examples/cnn# time python nvcnn.py --model=googlenet \ > --batch_size=256 \ > --num_gpus=4 \ > --fp16 WARNING:tensorflow:From /usr/local/lib/python3.5/dist-packages/tensorflow/contrib/learn/python/learn/datasets/base.py:198: retry (from tensorflow.contrib.learn.python.learn.datasets.base) is deprecated and will be removed in a future version. Instructions for updating: Use the retry module or similar alternatives. TensorFlow: 1.7.0 This script: nvcnn.py v1.6 Cmd line args: --model=googlenet --batch_size=256 --num_gpus=4 --fp16 Num images: Synthetic Model: googlenet Batch size: 1024 global 256.0 per device Devices: ['/gpu:0', '/gpu:1', '/gpu:2', '/gpu:3'] Data format: NCHW Data type: fp16 Have NCCL: True Using NCCL: True Using XLA: False Building training graph Creating session 2020-01-27 22:49:08.765571: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1344] Found device 0 with properties: name: TITAN RTX major: 7 minor: 5 memoryClockRate(GHz): 1.77 pciBusID: 0000:82:00.0 totalMemory: 23.65GiB freeMemory: 23.48GiB 2020-01-27 22:49:08.980744: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1344] Found device 1 with properties: name: TITAN RTX major: 7 minor: 5 memoryClockRate(GHz): 1.77 pciBusID: 0000:83:00.0 totalMemory: 23.65GiB freeMemory: 23.48GiB 2020-01-27 22:49:09.146293: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1344] Found device 2 with properties: name: GeForce RTX 2080 Ti major: 7 minor: 5 memoryClockRate(GHz): 1.635 pciBusID: 0000:02:00.0 totalMemory: 10.76GiB freeMemory: 9.79GiB 2020-01-27 22:49:09.301730: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1344] Found device 3 with properties: name: GeForce RTX 2080 Ti major: 7 minor: 5 memoryClockRate(GHz): 1.635 pciBusID: 0000:03:00.0 totalMemory: 10.76GiB freeMemory: 10.60GiB 2020-01-27 22:49:09.301994: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1423] Adding visible gpu devices: 0, 1, 2, 3 2020-01-27 22:49:10.479032: I tensorflow/core/common_runtime/gpu/gpu_device.cc:911] Device interconnect StreamExecutor with strength 1 edge matrix: 2020-01-27 22:49:10.479092: I tensorflow/core/common_runtime/gpu/gpu_device.cc:917] 0 1 2 3 2020-01-27 22:49:10.479106: I tensorflow/core/common_runtime/gpu/gpu_device.cc:930] 0: N N N N 2020-01-27 22:49:10.479115: I tensorflow/core/common_runtime/gpu/gpu_device.cc:930] 1: N N N N 2020-01-27 22:49:10.479124: I tensorflow/core/common_runtime/gpu/gpu_device.cc:930] 2: N N N N 2020-01-27 22:49:10.479132: I tensorflow/core/common_runtime/gpu/gpu_device.cc:930] 3: N N N N 2020-01-27 22:49:10.479313: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1041] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 22776 MB memory) -> physical GPU (device: 0, name: TITAN RTX, pci bus id: 0000:82:00.0, compute capability: 7.5) 2020-01-27 22:49:10.937430: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1041] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:1 with 22776 MB memory) -> physical GPU (device: 1, name: TITAN RTX, pci bus id: 0000:83:00.0, compute capability: 7.5) 2020-01-27 22:49:11.396154: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1041] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:2 with 9456 MB memory) -> physical GPU (device: 2, name: GeForce RTX 2080 Ti, pci bus id: 0000:02:00.0, compute capability: 7.5) 2020-01-27 22:49:11.587216: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1041] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:3 with 10243 MB memory) -> physical GPU (device: 3, name: GeForce RTX 2080 Ti, pci bus id: 0000:03:00.0, compute capability: 7.5) Initializing variables Pre-filling input pipeline Training Step Epoch Img/sec Loss LR 1 1 48.3 7.369 0.04000 2 1 296.2 7.369 0.03842 3 1 555.9 7.369 0.03686 4 1 559.1 7.369 0.03534 5 1 918.5 7.369 0.03386 6 1 958.8 7.369 0.03240 7 1 1583.8 7.369 0.03098 8 1 1581.5 7.369 0.02958 9 1 2426.6 7.369 0.02822 10 1 2437.0 7.369 0.02690 11 1 2806.5 7.369 0.02560 12 1 2789.2 7.369 0.02434 13 1 3059.8 7.369 0.02310 14 1 3054.8 7.369 0.02190 15 1 3042.3 7.369 0.02074 16 1 3039.0 7.369 0.01960 17 1 3916.1 7.369 0.01850 18 1 3896.6 7.369 0.01742 19 1 3851.6 7.369 0.01638 20 1 3839.2 7.369 0.01538 21 1 3899.6 7.369 0.01440 22 1 3888.1 7.369 0.01346 23 1 3872.2 7.369 0.01254 24 1 3877.5 7.369 0.01166 25 1 3896.7 7.369 0.01082 26 1 3904.5 7.369 0.01000 27 1 3869.5 7.369 0.00922 28 1 3905.0 7.369 0.00846 29 1 3845.2 7.369 0.00774 30 1 3885.3 7.369 0.00706 31 1 3895.1 7.369 0.00640 32 1 3881.6 7.369 0.00578 33 1 3911.9 7.369 0.00518 34 1 3875.9 7.369 0.00462 35 1 3900.0 7.369 0.00410 36 1 3870.7 7.369 0.00360 37 1 3862.4 7.369 0.00314 38 1 3860.6 7.369 0.00270 39 1 3871.3 7.369 0.00230 40 1 3883.5 7.369 0.00194 41 1 3868.9 7.369 0.00160 42 1 3821.6 7.369 0.00130 43 1 3866.4 7.369 0.00102 44 1 3875.1 7.369 0.00078 45 1 3908.3 7.369 0.00058 46 1 3898.4 7.369 0.00040 47 1 3890.0 7.369 0.00026 48 1 3858.0 7.369 0.00014 49 1 3878.8 7.369 0.00006 50 1 3860.1 7.368 0.00002 ---------------------------------------------------------------- Images/sec: 3879.4 +/- 3.7 (jitter = 20.4) ---------------------------------------------------------------- real 1m8.356s user 1m29.583s sys 0m31.795s root@2083bb347905:/workspace/nvidia-examples/cnn# time python nvcnn.py --model=googlenet --batch_size=256 --num_gpus=2 --fp16 WARNING:tensorflow:From /usr/local/lib/python3.5/dist-packages/tensorflow/contrib/learn/python/learn/datasets/base.py:198: retry (from tensorflow.contrib.learn.python.learn.datasets.base) is deprecated and will be removed in a future version. Instructions for updating: Use the retry module or similar alternatives. TensorFlow: 1.7.0 This script: nvcnn.py v1.6 Cmd line args: --model=googlenet --batch_size=256 --num_gpus=2 --fp16 Num images: Synthetic Model: googlenet Batch size: 512 global 256.0 per device Devices: ['/gpu:0', '/gpu:1'] Data format: NCHW Data type: fp16 Have NCCL: True Using NCCL: True Using XLA: False Building training graph Creating session 2020-01-27 22:52:56.131714: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1344] Found device 0 with properties: name: TITAN RTX major: 7 minor: 5 memoryClockRate(GHz): 1.77 pciBusID: 0000:82:00.0 totalMemory: 23.65GiB freeMemory: 23.48GiB 2020-01-27 22:52:56.343492: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1344] Found device 1 with properties: name: TITAN RTX major: 7 minor: 5 memoryClockRate(GHz): 1.77 pciBusID: 0000:83:00.0 totalMemory: 23.65GiB freeMemory: 23.48GiB 2020-01-27 22:52:56.512256: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1344] Found device 2 with properties: name: GeForce RTX 2080 Ti major: 7 minor: 5 memoryClockRate(GHz): 1.635 pciBusID: 0000:02:00.0 totalMemory: 10.76GiB freeMemory: 9.79GiB 2020-01-27 22:52:56.673778: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1344] Found device 3 with properties: name: GeForce RTX 2080 Ti major: 7 minor: 5 memoryClockRate(GHz): 1.635 pciBusID: 0000:03:00.0 totalMemory: 10.76GiB freeMemory: 10.60GiB 2020-01-27 22:52:56.674086: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1423] Adding visible gpu devices: 0, 1, 2, 3 2020-01-27 22:52:57.796324: I tensorflow/core/common_runtime/gpu/gpu_device.cc:911] Device interconnect StreamExecutor with strength 1 edge matrix: 2020-01-27 22:52:57.796387: I tensorflow/core/common_runtime/gpu/gpu_device.cc:917] 0 1 2 3 2020-01-27 22:52:57.796396: I tensorflow/core/common_runtime/gpu/gpu_device.cc:930] 0: N N N N 2020-01-27 22:52:57.796403: I tensorflow/core/common_runtime/gpu/gpu_device.cc:930] 1: N N N N 2020-01-27 22:52:57.796409: I tensorflow/core/common_runtime/gpu/gpu_device.cc:930] 2: N N N N 2020-01-27 22:52:57.796416: I tensorflow/core/common_runtime/gpu/gpu_device.cc:930] 3: N N N N 2020-01-27 22:52:57.796645: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1041] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 22776 MB memory) -> physical GPU (device: 0, name: TITAN RTX, pci bus id: 0000:82:00.0, compute capability: 7.5) 2020-01-27 22:52:58.212946: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1041] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:1 with 22776 MB memory) -> physical GPU (device: 1, name: TITAN RTX, pci bus id: 0000:83:00.0, compute capability: 7.5) 2020-01-27 22:52:58.628584: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1041] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:2 with 9456 MB memory) -> physical GPU (device: 2, name: GeForce RTX 2080 Ti, pci bus id: 0000:02:00.0, compute capability: 7.5) 2020-01-27 22:52:58.801928: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1041] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:3 with 10243 MB memory) -> physical GPU (device: 3, name: GeForce RTX 2080 Ti, pci bus id: 0000:03:00.0, compute capability: 7.5) Initializing variables Pre-filling input pipeline Training Step Epoch Img/sec Loss LR 1 1 63.9 7.369 0.04000 2 1 162.0 7.369 0.03842 3 1 452.8 7.369 0.03686 4 1 464.5 7.369 0.03534 5 1 970.3 7.369 0.03386 6 1 974.6 7.369 0.03240 7 1 1374.2 7.369 0.03098 8 1 1388.8 7.369 0.02958 9 1 1464.2 7.369 0.02822 10 1 1475.4 7.369 0.02690 11 1 1873.6 7.369 0.02560 12 1 1904.5 7.369 0.02434 13 1 1868.0 7.369 0.02310 14 1 1911.1 7.369 0.02190 15 1 1896.1 7.369 0.02074 16 1 1893.0 7.369 0.01960 17 1 1892.0 7.369 0.01850 18 1 1889.9 7.369 0.01742 19 1 1883.7 7.369 0.01638 20 1 1895.8 7.369 0.01538 21 1 1891.0 7.369 0.01440 22 1 1881.3 7.369 0.01346 23 1 1899.6 7.369 0.01254 24 1 1919.6 7.369 0.01166 25 1 1877.0 7.369 0.01082 26 1 1895.5 7.369 0.01000 27 1 1903.5 7.369 0.00922 28 1 1874.6 7.369 0.00846 29 1 1876.6 7.369 0.00774 30 1 1891.1 7.369 0.00706 31 1 1850.9 7.369 0.00640 32 1 1876.9 7.369 0.00578 33 1 1872.6 7.369 0.00518 34 1 1876.9 7.369 0.00462 35 1 1901.3 7.369 0.00410 36 1 1885.2 7.369 0.00360 37 1 1882.8 7.369 0.00314 38 1 1867.4 7.369 0.00270 39 1 1889.5 7.369 0.00230 40 1 1901.9 7.369 0.00194 41 1 1861.7 7.369 0.00160 42 1 1888.2 7.368 0.00130 43 1 1901.8 7.368 0.00102 44 1 1866.0 7.368 0.00078 45 1 1873.0 7.368 0.00058 46 1 1880.7 7.369 0.00040 47 1 1870.1 7.369 0.00026 48 1 1883.6 7.369 0.00014 49 1 1884.5 7.369 0.00006 50 1 1900.3 7.369 0.00002 ---------------------------------------------------------------- Images/sec: 1884.2 +/- 2.7 (jitter = 13.9) ---------------------------------------------------------------- real 0m43.418s user 0m43.225s sys 0m18.900s root@2083bb347905:/workspace/nvidia-examples/cnn# cat /etc/os-release NAME="Ubuntu" VERSION="16.04.3 LTS (Xenial Xerus)" ID=ubuntu ID_LIKE=debian PRETTY_NAME="Ubuntu 16.04.3 LTS" VERSION_ID="16.04" HOME_URL="http://www.ubuntu.com/" SUPPORT_URL="http://help.ubuntu.com/" BUG_REPORT_URL="http://bugs.launchpad.net/ubuntu/" VERSION_CODENAME=xenial UBUNTU_CODENAME=xenial root@2083bb347905:/workspace/nvidia-examples/cnn# nvcc -V nvcc: NVIDIA (R) Cuda compiler driver Copyright (c) 2005-2017 NVIDIA Corporation Built on Fri_Sep__1_21:08:03_CDT_2017 Cuda compilation tools, release 9.0, V9.0.176 root@2083bb347905:/workspace/nvidia-examples/cnn# exit exit chibi@1604:~$ cat /etc/os-release NAME="Ubuntu" VERSION="16.04.6 LTS (Xenial Xerus)" ID=ubuntu ID_LIKE=debian PRETTY_NAME="Ubuntu 16.04.6 LTS" VERSION_ID="16.04" HOME_URL="http://www.ubuntu.com/" SUPPORT_URL="http://help.ubuntu.com/" BUG_REPORT_URL="http://bugs.launchpad.net/ubuntu/" VERSION_CODENAME=xenial UBUNTU_CODENAME=xenial chibi@1604:~$ nvcc -V nvcc: NVIDIA (R) Cuda compiler driver Copyright (c) 2005-2019 NVIDIA Corporation Built on Wed_Oct_23_19:24:38_PDT_2019 Cuda compilation tools, release 10.2, V10.2.89 chibi@1604:~$ sudo hddtemp /dev/sda [sudo] chibi のパスワード: /dev/sda: ST2000LX001-1RG174: 20°C chibi@1604:~$ nvidia-smi nvlink -c GPU 0: GeForce RTX 2080 Ti (UUID: GPU-1ac935c2-557f-282e-14e5-3f749ffd63ac) GPU 1: GeForce RTX 2080 Ti (UUID: GPU-13277ce5-e1e9-0cb1-8cee-6c9e6618e774) GPU 2: TITAN RTX (UUID: GPU-5a71d61e-f130-637a-b33d-4df555b0ed88) GPU 3: TITAN RTX (UUID: GPU-7fb51c1d-c1e7-35cc-aad7-66971f05ddb7) chibi@1604:~$