See https://ci.linaro.org/job/ldcg-python-manylinux-tensorflow-nightly/652/display/redirect
Changes:
------------------------------------------ Started by timer Started by user Marcin Juszkiewicz (marcin.juszkiewicz@linaro.org) Rebuilds build #651 Running as SYSTEM [EnvInject] - Loading node environment variables. Building remotely on ldcg-aarch64-02 (aarch64 leg-aarch64) in workspace https://ci.linaro.org/job/ldcg-python-manylinux-tensorflow-nightly/ws/ [ldcg-python-manylinux-tensorflow-nightly] $ /bin/bash /tmp/jenkins5192070800085698557.sh + dual_abi=false + cd https://ci.linaro.org/job/ldcg-python-manylinux-tensorflow-nightly/ws/ + rm -rf '*.sh' configs wheels cache_upload + mkdir wheels cache_upload + git clone --depth 1 https://git.linaro.org/ci/job/configs.git Cloning into 'configs'... + cd configs + git log --pretty=oneline -1 e1507d86c15436024fcea94fd6117cfaa4dfcafc ldcg-*-tensorflow: use template to store logs + cd .. + '[' '' = true ']' + '[' '' = true ']' + '[' '' = true ']' + '[' '' = true ']' + '[' '' = true ']' + '[' '' = true ']' + '[' '' = true ']' + '[' '' = true ']' + '[' '' = true ']' + '[' true = true ']' + echo ' - "git"' + dual_abi=true + '[' false = true ']' + '[' '' = true ']' + '[' true == true ']' + ansible-galaxy collection install community.docker Starting galaxy collection install process Nothing to do. All requested collections are already installed. If you want to reinstall them, consider using `--force`. + cd configs/ldcg-python-manylinux-tensorflow/ansible/ + sed -i -es+/tmp/workspace+https://ci.linaro.org/job/ldcg-python-manylinux-tensorflow-nightly/ws/+g vars/vars.yml + ansible-playbook playbooks/build-tf-dual-abi.yml [WARNING]: provided hosts list is empty, only localhost is available. Note that the implicit localhost does not match 'all'
PLAY [Build TensorFlow in Dual ABI container] **********************************
TASK [Gathering Facts] ********************************************************* Thursday 19 January 2023 06:05:30 +0000 (0:00:00.056) 0:00:00.056 ****** ok: [localhost]
TASK [Cleanup after previous build] ******************************************** Thursday 19 January 2023 06:05:33 +0000 (0:00:02.618) 0:00:02.674 ****** changed: [localhost] => (item=wheels) changed: [localhost] => (item=ml_wheels) ok: [localhost] => (item=tensorflow)
TASK [Create cache dirs] ******************************************************* Thursday 19 January 2023 06:05:34 +0000 (0:00:01.276) 0:00:03.951 ****** ok: [localhost] => (item=disk-cache) ok: [localhost] => (item=bazel-cache) changed: [localhost] => (item=wheels) changed: [localhost] => (item=ml_wheels)
TASK [Set dual abi variable] *************************************************** Thursday 19 January 2023 06:05:36 +0000 (0:00:01.396) 0:00:05.347 ****** ok: [localhost]
TASK [Go through each TensorFlow version] ************************************** Thursday 19 January 2023 06:05:36 +0000 (0:00:00.035) 0:00:05.383 ****** skipping: [localhost] => (item={'key': '2.11', 'value': {'tensorflow_version': '2.11.0', 'bazel_version': 'bazelisk', 'flatbuffers_version': '>= 1.12, < 3.0', 'gast_version': '==0.4.0', 'grpcio_version': '>= 1.37.0, < 2.0', 'h5py_version': '~=3.6.0', 'numpy_version': '~=1.21.6', 'absl_version': '>=1.0.0', 'scipy_version': '~=1.7.3', 'keras_version': '==2.11.0', 'tensorboard_version': '==2.11.0', 'tensorflow_estimator_version': '==2.11.0', 'python_install_packages': ['packaging', 'portpicker', 'keras-nightly', 'tb-nightly', 'tf-estimator-nightly'], 'run_unit_tests': True, 'run_serial_unit_tests': False, 'onednn_ut_exclude': ['//tensorflow/compiler/mlir/tfr/examples/mnist:mnist_ops_test', '//tensorflow/core/grappler/optimizers:auto_mixed_precision_test_cpu', '//tensorflow/core/grappler/optimizers:remapper_test_cpu', '//tensorflow/python/kernel_tests/nn_ops:atrous_conv2d_test_cpu', '//tensorflow/python/kernel_tests/nn_ops:conv_ops_test_cpu'], 'unit_tests_exclude': ['//tensorflow/lite/...', '//tensorflow/compiler/mlir/lite/tests:const-fold.mlir.test', '//tensorflow/core/distributed_runtime/integration_test:c_api_session_coordination_test_cpu', '//tensorflow/python/client:session_list_devices_test', '//tensorflow/python/data/kernel_tests:iterator_test_cpu', '//tensorflow/python/training:server_lib_test'], 'serial_unit_tests_exclude': '', 'unit_tests_filter_flags': '-no_oss,-oss_serial,-gpu,-tpu,-benchmark-test,-v1only,-no_aarch64', 'python_versions': ['cp37-cp37m', 'cp38-cp38', 'cp39-cp39', 'cp310-cp310']}}) skipping: [localhost] => (item={'key': '2.10', 'value': {'tensorflow_version': '2.10.1', 'bazel_version': 'bazelisk', 'flatbuffers_version': '>= 1.12, < 3.0', 'gast_version': '==0.4.0', 'grpcio_version': '>= 1.37.0, < 2.0', 'h5py_version': '~=3.6.0', 'numpy_version': '~=1.21.6', 'absl_version': '>=1.0.0', 'scipy_version': '~=1.7.3', 'keras_version': '==2.10.0', 'tensorboard_version': '==2.10.1', 'tensorflow_estimator_version': '==2.10.0', 'python_install_packages': ['packaging', 'portpicker', 'keras', 'tensorboard', 'tensorflow-estimator'], 'run_unit_tests': True, 'run_serial_unit_tests': False, 'onednn_ut_exclude': ['//tensorflow/compiler/mlir/tensorflow/tests/...', '//tensorflow/compiler/mlir/tfr/examples/mnist:mnist_ops_test', '//tensorflow/core/grappler/optimizers:auto_mixed_precision_test_cpu', '//tensorflow/core/grappler/optimizers:remapper_test_cpu', '//tensorflow/python/kernel_tests/nn_ops:atrous_conv2d_test_cpu', '//tensorflow/python/kernel_tests/nn_ops:conv_ops_test_cpu'], 'unit_tests_exclude': ['//tensorflow/lite/...', '//tensorflow/compiler/mlir/lite/tests:const-fold.mlir.test', '//tensorflow/compiler/tests:jit_test_cpu', '//tensorflow/core/distributed_runtime/integration_test:c_api_session_coordination_test_cpu', '//tensorflow/python/client:session_list_devices_test', '//tensorflow/python/data/kernel_tests:iterator_test_cpu', '//tensorflow/python/training:server_lib_test'], 'serial_unit_tests_exclude': '', 'unit_tests_filter_flags': '-no_oss,-oss_serial,-gpu,-tpu,-benchmark-test,-v1only,-no_aarch64,-requires-gpu', 'python_versions': ['cp37-cp37m', 'cp38-cp38', 'cp39-cp39', 'cp310-cp310']}}) skipping: [localhost] => (item={'key': '2.9', 'value': {'tensorflow_version': '2.9.3', 'bazel_version': 'bazelisk', 'flatbuffers_version': '>= 1.12, < 3.0', 'gast_version': '==0.4.0', 'grpcio_version': '>= 1.37.0, < 2.0', 'h5py_version': '~=3.6.0', 'numpy_version': '~=1.21.6', 'absl_version': '>=1.0.0', 'scipy_version': '~=1.7.3', 'keras_version': '==2.9.0', 'tensorboard_version': '==2.9.0', 'tensorflow_estimator_version': '==2.9.0', 'python_install_packages': ['packaging', 'portpicker', 'keras', 'tensorboard', 'tensorflow-estimator'], 'run_unit_tests': True, 'run_serial_unit_tests': False, 'onednn_ut_exclude': '', 'unit_tests_exclude': ['//tensorflow/lite/...', '//tensorflow/compiler/mlir/lite/tests:const-fold.mlir.test'], 'serial_unit_tests_exclude': '', 'unit_tests_filter_flags': '-no_oss,-oss_serial,-gpu,-tpu,-benchmark-test,-v1only,-no_aarch64,-requires-gpu', 'python_versions': ['cp37-cp37m', 'cp38-cp38', 'cp39-cp39', 'cp310-cp310']}}) skipping: [localhost] => (item={'key': '2.8', 'value': {'tensorflow_version': '2.8.4', 'bazel_version': 'bazelisk', 'flatbuffers_version': '>= 1.12, < 3.0', 'gast_version': '==0.4.0', 'grpcio_version': '>= 1.37.0, < 2.0', 'h5py_version': '~=3.6.0', 'numpy_version': '~=1.21.6', 'absl_version': '>=1.0.0', 'scipy_version': '~=1.7.3', 'keras_version': '==2.8.0', 'tensorboard_version': '==2.8.0', 'tensorflow_estimator_version': '==2.8.0', 'python_install_packages': ['packaging', 'portpicker', 'keras', 'tensorboard', 'tensorflow-estimator'], 'run_unit_tests': True, 'run_serial_unit_tests': False, 'onednn_ut_exclude': '', 'unit_tests_exclude': ['//tensorflow/lite/...', '//tensorflow/core/ir/...', '//tensorflow/python/tools/...', '//tensorflow/compiler/mlir/lite/tests:const-fold.mlir.test', '//tensorflow/compiler/mlir/lite/tests:prepare-tf.mlir.test', '//tensorflow/compiler/mlir/tensorflow:dump_graph_test', '//tensorflow/core/common_runtime:direct_session_test', '//tensorflow/core/common_runtime:direct_session_with_debug_test', '//tensorflow/core/kernels:quantized_bias_add_op_test', '//tensorflow/core/kernels:quantize_op_test', '//tensorflow/core/kernels:requantize_op_test', '//tensorflow/python/data/experimental/kernel_tests/service:fault_tolerance_test', '//tensorflow/python/eager:function_test', '//tensorflow/python/kernel_tests/linalg:linear_operator_circulant_test', '//tensorflow/python/kernel_tests/linalg:self_adjoint_eig_op_test', '//tensorflow/python/ops/ragged:ragged_dispatch_test', '//tensorflow/python:quantized_ops_test', '//tensorflow/python:script_ops_test', '//tensorflow/tools/graph_transforms:transforms_test'], 'serial_unit_tests_exclude': '', 'unit_tests_filter_flags': '-no_oss,-oss_serial,-gpu,-tpu,-benchmark-test,-v1only,-no_aarch64', 'python_versions': ['cp37-cp37m', 'cp38-cp38', 'cp39-cp39', 'cp310-cp310']}}) skipping: [localhost] => (item={'key': '2.7', 'value': {'tensorflow_version': '2.7.4', 'bazel_version': 'bazelisk', 'flatbuffers_version': '>= 1.12, < 3.0', 'gast_version': '==0.4.0', 'grpcio_version': '~=1.37.0', 'h5py_version': '~=3.1.0', 'numpy_version': '~=1.19.5', 'absl_version': '~=0.10', 'scipy_version': '~=1.5.4', 'keras_version': '==2.7.0', 'tensorboard_version': '==2.7.0', 'tensorflow_estimator_version': '==2.7.0', 'python_install_packages': ['portpicker', 'keras', 'tensorboard', 'tensorflow-estimator'], 'run_unit_tests': True, 'run_serial_unit_tests': False, 'onednn_ut_exclude': '', 'unit_tests_exclude': ['//tensorflow/java/...', '//tensorflow/lite/...', '//tensorflow/compiler/xla/python/...', '//tensorflow/compiler/xla/service/cpu/...', '//tensorflow/compiler/xla/tests/...', '//tensorflow/core/ir/...', '//tensorflow/python/tools/...', '//tensorflow/compiler/mlir/lite/tests:const-fold.mlir.test', '//tensorflow/core/kernels:quantized_bias_add_op_test', '//tensorflow/core/kernels:requantize_op_test', '//tensorflow/core/kernels:sparse_matmul_op_test', '//tensorflow/python/data/experimental/kernel_tests/service:fault_tolerance_test', '//tensorflow/python/debug/cli:debugger_cli_common_test', '//tensorflow/python/debug/examples/v2:examples_v2_test', '//tensorflow/python/eager:function_test', '//tensorflow/python/keras/mixed_precision:model_test', '//tensorflow/python/keras/preprocessing:image_test', '//tensorflow/python/kernel_tests:init_ops_test', '//tensorflow/python/kernel_tests/linalg:linear_operator_circulant_test', '//tensorflow/python/kernel_tests:self_adjoint_eig_op_test', '//tensorflow/python/kernel_tests:sparse_matmul_op_test', '//tensorflow/python/lib/io:file_io_test', '//tensorflow/python/ops/ragged:ragged_dispatch_test', '//tensorflow/python/ops/ragged:ragged_print_op_test', '//tensorflow/python/profiler/integration_test:profiler_api_test', '//tensorflow/python/profiler:profiler_v2_test', '//tensorflow/python/util:dispatch_test', '//tensorflow/python/util:traceback_utils_test', '//tensorflow/tools/api/tests:api_compatibility_test', '//tensorflow/tools/docs:tf_doctest', '//tensorflow/tools/graph_transforms:transforms_test'], 'serial_unit_tests_exclude': '', 'unit_tests_filter_flags': '-no_oss,-oss_serial,-gpu,-tpu,-benchmark-test,-v1only', 'python_versions': ['cp37-cp37m', 'cp38-cp38', 'cp39-cp39']}}) skipping: [localhost] => (item={'key': '2.6', 'value': {'tensorflow_version': '2.6.5', 'bazel_version': 'bazelisk', 'flatbuffers_version': '~=1.12.0', 'gast_version': '==0.4.0', 'grpcio_version': '~=1.37.0', 'h5py_version': '~=3.1.0', 'numpy_version': '~=1.19.2', 'absl_version': '~=0.10', 'scipy_version': '~=1.5.4', 'keras_version': '==2.6.0', 'tensorboard_version': '==2.6.0', 'tensorflow_estimator_version': '==2.6.0', 'python_install_packages': ['keras', 'tensorboard', 'tensorflow-estimator'], 'run_unit_tests': False, 'run_serial_unit_tests': False, 'onednn_ut_exclude': '', 'unit_tests_exclude': '', 'serial_unit_tests_exclude': '', 'unit_tests_filter_flags': '', 'python_versions': ['cp36-cp36m', 'cp37-cp37m', 'cp38-cp38', 'cp39-cp39']}}) skipping: [localhost] => (item={'key': '2.5', 'value': {'tensorflow_version': '2.5.3', 'bazel_version': 'bazelisk', 'flatbuffers_version': '~=1.12.0', 'gast_version': '==0.4.0', 'grpcio_version': '~=1.34.0', 'h5py_version': '~=3.1.0', 'numpy_version': '~=1.19.2', 'absl_version': '~=0.10', 'scipy_version': '~=1.5.4', 'keras_version': '==2.5.0', 'tensorboard_version': '==2.5.0', 'tensorflow_estimator_version': '==2.5.0', 'python_install_packages': ['keras', 'tensorboard', 'tensorflow-estimator'], 'run_unit_tests': False, 'run_serial_unit_tests': False, 'onednn_ut_exclude': '', 'unit_tests_exclude': '', 'serial_unit_tests_exclude': '', 'unit_tests_filter_flags': '', 'python_versions': ['cp36-cp36m', 'cp37-cp37m', 'cp38-cp38', 'cp39-cp39']}}) skipping: [localhost] => (item={'key': '2.4', 'value': {'tensorflow_version': '2.4.4', 'bazel_version': 'bazelisk', 'flatbuffers_version': '~=1.12.0', 'gast_version': '==0.3.3', 'grpcio_version': '~=1.32.0', 'h5py_version': '~=2.10', 'numpy_version': '~=1.19.2', 'absl_version': '~=0.10', 'scipy_version': '~=1.5.4', 'keras_version': '==2.4.3', 'tensorboard_version': '==2.4.1', 'tensorflow_estimator_version': '==2.4.0', 'python_install_packages': ['keras', 'tensorboard', 'tensorflow-estimator'], 'run_unit_tests': False, 'run_serial_unit_tests': False, 'onednn_ut_exclude': '', 'unit_tests_exclude': '', 'serial_unit_tests_exclude': '', 'unit_tests_filter_flags': '', 'python_versions': ['cp36-cp36m', 'cp37-cp37m', 'cp38-cp38', 'cp39-cp39']}}) skipping: [localhost] => (item={'key': '1.15', 'value': {'tensorflow_version': '1.15.5', 'bazel_version': '0.29.1', 'flatbuffers_version': '~=1.12.0', 'gast_version': '==0.2.2', 'grpcio_version': '~=1.32.0', 'h5py_version': '==2.10', 'numpy_version': '~=1.18.0', 'absl_version': '~=0.10', 'scipy_version': '~=1.5.4', 'keras_version': '==1.2.2', 'tensorboard_version': '==1.15.0', 'tensorflow_estimator_version': '==1.15.2', 'python_install_packages': ['keras', 'tensorboard', 'tensorflow-estimator'], 'run_unit_tests': False, 'run_serial_unit_tests': False, 'onednn_ut_exclude': '', 'unit_tests_exclude': '', 'serial_unit_tests_exclude': '', 'unit_tests_filter_flags': '', 'python_versions': ['cp36-cp36m', 'cp37-cp37m']}})
TASK [tensorflow : Set TensorFlow and other package versions] ****************** Thursday 19 January 2023 06:05:36 +0000 (0:00:00.239) 0:00:05.622 ****** ok: [localhost]
TASK [tensorflow : Set build dir for TensorFlow git] *************************** Thursday 19 January 2023 06:05:36 +0000 (0:00:00.081) 0:00:05.704 ****** ok: [localhost]
TASK [tensorflow : Remove TensorFlow git sources from previous build] ********** Thursday 19 January 2023 06:05:36 +0000 (0:00:00.045) 0:00:05.750 ****** changed: [localhost]
TASK [tensorflow : Fetch TensorFlow] ******************************************* Thursday 19 January 2023 06:05:39 +0000 (0:00:03.168) 0:00:08.918 ****** included: https://ci.linaro.org/job/ldcg-python-manylinux-tensorflow-nightly/ws/configs/ldcg-python-manylinux-tensorflow/ansible/roles/tensorflow/tasks/fetch.yml for localhost
TASK [tensorflow : Fetch TensorFlow git source] ******************************** Thursday 19 January 2023 06:05:39 +0000 (0:00:00.099) 0:00:09.017 ****** skipping: [localhost]
TASK [tensorflow : Unpack TensorFlow git source] ******************************* Thursday 19 January 2023 06:05:39 +0000 (0:00:00.039) 0:00:09.057 ****** skipping: [localhost]
TASK [tensorflow : Rename source dir for oneDNN build] ************************* Thursday 19 January 2023 06:05:39 +0000 (0:00:00.035) 0:00:09.093 ****** skipping: [localhost]
TASK [tensorflow : Fetch TensorFlow from git] ********************************** Thursday 19 January 2023 06:05:39 +0000 (0:00:00.038) 0:00:09.132 ****** changed: [localhost]
TASK [tensorflow : Set world write access to source] *************************** Thursday 19 January 2023 06:06:04 +0000 (0:00:24.430) 0:00:33.562 ****** changed: [localhost]
TASK [tensorflow : Get oneline version of git log for this checkout] *********** Thursday 19 January 2023 06:06:04 +0000 (0:00:00.372) 0:00:33.935 ****** changed: [localhost]
TASK [tensorflow : Show git log] *********************************************** Thursday 19 January 2023 06:06:05 +0000 (0:00:00.582) 0:00:34.517 ****** ok: [localhost] => git_log.stdout: 99c0dc2bb045596fd9b47224ec031ddbf9d17dfd Copy quantization_config for tpu_embedding_layer
TASK [tensorflow : Record git commit hash to variable] ************************* Thursday 19 January 2023 06:06:05 +0000 (0:00:00.043) 0:00:34.560 ****** ok: [localhost]
TASK [tensorflow : Save git commit hash to file] ******************************* Thursday 19 January 2023 06:06:05 +0000 (0:00:00.061) 0:00:34.621 ****** changed: [localhost]
TASK [Build Bazel] ************************************************************* Thursday 19 January 2023 06:06:06 +0000 (0:00:01.141) 0:00:35.763 ****** skipping: [localhost]
TASK [Build TensorFlow for each Python version] ******************************** Thursday 19 January 2023 06:06:06 +0000 (0:00:00.035) 0:00:35.798 ****** skipping: [localhost] => (item=3.8) skipping: [localhost] => (item=3.9) skipping: [localhost] => (item=3.10)
TASK [Build TensorFlow for each Python version (dual abi)] ********************* Thursday 19 January 2023 06:06:06 +0000 (0:00:00.072) 0:00:35.871 ******
TASK [tensorflow_in_container : Reset checkout to clean state] ***************** Thursday 19 January 2023 06:06:07 +0000 (0:00:00.451) 0:00:36.322 ****** changed: [localhost]
TASK [tensorflow_in_container : Remove TensorFlow git sources from previous build] *** Thursday 19 January 2023 06:06:07 +0000 (0:00:00.623) 0:00:36.946 ****** skipping: [localhost]
TASK [Fetch TensorFlow] ******************************************************** Thursday 19 January 2023 06:06:07 +0000 (0:00:00.035) 0:00:36.982 ****** skipping: [localhost]
TASK [(Python 3.8) patch TensorFlow tree] ************************************** Thursday 19 January 2023 06:06:07 +0000 (0:00:00.037) 0:00:37.019 ******
TASK [tensorflow : Create linklibs shell script] ******************************* Thursday 19 January 2023 06:06:07 +0000 (0:00:00.142) 0:00:37.162 ****** changed: [localhost]
TASK [tensorflow : Create readme file about unit tests] ************************ Thursday 19 January 2023 06:06:08 +0000 (0:00:00.691) 0:00:37.853 ****** changed: [localhost]
TASK [tensorflow : Add to the copied files] ************************************ Thursday 19 January 2023 06:06:09 +0000 (0:00:00.702) 0:00:38.556 ****** changed: [localhost]
TASK [tensorflow : Add to the long_description in setup.py] ******************** Thursday 19 January 2023 06:06:09 +0000 (0:00:00.559) 0:00:39.115 ****** changed: [localhost]
TASK [tensorflow : Use nightly package requirements] *************************** Thursday 19 January 2023 06:06:10 +0000 (0:00:00.549) 0:00:39.665 ****** changed: [localhost]
TASK [tensorflow_in_container : (Python 3.8) start build container] ************ Thursday 19 January 2023 06:06:10 +0000 (0:00:00.388) 0:00:40.053 ****** fatal: [localhost]: FAILED! => changed=false msg: 'Error removing container e3109365d4dd30b7d5015e7d1da981f87cb35c2ff7ad281bb0644614e424393a: UnixHTTPConnectionPool(host=''localhost'', port=None): Read timed out. (read timeout=60)'
PLAY RECAP ********************************************************************* localhost : ok=20 changed=13 unreachable=0 failed=1 skipped=7 rescued=0 ignored=0
Thursday 19 January 2023 06:07:29 +0000 (0:01:18.448) 0:01:58.502 ****** =============================================================================== tensorflow_in_container : (Python 3.8) start build container ----------- 78.45s tensorflow : Fetch TensorFlow from git --------------------------------- 24.43s tensorflow : Remove TensorFlow git sources from previous build ---------- 3.17s Gathering Facts --------------------------------------------------------- 2.62s Create cache dirs ------------------------------------------------------- 1.40s Cleanup after previous build -------------------------------------------- 1.28s tensorflow : Save git commit hash to file ------------------------------- 1.14s tensorflow : Create readme file about unit tests ------------------------ 0.70s tensorflow : Create linklibs shell script ------------------------------- 0.69s tensorflow_in_container : Reset checkout to clean state ----------------- 0.62s tensorflow : Get oneline version of git log for this checkout ----------- 0.58s tensorflow : Add to the copied files ------------------------------------ 0.56s tensorflow : Add to the long_description in setup.py -------------------- 0.55s Build TensorFlow for each Python version (dual abi) --------------------- 0.45s tensorflow : Use nightly package requirements --------------------------- 0.39s tensorflow : Set world write access to source --------------------------- 0.37s Go through each TensorFlow version -------------------------------------- 0.24s (Python 3.8) patch TensorFlow tree -------------------------------------- 0.14s tensorflow : Fetch TensorFlow ------------------------------------------- 0.10s tensorflow : Set TensorFlow and other package versions ------------------ 0.08s Build step 'Execute shell' marked build as failure