See https://ci.linaro.org/job/ldcg-python-manylinux-tensorflow-onednn-nightly/349/display/redirect
Changes:
------------------------------------------ [...truncated 16.71 KB...]
TASK [tensorflow : Remove TensorFlow-oneDNN-ACL git sources from previous build] *** Friday 24 February 2023 01:56:10 +0000 (0:00:00.044) 0:00:05.786 ******* changed: [localhost]
TASK [tensorflow : Fetch TensorFlow] ******************************************* Friday 24 February 2023 01:56:13 +0000 (0:00:03.453) 0:00:09.239 ******* included: https://ci.linaro.org/job/ldcg-python-manylinux-tensorflow-onednn-nightly/ws/configs/ldcg-python-manylinux-tensorflow/ansible/roles/tensorflow/tasks/fetch.yml for localhost
TASK [tensorflow : Fetch TensorFlow-oneDNN-ACL git source] ********************* Friday 24 February 2023 01:56:13 +0000 (0:00:00.115) 0:00:09.354 ******* skipping: [localhost]
TASK [tensorflow : Unpack TensorFlow-oneDNN-ACL git source] ******************** Friday 24 February 2023 01:56:13 +0000 (0:00:00.040) 0:00:09.395 ******* skipping: [localhost]
TASK [tensorflow : Rename source dir for oneDNN build] ************************* Friday 24 February 2023 01:56:13 +0000 (0:00:00.037) 0:00:09.432 ******* skipping: [localhost]
TASK [tensorflow : Fetch TensorFlow-oneDNN-ACL from git] *********************** Friday 24 February 2023 01:56:13 +0000 (0:00:00.038) 0:00:09.471 ******* changed: [localhost]
TASK [tensorflow : Fetch TensorFlow-oneDNN-ACL from git at commit] ************* Friday 24 February 2023 01:56:35 +0000 (0:00:21.968) 0:00:31.440 ******* skipping: [localhost]
TASK [tensorflow : Set world write access to source] *************************** Friday 24 February 2023 01:56:35 +0000 (0:00:00.036) 0:00:31.477 ******* changed: [localhost]
TASK [tensorflow : Get oneline version of git log for this checkout] *********** Friday 24 February 2023 01:56:36 +0000 (0:00:00.374) 0:00:31.852 ******* changed: [localhost]
TASK [tensorflow : Show git log] *********************************************** Friday 24 February 2023 01:56:36 +0000 (0:00:00.585) 0:00:32.438 ******* ok: [localhost] => git_log.stdout: 3f43b4eb7c183b3ce077dbee5b06c1b1ac201d72 Support quantizing models that use asset files to initialize resources.
TASK [tensorflow : Record git commit hash to variable] ************************* Friday 24 February 2023 01:56:36 +0000 (0:00:00.044) 0:00:32.482 ******* ok: [localhost]
TASK [tensorflow : Save git commit hash to file] ******************************* Friday 24 February 2023 01:56:36 +0000 (0:00:00.062) 0:00:32.544 ******* changed: [localhost]
TASK [Build Bazel] ************************************************************* Friday 24 February 2023 01:56:38 +0000 (0:00:01.146) 0:00:33.691 ******* skipping: [localhost]
TASK [Build TensorFlow for each Python version] ******************************** Friday 24 February 2023 01:56:38 +0000 (0:00:00.059) 0:00:33.751 ******* skipping: [localhost] => (item=3.8) skipping: [localhost] => (item=3.9) skipping: [localhost] => (item=3.10) skipping: [localhost] => (item=3.11)
TASK [Build TensorFlow for each Python version (dual abi)] ********************* Friday 24 February 2023 01:56:38 +0000 (0:00:00.055) 0:00:33.807 *******
TASK [tensorflow_in_container : Reset checkout to clean state] ***************** Friday 24 February 2023 01:56:38 +0000 (0:00:00.632) 0:00:34.439 ******* changed: [localhost]
TASK [tensorflow_in_container : Remove TensorFlow-oneDNN-ACL git sources from previous build] *** Friday 24 February 2023 01:56:39 +0000 (0:00:00.828) 0:00:35.268 ******* skipping: [localhost]
TASK [Fetch TensorFlow] ******************************************************** Friday 24 February 2023 01:56:39 +0000 (0:00:00.037) 0:00:35.306 ******* skipping: [localhost]
TASK [(Python 3.8) patch TensorFlow tree] ************************************** Friday 24 February 2023 01:56:39 +0000 (0:00:00.037) 0:00:35.344 *******
TASK [tensorflow : Create linklibs shell script] ******************************* Friday 24 February 2023 01:56:39 +0000 (0:00:00.172) 0:00:35.516 ******* changed: [localhost]
TASK [tensorflow : Create readme file about unit tests] ************************ Friday 24 February 2023 01:56:40 +0000 (0:00:00.692) 0:00:36.209 ******* changed: [localhost]
TASK [tensorflow : Add to the copied files] ************************************ Friday 24 February 2023 01:56:41 +0000 (0:00:00.700) 0:00:36.909 ******* changed: [localhost]
TASK [tensorflow : Add to the long_description in setup.py] ******************** Friday 24 February 2023 01:56:41 +0000 (0:00:00.596) 0:00:37.506 ******* changed: [localhost]
TASK [tensorflow : Use nightly package requirements] *************************** Friday 24 February 2023 01:56:42 +0000 (0:00:00.581) 0:00:38.088 ******* changed: [localhost]
TASK [tensorflow_in_container : (Python 3.8) start build container] ************ Friday 24 February 2023 01:56:42 +0000 (0:00:00.392) 0:00:38.480 ******* changed: [localhost]
TASK [tensorflow_in_container : (Python 3.8) create buildslave bazel cache dir] *** Friday 24 February 2023 01:57:18 +0000 (0:00:35.153) 0:01:13.634 ******* changed: [localhost]
TASK [tensorflow_in_container : (Python 3.8) create link for bazel cache] ****** Friday 24 February 2023 01:57:19 +0000 (0:00:01.227) 0:01:14.862 ******* changed: [localhost]
TASK [tensorflow_in_container : (Python 3.8) create link for bazelisk cache] *** Friday 24 February 2023 01:57:20 +0000 (0:00:00.814) 0:01:15.677 ******* changed: [localhost]
TASK [tensorflow_in_container : (Python 3.8) create link for pip cache] ******** Friday 24 February 2023 01:57:20 +0000 (0:00:00.822) 0:01:16.499 ******* changed: [localhost]
TASK [tensorflow_in_container : (Python 3.8) copy upper constraints] *********** Friday 24 February 2023 01:57:21 +0000 (0:00:00.788) 0:01:17.288 ******* changed: [localhost]
TASK [tensorflow_in_container : (Python 3.8) install Python packages] ********** Friday 24 February 2023 01:57:22 +0000 (0:00:00.718) 0:01:18.006 ******* changed: [localhost]
TASK [tensorflow_in_container : (Python 3.8) build TensorFlow-oneDNN-ACL git] *** Friday 24 February 2023 01:57:54 +0000 (0:00:31.677) 0:01:49.683 ******* fatal: [localhost]: FAILED! => changed=true rc: 1 stderr: |- Extracting Bazel installation... Starting local Bazel server and connecting to it... INFO: Invocation ID: 8c55d19e-948a-4502-964f-50a104c0c7bf INFO: Options provided by the client: Inherited 'common' options: --isatty=0 --terminal_columns=80 INFO: Reading rc options for 'build' from /tf/tensorflow/.bazelrc: Inherited 'common' options: --experimental_repo_remote_exec INFO: Reading rc options for 'build' from /tf/tensorflow/.bazelrc: 'build' options: --define framework_shared_object=true --define tsl_protobuf_header_only=true --define=use_fast_cpp_protos=true --define=allow_oversize_protos=true --spawn_strategy=standalone -c opt --announce_rc --define=grpc_no_ares=true --noincompatible_remove_legacy_whole_archive --enable_platform_specific_config --define=with_xla_support=true --config=short_logs --config=v2 --define=no_aws_support=true --define=no_hdfs_support=true --experimental_cc_shared_library --experimental_link_static_libraries_once=false --incompatible_enforce_config_setting_visibility --deleted_packages=tensorflow/compiler/mlir/tfrt,tensorflow/compiler/mlir/tfrt/benchmarks,tensorflow/compiler/mlir/tfrt/jit/python_binding,tensorflow/compiler/mlir/tfrt/jit/transforms,tensorflow/compiler/mlir/tfrt/python_tests,tensorflow/compiler/mlir/tfrt/tests,tensorflow/compiler/mlir/tfrt/tests/ir,tensorflow/compiler/mlir/tfrt/tests/analysis,tensorflow/compiler/mlir/tfrt/tests/jit,tensorflow/compiler/mlir/tfrt/tests/lhlo_to_tfrt,tensorflow/compiler/mlir/tfrt/tests/lhlo_to_jitrt,tensorflow/compiler/mlir/tfrt/tests/tf_to_corert,tensorflow/compiler/mlir/tfrt/tests/tf_to_tfrt_data,tensorflow/compiler/mlir/tfrt/tests/saved_model,tensorflow/compiler/mlir/tfrt/transforms/lhlo_gpu_to_tfrt_gpu,tensorflow/core/runtime_fallback,tensorflow/core/runtime_fallback/conversion,tensorflow/core/runtime_fallback/kernel,tensorflow/core/runtime_fallback/opdefs,tensorflow/core/runtime_fallback/runtime,tensorflow/core/runtime_fallback/util,tensorflow/core/tfrt/eager,tensorflow/core/tfrt/eager/backends/cpu,tensorflow/core/tfrt/eager/backends/gpu,tensorflow/core/tfrt/eager/core_runtime,tensorflow/core/tfrt/eager/cpp_tests/core_runtime,tensorflow/core/tfrt/gpu,tensorflow/core/tfrt/run_handler_thread_pool,tensorflow/core/tfrt/runtime,tensorflow/core/tfrt/saved_model,tensorflow/core/tfrt/graph_executor,tensorflow/core/tfrt/saved_model/tests,tensorflow/core/tfrt/tpu,tensorflow/core/tfrt/utils INFO: Reading rc options for 'build' from /usertools/aarch64.bazelrc: 'build' options: --action_env=CACHEBUSTER=20220325 --action_env PYTHON_BIN_PATH=/usr/bin/python3 --action_env PYTHON_LIB_PATH=/usr/lib/tf_python --python_path=/usr/bin/python3 --define=tf_api_version=2 --action_env=TF2_BEHAVIOR=1 --distinct_host_configuration=false --profile=/tf/pkg/profile.json.gz --crosstool_top=@ml2014_aarch64_config_aarch64//crosstool:toolchain --copt=-mtune=generic --copt=-march=armv8-a --copt=-O3 INFO: Found applicable config definition build:short_logs in file /tf/tensorflow/.bazelrc: --output_filter=DONT_MATCH_ANYTHING INFO: Found applicable config definition build:v2 in file /tf/tensorflow/.bazelrc: --define=tf_api_version=2 --action_env=TF2_BEHAVIOR=1 INFO: Found applicable config definition build:sigbuild_local_cache in file /usertools/aarch64.bazelrc: --disk_cache=/tf/cache INFO: Found applicable config definition build:ambe in file /usertools/aarch64.bazelrc: --distinct_host_configuration=false --flaky_test_attempts=3 --action_env=PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/local/go/bin --host_crosstool_top=@ml2014_aarch64_config_aarch64//crosstool:toolchain --crosstool_top=@ml2014_aarch64_config_aarch64//crosstool:toolchain --extra_toolchains=@ml2014_aarch64_config_aarch64//crosstool:toolchain-linux-aarch64 --extra_execution_platforms=@ml2014_aarch64_config_aarch64_platform//:platform --host_platform=@ml2014_aarch64_config_aarch64_platform//:platform --platforms=@ml2014_aarch64_config_aarch64_platform//:platform --repo_env=TF_PYTHON_CONFIG_REPO=@ml2014_aarch64_config_python INFO: Found applicable config definition build:mkl_aarch64_threadpool in file /tf/tensorflow/.bazelrc: --define=build_with_mkl_aarch64=true -c opt INFO: Found applicable config definition build:linux in file /tf/tensorflow/.bazelrc: --host_copt=-w --copt=-Wno-all --copt=-Wno-extra --copt=-Wno-deprecated --copt=-Wno-deprecated-declarations --copt=-Wno-ignored-attributes --copt=-Wno-array-bounds --copt=-Wunused-result --copt=-Werror=unused-result --copt=-Wswitch --copt=-Werror=switch --copt=-Wno-error=unused-but-set-variable --define=PREFIX=/usr --define=LIBDIR=$(PREFIX)/lib --define=INCLUDEDIR=$(PREFIX)/include --define=PROTOBUF_INCLUDE_PATH=$(PREFIX)/include --cxxopt=-std=c++17 --host_cxxopt=-std=c++17 --config=dynamic_kernels --distinct_host_configuration=false --experimental_guard_against_concurrent_changes INFO: Found applicable config definition build:dynamic_kernels in file /tf/tensorflow/.bazelrc: --define=dynamic_loaded_kernels=true --copt=-DAUTOLOAD_DYNAMIC_KERNELS Loading: Loading: 0 packages loaded Loading: 0 packages loaded Loading: 0 packages loaded Loading: 0 packages loaded WARNING: Download from https://storage.googleapis.com/mirror.tensorflow.org/github.com/tensorflow/r... failed: class java.io.FileNotFoundException GET returned 404 Not Found Loading: 0 packages loaded Loading: 0 packages loaded Loading: 0 packages loaded Loading: 0 packages loaded Loading: 0 packages loaded Loading: 0 packages loaded Loading: 0 packages loaded Loading: 0 packages loaded Loading: 0 packages loaded Loading: 0 packages loaded Loading: 0 packages loaded Loading: 0 packages loaded WARNING: Download from https://storage.googleapis.com/mirror.tensorflow.org/github.com/llvm/llvm-pr... failed: class java.io.FileNotFoundException GET returned 404 Not Found Loading: 0 packages loaded Analyzing: target //tensorflow/tools/pip_package:build_pip_package (1 packages loaded, 0 targets configured) WARNING: Download from https://storage.googleapis.com/mirror.tensorflow.org/github.com/openxla/stab... failed: class java.io.FileNotFoundException GET returned 404 Not Found Analyzing: target //tensorflow/tools/pip_package:build_pip_package (289 packages loaded, 4306 targets configured) INFO: Analyzed target //tensorflow/tools/pip_package:build_pip_package (593 packages loaded, 37030 targets configured). INFO: Found 1 target... [0 / 183] [Prepa] Creating source manifest for //tensorflow/lite/python:tflite_convert ... (5 actions, 0 running) [12,136 / 23,108] Compiling tensorflow/lite/delegates/xnnpack/quantization_util.cc; 11s local, remote-cache ... (111 actions, 110 running) [12,210 / 23,108] Compiling tensorflow/compiler/xla/literal_comparison.cc; 21s local, remote-cache ... (111 actions, 110 running) [12,294 / 23,108] Compiling tensorflow/compiler/xla/literal.cc; 36s local, remote-cache ... (111 actions, 110 running) [12,387 / 23,108] Compiling tensorflow/compiler/xla/literal.cc; 52s local, remote-cache ... (112 actions running) [12,622 / 23,108] Compiling tensorflow/compiler/xla/hlo/ir/hlo_instruction.cc; 38s local, remote-cache ... (112 actions running) [12,873 / 23,108] Compiling tensorflow/compiler/xla/service/memory_space_assignment.cc; 51s local, remote-cache ... (112 actions running) [13,013 / 23,108] Compiling tensorflow/compiler/xla/service/algebraic_simplifier.cc; 63s local, remote-cache ... (112 actions running) [13,169 / 23,108] Compiling tensorflow/python/client/tf_session_wrapper.cc; 55s local, remote-cache ... (112 actions running) [13,383 / 23,108] Compiling tensorflow/core/kernels/rnn/lstm_ops.cc; 66s local, remote-cache ... (112 actions running) [13,695 / 23,108] Compiling tensorflow/core/kernels/padding_fifo_queue.cc; 52s local, remote-cache ... (112 actions running) [13,919 / 23,108] Compiling tensorflow/core/kernels/lrn_op.cc; 66s local, remote-cache ... (112 actions running) [14,212 / 23,108] Compiling tensorflow/core/kernels/image/mirror_pad_op_cpu_impl_5.cc; 53s local, remote-cache ... (112 actions running) [14,634 / 23,108] Compiling tensorflow/core/kernels/training_ops.cc; 64s local, remote-cache ... (112 actions running) [17,905 / 24,452] Compiling tensorflow/core/kernels/training_ops.cc; 131s local, remote-cache ... (112 actions running) [18,516 / 24,452] Compiling mlir/lib/Dialect/PDLInterp/IR/PDLInterp.cpp; 42s local, remote-cache ... (112 actions running) [18,910 / 24,452] Compiling tensorflow/compiler/mlir/tensorflow/ir/tf_ops.cc; 66s local, remote-cache ... (112 actions running) [19,369 / 24,452] Compiling tensorflow/compiler/mlir/tensorflow/ir/tf_ops.cc; 168s local, remote-cache ... (111 actions, 110 running) [19,627 / 24,452] Compiling tensorflow/compiler/mlir/tensorflow/ir/tf_ops.cc; 285s local, remote-cache ... (112 actions running) [19,805 / 24,452] Compiling tensorflow/core/kernels/cwise_op_select.cc; 181s local, remote-cache ... (112 actions running) [20,435 / 24,452] Compiling tensorflow/core/kernels/matmul_op_real.cc; 240s local, remote-cache ... (112 actions running) [21,299 / 24,452] Compiling tensorflow/core/kernels/matmul_op_real.cc; 418s local, remote-cache ... (112 actions running) [22,060 / 24,452] Compiling tensorflow/compiler/xla/mlir_hlo/mhlo/IR/hlo_ops.cc; 135s local, remote-cache ... (112 actions running) [22,817 / 24,452] Compiling tensorflow/dtensor/mlir/expansions/iterator_spmd_expander.cc; 40s local, remote-cache ... (112 actions running) [24,042 / 24,484] Compiling tensorflow/dtensor/cc/dtensor_device.cc; 60s local, remote-cache ... (111 actions, 110 running) [24,592 / 24,659] Linking tensorflow/libtensorflow_cc.so.2.13.0; 161s local, remote-cache ERROR: /tf/tensorflow/tensorflow/BUILD:1635:19: Executing genrule //tensorflow:tf_python_api_gen_v2 failed: (Exit 1): bash failed: error executing command /bin/bash -c ... (remaining 1 argument skipped) Traceback (most recent call last): File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/create_tensorflow.python_api_tf_python_api_gen_v2.runfiles/org_tensorflow/tensorflow/python/tools/api/generator/create_python_api.py", line 22, in <module> from tensorflow.python.tools.api.generator import doc_srcs File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/create_tensorflow.python_api_tf_python_api_gen_v2.runfiles/org_tensorflow/tensorflow/python/__init__.py", line 37, in <module> from tensorflow.python.eager import context File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/create_tensorflow.python_api_tf_python_api_gen_v2.runfiles/org_tensorflow/tensorflow/python/eager/context.py", line 28, in <module> from tensorflow.core.framework import function_pb2 File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/create_tensorflow.python_api_tf_python_api_gen_v2.runfiles/org_tensorflow/tensorflow/core/framework/function_pb2.py", line 5, in <module> from google.protobuf.internal import builder as _builder ModuleNotFoundError: No module named 'google.protobuf' Target //tensorflow/tools/pip_package:build_pip_package failed to build Use --verbose_failures to see the command lines of failed build steps. ERROR: /tf/tensorflow/tensorflow/python/tools/BUILD:282:10 Middleman _middlemen/tensorflow_Spython_Stools_Sprint_Uselective_Uregistration_Uheader-runfiles failed: (Exit 1): bash failed: error executing command /bin/bash -c ... (remaining 1 argument skipped) INFO: Elapsed time: 2802.608s, Critical Path: 1106.80s INFO: 24658 processes: 11508 disk cache hit, 5600 internal, 7550 local. FAILED: Build did NOT complete successfully FAILED: Build did NOT complete successfully stderr_lines: <omitted> stdout: '' stdout_lines: <omitted>
PLAY RECAP ********************************************************************* localhost : ok=27 changed=20 unreachable=0 failed=1 skipped=8 rescued=0 ignored=0
Friday 24 February 2023 02:44:37 +0000 (0:46:43.868) 0:48:33.552 ******* =============================================================================== tensorflow_in_container : (Python 3.8) build TensorFlow-oneDNN-ACL git 2803.87s tensorflow_in_container : (Python 3.8) start build container ----------- 35.15s tensorflow_in_container : (Python 3.8) install Python packages --------- 31.68s tensorflow : Fetch TensorFlow-oneDNN-ACL from git ---------------------- 21.97s tensorflow : Remove TensorFlow-oneDNN-ACL git sources from previous build --- 3.45s Gathering Facts --------------------------------------------------------- 2.63s Create cache dirs ------------------------------------------------------- 1.40s Cleanup after previous build -------------------------------------------- 1.28s tensorflow_in_container : (Python 3.8) create buildslave bazel cache dir --- 1.23s tensorflow : Save git commit hash to file ------------------------------- 1.15s tensorflow_in_container : Reset checkout to clean state ----------------- 0.83s tensorflow_in_container : (Python 3.8) create link for bazelisk cache --- 0.82s tensorflow_in_container : (Python 3.8) create link for bazel cache ------ 0.81s tensorflow_in_container : (Python 3.8) create link for pip cache -------- 0.79s tensorflow_in_container : (Python 3.8) copy upper constraints ----------- 0.72s tensorflow : Create readme file about unit tests ------------------------ 0.70s tensorflow : Create linklibs shell script ------------------------------- 0.69s Build TensorFlow for each Python version (dual abi) --------------------- 0.63s tensorflow : Add to the copied files ------------------------------------ 0.60s tensorflow : Get oneline version of git log for this checkout ----------- 0.59s Build step 'Execute shell' marked build as failure