Issues running these models on TPU

[juanlp]

Does anyone else have any problem running these models on TPU? How do we make the library work in TPU environment? Thanks.

[juanlp]

InvalidArgumentError Traceback (most recent call last)
in ()
6 callbacks=callbacks_list,
7 # steps_per_epoch=STEPS_PER_EPOCH,
----> 8 validation_data=valid_dataset)

1 frames
/usr/local/lib/python3.7/dist-packages/keras/utils/traceback_utils.py in error_handler(*args, **kwargs)
65 except Exception as e: # pylint: disable=broad-except
66 filtered_tb = _process_traceback_frames(e.traceback)
---> 67 raise e.with_traceback(filtered_tb) from None
68 finally:
69 del filtered_tb

/usr/local/lib/python3.7/dist-packages/tensorflow/python/eager/context.py in sync_executors(self)
692 """
693 if self._context_handle:
--> 694 pywrap_tfe.TFE_ContextSyncExecutors(self._context_handle)
695 else:
696 raise ValueError("Context is not initialized.")

InvalidArgumentError: 9 root error(s) found.
(0) INVALID_ARGUMENT: {{function_node __inference_train_function_56513}} Squeeze op does not support bounded dynamic dimensions. Input shape: element_type: F32
dimensions: 8
dimensions: 256
dimensions: 256
dimensions: 1
layout {
minor_to_major: 3
minor_to_major: 2
minor_to_major: 1
minor_to_major: 0
format: DENSE
}
is_dynamic_dimension: true
is_dynamic_dimension: false
is_dynamic_dimension: false
is_dynamic_dimension: false

 [[{{node hybrid_loss/Squeeze_1}}]]
 [[TPUReplicate/_compile/_14331551586166216331/_4]]

(1) INVALID_ARGUMENT: {{function_node __inference_train_function_56513}} Squeeze op does not support bounded dynamic dimensions. Input shape: element_type: F32
dimensions: 8
dimensions: 256
dimensions: 256
dimensions: 1
layout {
minor_to_major: 3
minor_to_major: 2
minor_to_major: 1
minor_to_major: 0
format: DENSE
}
is_dynamic_dimension: true
is_dynamic_dimension: false
is_dynamic_dimension: false
is_dynamic_dimension: false

 [[{{node hybrid_loss/Squeeze_1}}]]
 [[TPUReplicate/_compile/_14331551586166216331/_4]]
 [[tpu_compile_succeeded_assert/_11746012695527891760/_5/_303]]

(2) INVALID_ARGUMENT: {{function_node __inference_train_function_56513}} Squeeze op does not support bounded dynamic dimensions. Input shape: element_type: F32
dimensions: 8
dimensions: 256
dimensions: 256
dimensions: 1
layout {
minor_to_major: 3
minor_to_major: 2
minor_to_major: 1
minor_to_major: 0
format: DENSE
}
is_dynamic_dimension: true
is_dynamic_dimension: false
is_dynamic_dimension: false
is_dynamic_dimension: false

 [[{{node hybrid_loss/Squeeze_1}}]]
 [[TPUReplicate/_compile/_14331551586166216331/_4]]
 [[tpu_compile_succeeded_assert/_11746012695527891760/_5/_287]]

(3) INVALID_ARGUMENT: {{function_node __inference_train_function_56513}} Squeeze op does not support bounded dynamic dimensions. Input shape: element_type: F32
dimensions: 8
dimensions: 256
dimensions: 256
dimensions: 1
layout {
minor_to_major: 3
minor_to_major: 2
minor_to_major: 1
minor_to_major: 0
format: DENSE
}
is_dynamic_dimension: true
is_dynamic_dimension: false
is_dynamic_dimension: false
is_dynamic_dimension: false

 [[{{node hybrid_loss/Squeeze_1}}]]
 [[TPUReplicate/_compile/_14331551586166216331/_4]]
 [[tpu_compile_succeeded_assert/_11746012695527891760/_5/_271]]

(4) INVALID_ARGUMENT: {{function_node __inference_train_function_56513}} Squeeze op does not support bounded dynamic dimensions. Input shape: element_type: F32
dimensions: 8
dimensions: 256
dimensions: 256
dimensions: 1
layout {
minor_to_major: 3
minor_to_major: 2
minor_to_major: 1
minor_to_major: 0
format: DENSE
}
is_dynamic_dimension: true
is_dynamic_dimension: false
is_dynamic_dimension: false
is_dynamic_dimension: false

 [[{{node hybrid_loss/Squeeze_1}}]]
 [[TPUReplicate/_compile/_14331551586166216331/_4]]
 [[tpu_compile_succeeded_assert/_11746012695527891760/_5/_255]]

(5) INVALID_ARGUMENT: {{function_node __inference_train_function_56513}} Squeeze op does not support bounded dynamic dimensions ... [truncated]