FileNotFoundError in temporary job location during Cloud Dataflow Shuffle operation - google-cloud-dataflow

Question

While running a batch job in Google Cloud Dataflow, I am experiencing an error during a particular step that uses Dataflow's service-based Shuffle feature. The error claims that a specific file is no longer present in the temporary job location I specified for this pipeline.
Here is the most relevant piece of the full stacktrace:
An exception was raised when trying to execute the workitem 2931621256965625980 : Traceback (most recent call last):
File "/usr/local/lib/python3.7/site-packages/apache_beam/io/gcp/gcsio.py", line 490, in __init__
metadata = self._get_object_metadata(self._get_request)
File "/usr/local/lib/python3.7/site-packages/apache_beam/utils/retry.py", line 206, in wrapper
return fun(*args, **kwargs)
File "/usr/local/lib/python3.7/site-packages/apache_beam/io/gcp/gcsio.py", line 513, in _get_object_metadata
return self._client.objects.Get(get_request)
File "/usr/local/lib/python3.7/site-packages/apache_beam/io/gcp/internal/clients/storage/storage_v1_client.py", line 1100, in Get
download=download)
File "/usr/local/lib/python3.7/site-packages/apitools/base/py/base_api.py", line 731, in _RunMethod
return self.ProcessHttpResponse(method_config, http_response, request)
File "/usr/local/lib/python3.7/site-packages/apitools/base/py/base_api.py", line 737, in ProcessHttpResponse
self.__ProcessHttpResponse(method_config, http_response, request))
File "/usr/local/lib/python3.7/site-packages/apitools/base/py/base_api.py", line 604, in __ProcessHttpResponse
http_response, method_config=method_config, request=request)
apitools.base.py.exceptions.HttpNotFoundError: HttpError accessing <https://www.googleapis.com/storage/v1/b/<CLOUD STORAGE PATH FOR TEMPORARY JOB FILES>%2F<DATAFLOW JOB NAME>.1571774420.011973%2Ftmp-626a66561e20e8b6-00000-of-00003.avro?alt=json>: response: <{'x-guploader-uploadid': 'AEnB2UrVuWRWrrcneEjgvuGSwYR82tBqDdVa727Ylo8tVW6ucnPdeNbE2A8DXf7mDYqKKP42NdJapXZLR1UbCjvJ8n7w2SOVTMGFsrcbywKD1K9yxMWez7k', 'content-type': 'application/json; charset=UTF-8', 'date': 'Tue, 22 Oct 2019 20:43:59 GMT', 'vary': 'Origin, X-Origin', 'cache-control': 'no-cache, no-store, max-age=0, must-revalidate', 'expires': 'Mon, 01 Jan 1990 00:00:00 GMT', 'pragma': 'no-cache', 'content-length': '473', 'server': 'UploadServer', 'status': '404'}>, content <{
"error": {
"code": 404,
"message": "No such object: <CLOUD STORAGE PATH FOR TEMPORARY JOB FILES>/<DATAFLOW JOB NAME>.1571774420.011973/tmp-626a66561e20e8b6-00000-of-00003.avro",
"errors": [
{
"message": "No such object: <CLOUD STORAGE PATH FOR TEMPORARY JOB FILES>/<DATAFLOW JOB NAME>.1571774420.011973/tmp-626a66561e20e8b6-00000-of-00003.avro",
"domain": "global",
"reason": "notFound"
}
]
}
}
Any advice on how to resolve this error would be appreciated. It seems to me that Dataflow created a temporary file during the course of the shuffle. That file must have been deleted shortly afterward, but sometime later Dataflow is attempting to access that file and causing this error.

Answer 1

After a lot of trial and error, I never really found an answer to this problem. The root cause is Dataflow's Shuffle service--it seems that if a particular shuffle step is very very expensive, these types of intermittent connection issues eventually cause the job to error out.
I ultimately solved this problem by re-working the data set to cut the amount of shuffling required by about half. The Shuffle service now runs reliably for me.
Cloud Dataflow shuffle is still an experimental feature--I am hoping that this kind of instability disappears as it matures.

Answer 2

You will need to add stagingLocation or gcpTempLocation to resolve this error.
You can check here [1] for further details.
1 - https://cloud.google.com/dataflow/docs/guides/specifying-exec-params#configuring-pipelineoptions-for-execution-on-the-cloud-dataflow-service