Migration from ES 2.x to ES 5.x Elasticsearch - elasticsearch-5

Question

We have created new ES cluster v5.x. We have added new backup repository for restoring (old ES 2.x).
We have restored from our snapshot and everything was fine. We have retention period for our ES snapshots.
We are using s3 storage and repository-s3 plugin for backups.
And while our retention period has deleted this snapshot we got an error:
{
"error": {
"root_cause": [
{
"type": "snapshot_missing_exception",
"reason": "[s3_repository:snapshot_201701040203/snapshot_201701040203] is missing"
}
],
"type": "snapshot_exception",
"reason": "[s3_repository:snapshot_201701040203/snapshot_201701040203] Snapshot could not be read",
"caused_by": {
"type": "snapshot_missing_exception",
"reason": "[s3_repository:snapshot_201701040203/snapshot_201701040203] is missing",
"caused_by": {
"type": "no_such_file_exception",
"reason": "Blob object [snap-snapshot_201701040203.dat] not found: The specified key does not exist. (Service: Amazon S3; Status Code: 404; Error Code: NoSuchKey; Request ID: DB308DF310809F58)"
}
}
},
"status": 500
}
Full log:
[2017-01-30T13:18:57,344][WARN ][r.suppressed ] path: /_snapshot/s3_repository/_all, params: {repository=s3_repository, snapshot=_all}
org.elasticsearch.snapshots.SnapshotException: [s3_repository:snapshot_201701040203/snapshot_201701040203] Snapshot could not be read
at org.elasticsearch.snapshots.SnapshotsService.snapshots(SnapshotsService.java:187) ~[elasticsearch-5.1.2.jar:5.1.2]
at org.elasticsearch.action.admin.cluster.snapshots.get.TransportGetSnapshotsAction.masterOperation(TransportGetSnapshotsAction.java:122) [elasticsearch-5.1.2.jar:5.1.2]
at org.elasticsearch.action.admin.cluster.snapshots.get.TransportGetSnapshotsAction.masterOperation(TransportGetSnapshotsAction.java:50) [elasticsearch-5.1.2.jar:5.1.2]
at org.elasticsearch.action.support.master.TransportMasterNodeAction.masterOperation(TransportMasterNodeAction.java:86) [elasticsearch-5.1.2.jar:5.1.2]
at org.elasticsearch.action.support.master.TransportMasterNodeAction$AsyncSingleAction$3.doRun(TransportMasterNodeAction.java:170) [elasticsearch-5.1.2.jar:5.1.2]
at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:527) [elasticsearch-5.1.2.jar:5.1.2]
at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) [elasticsearch-5.1.2.jar:5.1.2]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [?:1.8.0_111]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [?:1.8.0_111]
at java.lang.Thread.run(Thread.java:745) [?:1.8.0_111]
Caused by: org.elasticsearch.snapshots.SnapshotMissingException: [s3_repository:snapshot_201701040203/snapshot_201701040203] is missing
at org.elasticsearch.repositories.blobstore.BlobStoreRepository.getSnapshotInfo(BlobStoreRepository.java:566) ~[elasticsearch-5.1.2.jar:5.1.2]
at org.elasticsearch.snapshots.SnapshotsService.snapshots(SnapshotsService.java:182) ~[elasticsearch-5.1.2.jar:5.1.2]
... 9 more
Caused by: java.nio.file.NoSuchFileException: Blob object [snap-snapshot_201701040203.dat] not found: The specified key does not exist. (Service: Amazon S3; Status Code: 404; Error Code: NoSuchKey; Request ID: 38087D5B4A20B627)
at org.elasticsearch.cloud.aws.blobstore.S3BlobContainer.readBlob(S3BlobContainer.java:92) ~[?:?]
at org.elasticsearch.repositories.blobstore.ChecksumBlobStoreFormat.readBlob(ChecksumBlobStoreFormat.java:100) ~[elasticsearch-5.1.2.jar:5.1.2]
at org.elasticsearch.repositories.blobstore.BlobStoreFormat.read(BlobStoreFormat.java:89) ~[elasticsearch-5.1.2.jar:5.1.2]
at org.elasticsearch.repositories.blobstore.BlobStoreRepository.getSnapshotInfo(BlobStoreRepository.java:560) ~[elasticsearch-5.1.2.jar:5.1.2]
at org.elasticsearch.snapshots.SnapshotsService.snapshots(SnapshotsService.java:182) ~[elasticsearch-5.1.2.jar:5.1.2]
I have tried to remove this snapshot repository, remove all indeces. But if i will add this repository again, i am getting the same error.
How can i restore ES? Where ES takes information about old snapshot?
Best regards.

Answer 1

I ran into this issue today. My situation is slightly different, but maybe this will help you. It appears that ES 5 is maintaining an index of snapshots in the bucket where ES 2.x does not.
In my scenario, I'm transitioning from 2.x to 5.x. Yesterday, I restored a bunch of snapshots created by my 2.x environment onto the 5.x environment. Last night another snapshot was created by my 2.x environment. When I go to restore the snapshot, I get an error that the snapshot doesn't exist (even though it does).
If I rename two files in the s3 bucket, ES 5 will rebuild it's index and see the new snapshot. The two files are: index-0 and index.latest.
Hope that helps.

Answer 2

I got a similar issue and solved it following that advice: https://discuss.elastic.co/t/migration-from-es-2-x-to-es-5-x/73211/6 (in my case, deleting the index-0 file and re-registering the repository with my ES 5)