We're getting an intermittent AccessDeniedException when creating snapshots on our cluster. As far as I can see all permissions are correct and config is the same across all servers.

ES Version: 7.6.0

The /home/shopfiles directory is a shared NFS mount across all servers.

The exception:

RepositoryException[[shopfiles] cannot create blob store]; nested: AccessControlException[access denied ("java.io.FilePermission" "/home/shopfiles/private/elasticsearch" "read")];
	at org.elasticsearch.repositories.blobstore.BlobStoreRepository.blobStore(BlobStoreRepository.java:424)
	at org.elasticsearch.repositories.blobstore.BlobStoreRepository.shardContainer(BlobStoreRepository.java:987)
	at org.elasticsearch.repositories.blobstore.BlobStoreRepository.shardContainer(BlobStoreRepository.java:983)
	at org.elasticsearch.repositories.blobstore.BlobStoreRepository.snapshotShard(BlobStoreRepository.java:1468)
	at org.elasticsearch.snapshots.SnapshotShardsService.snapshot(SnapshotShardsService.java:340)
	at org.elasticsearch.snapshots.SnapshotShardsService.lambda$startNewShards$1(SnapshotShardsService.java:286)
	at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:633)
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
	at java.base/java.lang.Thread.run(Thread.java:830)
	Caused by: java.security.AccessControlException: access denied ("java.io.FilePermission" "/home/shopfiles/private/elasticsearch" "read")
	at java.base/java.security.AccessControlContext.checkPermission(AccessControlContext.java:472)
	at java.base/java.security.AccessController.checkPermission(AccessController.java:1036)
	at java.base/java.lang.SecurityManager.checkPermission(SecurityManager.java:408)
	at java.base/java.lang.SecurityManager.checkRead(SecurityManager.java:747)
	at java.base/sun.nio.fs.UnixPath.checkRead(UnixPath.java:818)
	at java.base/sun.nio.fs.UnixFileSystemProvider.checkAccess(UnixFileSystemProvider.java:295)
	at java.base/java.nio.file.Files.createDirectories(Files.java:765)
	at org.elasticsearch.common.blobstore.fs.FsBlobStore.<init>(FsBlobStore.java:46)
	at org.elasticsearch.repositories.fs.FsRepository.createBlobStore(FsRepository.java:119)
	at org.elasticsearch.repositories.blobstore.BlobStoreRepository.blobStore(BlobStoreRepository.java:420)
	... 9 more
Config on all servers:
root@es1 /e/elasticsearch# cat /etc/elasticsearch/elasticsearch.yml | grep path.repo
path.repo: ["/home/shopfiles/private/elasticsearch/snapshots"]
Listing directory mentioned in exception as Elasticsearch user:
root@es1 /e/elasticsearch# sudo -u elasticsearch -H ls -al /home/shopfiles/private/elasticsearch
total 32
drwxrwxr-x 8 myuser    myuser    4096 Dec 19  2018 .
drwxr-xr-x 5 myuser    myuser    4096 Jan 31 21:49 ..
drwxrwxr-x 3 elasticsearch elasticsearch 4096 Mar  3 10:15 snapshots
I have checked the UID / GID of Elasticsearch on all nodes, they are the same.
root@es5 /e/elasticsearch# cat /etc/passwd | grep elasticsearch
elasticsearch:x:113:116::/nonexistent:/bin/false
Any ideas?
              
As an update to this, I couldn't make any progress so I tried deleting the repository from Elasticsearch.
When attempting to recreate it with the same settings, I had the same error. For the sake of thoroughness I tried chmoding the parent directory to 777. After this I was able to recreate the repository.
I then changed the perms back to 775 and it still worked!? Something funky going on internally here perhaps?
Before:
drwxrwxr-x 8 myuser    myuser    4096 Dec 19  2018 ./
drwxr-xr-x 5 myuser    myuser    4096 Jan 31 21:49 ../
drwxrwxr-x 3 elasticsearch elasticsearch 4096 Mar  3 10:15 snapshots
PUT http://xxxxxxx:9200/_snapshot/shopfiles
    "type": "fs",
    "settings": {
        "location": "/home/shopfiles/private/elasticsearch/snapshots",
        "compress": true
  "error": {
    "root_cause": [
        "type": "repository_exception",
        "reason": "[shopfiles] cannot create blob store"
    "type": "repository_exception",
    "reason": "[shopfiles] cannot create blob store",
    "caused_by": {
      "type": "security_exception",
      "reason": "access denied (\"java.io.FilePermission\" \"/home/shopfiles/private/elasticsearch\" \"read\")"
  "status": 500
777 Test:
chmod 777 /home/shopfiles/private/elasticsearch
drwxrwxrwx 8 myuser    myuser    4096 Dec 19  2018 ./
drwxr-xr-x 5 myuser    myuser    4096 Jan 31 21:49 ../
drwxrwxr-x 3 elasticsearch elasticsearch 4096 Mar  9 11:10 snapshots/
PUT http://xxxxxxx:9200/_snapshot/shopfiles
    "type": "fs",
    "settings": {
        "location": "/home/shopfiles/private/elasticsearch/snapshots",
        "compress": true
  "acknowledged": true
I thought this was really odd, so I deleted the repository again and gave it another go with the right permissions:
After:
chmod 775 /home/shopfiles/private/elasticsearch
drwxrwxrwx 8 myuser    myuser    4096 Dec 19  2018 ./
drwxr-xr-x 5 myuser    myuser    4096 Jan 31 21:49 ../
drwxrwxr-x 3 elasticsearch elasticsearch 4096 Mar  9 11:10 snapshots/
PUT http://xxxxxxx:9200/_snapshot/shopfiles
    "type": "fs",
    "settings": {
        "location": "/home/shopfiles/private/elasticsearch/snapshots",
        "compress": true
  "acknowledged": true
What!??