Loading/initializing of salt is not an atomic operation

[pavel-jares-bcm]

During implementation of #4408 was found an issue with initialization of salt:

api-layer/zaas-service/src/main/java/org/zowe/apiml/zaas/security/service/token/ApimlAccessTokenProvider.java

Lines 154 to 166 in 344f6ff

	private String initializeSalt() throws CachingServiceClientException, SecureTokenInitializationException {
	String localSalt;
	try {
	CachingServiceClient.KeyValue keyValue = cachingServiceClient.read("salt");
	localSalt = keyValue.getValue();
	} catch (CachingServiceClientException | StorageException e) {
	byte[] newSalt = generateSalt();
	storeSalt(newSalt);
	localSalt = new String(newSalt);
	}
	return localSalt;
	}

There are two potential issues:

• if clusters are not in sync, multiple salt could be generated
• if two or more threads trying to initialize salt at the moment one has to fail.

It should be solved by using lock, but in this case the lock should be available through HTTP.

The best solution should be:

1. try to read the salt
2. if salt does not exist

• create a lock
• read salt
• if salt still doesn't exist create a new one
• unlock