Azure Data Lake

Azure Data Lake Gen 2

In order to use Azure DataLake Gen 2, reference FluentStorage.Azure.Blobs package first.

Gen 2 provider is 100% compatible with hierarchical namespaces. When you use blob path, the first part of the path is filesystem name, i.e. store.WriteTextAsync("filesystem/folder/subfolder/.../file.extension. Apparently you cannot create files in the root folder, they always need to be prefixed with filesystem name.

If filesystem doesn't exist, we will try to create it for you, if the account provided has enough permissions to do so.

Connect to Azure Data Lake Gen 2

You can authenticate in the ways described below. To use connection strings, don't forget to call StorageFactory.Modules.UseAzureDataLake() somewhere when your program starts.

Using Shared Key Authentication

IStore store = AzureDataLakeStorage.FromSharedKey(
   accountName,
   sharedKey);

or

IStore store = StorageFactory.FromConnectionString(
   "azure.datalake.gen2://account=...;key=...");

Using Service Principal

IStore store = AzureDataLakeStorage.FromAzureAd(
   accountName,
   tenantId,
   principalId,
   principalSecret);

or

IStore store = StorageFactory.FromConnectionString(
   "azure.datalake.gen2://account=...;tenantId=...;principalId=...;principalSecret=...");

Using Managed Service Identity

IStore store = AzureDataLakeStorage.FromMsi(
   accountName);

or

IStore store = StorageFactory.FromConnectionString(
   "azure.datalake.gen2://account=...;msi");

Perform storage operations

Use our simple polycloud API to perform file and object manipulation operations.

Permissions Management

ADLS Gen 2 supports RBAC and POSIX like permissions on both file and folder level. FluentStorage fully supports permissions management on those and exposes simplified easy-to-use API to drive them.

Because permission management is ADLS Gen 2 specific feature, you cannot use IStore interface, however you can cast it to IAzureDataLakeGen2BlobStorage which in turn implements IStore as well.

In order to get permissions for an object located on a specific path, you can call the API:

IStore genericStorage = AzureDataLakeStorage.FromClientSecret(name, key);
IAzureDataLakeGen2BlobStorage gen2Storage = (IAzureDataLakeGen2BlobStorage)genericStorage;

//get permissions
AccessControl access = await _storage.GetAccessControlAsync(path);

AccessControl is a self explanatory structure that contains information about owning user, owning group, their permissions, and any custom ACL entries assigned to this object.

In order to set permissions, you need to call SetAccessControlAsync passing back modified AccessControl structure. Let's say I'd like to add write access to a user with ID 6b157067-78b0-4478-ba7b-ade5c66f1a9a (Active Directory Object ID). I'd write code like this (using the structure we've just got back from GetAccessControlAsync):

// add user to custom ACL
access.Acl.Add(new AclEntry(ObjectType.User, userId, false, true, false));

//update the ACL on Gen 2 storage
await _storage.SetAccessControlAsync(path, access);