Gluster

Knowledgebase: Gluster

Gluster - General Configuration and Troubleshooting

NFS-Ganesha fixes

NFS-Ganesha service stops but Virtual IP does not failover / Pro-active nfs-ganesha restart

NFS-Ganesha occasionally runs into an issue where the NFS service on one of the pcs cluster nodes hangs but does not trigger a failover. The steps below can also be used to pro-actively restart nfs-ganesha on a Gluster NFS host that is running out of RAM. See Grafana for current RAM usage: https://mon.rc.nectar.org.au/melbourne-grafana/dashboard/db/gluster-hosts

To determine which node is the culprit, usually running showmount for each NFS VIP (172.26.22.15[0-5]) will reveal that a particular node is hanging.

showmount -e 172.26.22.150

Once the failing node is confirmed:

  • Float the NFS VIP to another node.
  • Restart NFS-Ganesha on the problematic host
  • Float the NFS VIP back to the home node.
Float the NFS VIP to another node.

On the problematic node (n[0-5]-gluster1-qh2):
Show the pcs resources:

pcs status

Confirm the relevant problematic server IP is residing on the expected node:

ip addr

Float the IP to another node:

pcs resource move n5-gluster1-qh2-cluster_ip-1 n0-gluster1-qh2

Confirm the IP is no longer on the node:

ip addr
Restart NFS-Ganesha

On the problematic node, restart the nfs-ganesha service:

systemctl restart nfs-ganesha

Wait for ganesha.log to display the following output to confirm nfs-ganesha has completed initialization (Will take approx 2-3 minutes).

tailf /var/log/ganesha.log
ganesha.nfsd-92855[main] nfs_start :NFS STARTUP :EVENT :-------------------------------------------------
ganesha.nfsd-92855[main] nfs_start :NFS STARTUP :EVENT :             NFS SERVER INITIALIZED
ganesha.nfsd-92855[main] nfs_start :NFS STARTUP :EVENT :-------------------------------------------------
ganesha.nfsd-92855[reaper] nfs_in_grace :STATE :EVENT :NFS Server Now NOT IN GRACE
Float the NFS VIP back to the home node.

Float the IP back to the home node:

pcs resource move n5-gluster1-qh2-cluster_ip-1 n5-gluster1-qh2

Confirm

pcs status

Check IP has returned:

ip addr

Test client mount of NFS exports from previously problematic NFS Server IP.

NFS-Ganesha service stops and Virtual IP has already successfully failed over

Run pcs status to show the location of the pcs resource virtual IPs.
For example, in output below, the IP resource belonging to n5-gluster1-qh2 (n5-gluster1-qh2-cluster_ip-1) is on n3-gluster1-qh2:

# pcs status                                                                                                                                            Cluster name: gluster1_ha
Stack: corosync
Current DC: admin-gluster1-qh2 (version 1.1.15-11.el7_3.4-e174ec8) - partition with quorum
Last updated: Fri Aug 11 15:15:08 2017          Last change: Fri Aug 11 15:10:55 2017 by root via cibadmin on n5-gluster1-qh2
 
7 nodes and 39 resources configured
 
Online: [ admin-gluster1-qh2 n0-gluster1-qh2 n1-gluster1-qh2 n2-gluster1-qh2 n3-gluster1-qh2 n4-gluster1-qh2 n5-gluster1-qh2 ]
 
Full list of resources:
 
 Clone Set: nfs_setup-clone [nfs_setup]
     Started: [ n0-gluster1-qh2 n1-gluster1-qh2 n2-gluster1-qh2 n3-gluster1-qh2 n4-gluster1-qh2 n5-gluster1-qh2 ]
     Stopped: [ admin-gluster1-qh2 ]
 Clone Set: nfs-mon-clone [nfs-mon]
     Started: [ n0-gluster1-qh2 n1-gluster1-qh2 n2-gluster1-qh2 n3-gluster1-qh2 n4-gluster1-qh2 n5-gluster1-qh2 ]
     Stopped: [ admin-gluster1-qh2 ]
 Clone Set: nfs-grace-clone [nfs-grace]
     Started: [ n0-gluster1-qh2 n1-gluster1-qh2 n2-gluster1-qh2 n3-gluster1-qh2 n4-gluster1-qh2 n5-gluster1-qh2 ]
     Stopped: [ admin-gluster1-qh2 ]
 Resource Group: n0-gluster1-qh2-group
     n0-gluster1-qh2-nfs_block  (ocf::heartbeat:portblock):     Started n0-gluster1-qh2
     n0-gluster1-qh2-cluster_ip-1       (ocf::heartbeat:IPaddr):        Started n0-gluster1-qh2
     n0-gluster1-qh2-nfs_unblock        (ocf::heartbeat:portblock):     Started n0-gluster1-qh2
 Resource Group: n1-gluster1-qh2-group
     n1-gluster1-qh2-nfs_block  (ocf::heartbeat:portblock):     Started n1-gluster1-qh2
     n1-gluster1-qh2-cluster_ip-1       (ocf::heartbeat:IPaddr):        Started n1-gluster1-qh2
     n1-gluster1-qh2-nfs_unblock        (ocf::heartbeat:portblock):     Started n1-gluster1-qh2
 Resource Group: n2-gluster1-qh2-group
     n2-gluster1-qh2-nfs_block  (ocf::heartbeat:portblock):     Started n2-gluster1-qh2
     n2-gluster1-qh2-cluster_ip-1       (ocf::heartbeat:IPaddr):        Started n2-gluster1-qh2
     n2-gluster1-qh2-nfs_unblock        (ocf::heartbeat:portblock):     Started n2-gluster1-qh2
 Resource Group: n3-gluster1-qh2-group
     n3-gluster1-qh2-nfs_block  (ocf::heartbeat:portblock):     Started n3-gluster1-qh2
     n3-gluster1-qh2-cluster_ip-1       (ocf::heartbeat:IPaddr):        Started n3-gluster1-qh2
     n3-gluster1-qh2-nfs_unblock        (ocf::heartbeat:portblock):     Started n3-gluster1-qh2
 Resource Group: n4-gluster1-qh2-group
     n4-gluster1-qh2-nfs_block  (ocf::heartbeat:portblock):     Started n4-gluster1-qh2
     n4-gluster1-qh2-cluster_ip-1       (ocf::heartbeat:IPaddr):        Started n4-gluster1-qh2
     n4-gluster1-qh2-nfs_unblock        (ocf::heartbeat:portblock):     Started n4-gluster1-qh2
 Resource Group: n5-gluster1-qh2-group
     n5-gluster1-qh2-nfs_block  (ocf::heartbeat:portblock):     Started n3-gluster1-qh2
     n5-gluster1-qh2-cluster_ip-1       (ocf::heartbeat:IPaddr):        Started n3-gluster1-qh2
     n5-gluster1-qh2-nfs_unblock        (ocf::heartbeat:portblock):     Started n3-gluster1-qh2
 
Daemon Status:
  corosync: active/disabled
  pacemaker: active/disabled
  pcsd: active/enabled

Confirm nfs-ganesha service has stopped on problem node

systemctl status nfs-ganesha

Starting the nfs-ganesha service will trigger the Virtual IP resource to automatically failback to the original node. We want to prevent this automatic failback because the IP is transferred immediately while the nfs-ganesha service takes 2-3 minutes to complete initialisation of all the exports. To prevent a long I/O interruption we temporarily add a pcs constraint (i.e. prevent the Virtual IP resource from moving to the node that is restarting the nfs-ganesha service).

pcs constraint location n5-gluster1-qh2-cluster_ip-1 avoids n5-gluster1-qh2

Restart nfs-ganesha on the problem node

systemctl start nfs-ganesha

Wait for ganesha.log to display the following output to confirm nfs-ganesha has completed initialization. Will take approx 2-3 minutes.

tailf /var/log/ganesha.log
ganesha.nfsd-92855[main] nfs_start :NFS STARTUP :EVENT :-------------------------------------------------
ganesha.nfsd-92855[main] nfs_start :NFS STARTUP :EVENT :             NFS SERVER INITIALIZED
ganesha.nfsd-92855[main] nfs_start :NFS STARTUP :EVENT :-------------------------------------------------
ganesha.nfsd-92855[reaper] nfs_in_grace :STATE :EVENT :NFS Server Now NOT IN GRACE

Show the list of pcs constraints to obtain the ID of the constraint we created earlier.

pcs constraint list --full
 
***snip***
  Resource: n5-gluster1-qh2-cluster_ip-1
    Enabled on: n3-gluster1-qh2 (score:INFINITY) (role: Started) (id:cli-prefer-n5-gluster1-qh2-cluster_ip-1)
    Disabled on: n5-gluster1-qh2 (score:-INFINITY) (id:location-n5-gluster1-qh2-cluster_ip-1-n5-gluster1-qh2--INFINITY)
  Resource: n5-gluster1-qh2-group
    Enabled on: n0-gluster1-qh2 (score:1000) (id:location-n5-gluster1-qh2-group-n0-gluster1-qh2-1000)
    Enabled on: n1-gluster1-qh2 (score:2000) (id:location-n5-gluster1-qh2-group-n1-gluster1-qh2-2000)
    Enabled on: n2-gluster1-qh2 (score:3000) (id:location-n5-gluster1-qh2-group-n2-gluster1-qh2-3000)
    Enabled on: n3-gluster1-qh2 (score:4000) (id:location-n5-gluster1-qh2-group-n3-gluster1-qh2-4000)
    Enabled on: n4-gluster1-qh2 (score:5000) (id:location-n5-gluster1-qh2-group-n4-gluster1-qh2-5000)
    Enabled on: n5-gluster1-qh2 (score:6000) (id:location-n5-gluster1-qh2-group-n5-gluster1-qh2-6000)
    Constraint: location-n5-gluster1-qh2-group
      Rule: score=-INFINITY  (id:location-n5-gluster1-qh2-group-rule)
        Expression: ganesha-active ne 1  (id:location-n5-gluster1-qh2-group-rule-expr)
***snip***

Relevant line from output above is:

Disabled on: n5-gluster1-qh2 (score:-INFINITY) (id:location-n5-gluster1-qh2-cluster_ip-1-n5-gluster1-qh2--INFINITY)

Remove the constraint

pcs constraint remove location-n5-gluster1-qh2-cluster_ip-1-n5-gluster1-qh2--INFINITY

Confirm constraint has been removed

pcs constraint list --full
 
***snip***
  Resource: n5-gluster1-qh2-cluster_ip-1
    Enabled on: n3-gluster1-qh2 (score:INFINITY) (role: Started) (id:cli-prefer-n5-gluster1-qh2-cluster_ip-1)
  Resource: n5-gluster1-qh2-group
    Enabled on: n0-gluster1-qh2 (score:1000) (id:location-n5-gluster1-qh2-group-n0-gluster1-qh2-1000)
    Enabled on: n1-gluster1-qh2 (score:2000) (id:location-n5-gluster1-qh2-group-n1-gluster1-qh2-2000)
    Enabled on: n2-gluster1-qh2 (score:3000) (id:location-n5-gluster1-qh2-group-n2-gluster1-qh2-3000)
    Enabled on: n3-gluster1-qh2 (score:4000) (id:location-n5-gluster1-qh2-group-n3-gluster1-qh2-4000)
    Enabled on: n4-gluster1-qh2 (score:5000) (id:location-n5-gluster1-qh2-group-n4-gluster1-qh2-5000)
    Enabled on: n5-gluster1-qh2 (score:6000) (id:location-n5-gluster1-qh2-group-n5-gluster1-qh2-6000)
    Constraint: location-n5-gluster1-qh2-group
      Rule: score=-INFINITY  (id:location-n5-gluster1-qh2-group-rule)
        Expression: ganesha-active ne 1  (id:location-n5-gluster1-qh2-group-rule-expr)
***snip***

Manually move IP back to its home node

pcs resource move n5-gluster1-qh2-cluster_ip-1 n5-gluster1-qh2

Confirm all pcs resources are at their home node

pcs status

Confirm mounts work. Can use the following script to check showmount on each NFS Virtual IP

#!/bin/bash
for i in {0..5}; do
  mountcount=$(timeout -k 5s 5s showmount -e 172.26.22.15$i | wc -l)
  echo -n "n$i-gluster1-qh2: "
  if (( $mountcount >= 35 )); then
    echo "Success (mountcount=$mountcount)"
  else
    echo "Fail (mountcount=$mountcount)"
  fi
done

Samba Node fixes

Running out of memory / Pro-active smb service restart

On a Samba node (n[6-11]-gluster1-qh2), find the client accessible IPs the problem node is currently hosting: ctdb ip -v

Move any IPs currently hosted by that node to another CTDB host:

  • Requires ctdb tunables to be set (these tunables should already be configured):
ctdb listvars | grep IP
ctdb setvar DeterministicIPs 0
ctdb setvar NoIPFailback 1
  • Move IP to another active CTDB node:
ctdb moveip <IP_ADDRESS> <DESTINATION_CTDB_NODE_NUMBER>
ctdb moveip 172.26.22.210 6
  • Restart smb services on problem node:
systemctl restart smb
  • Move IP back to original node:
ctdb moveip <IP_ADDRESS> <DESTINATION_CTDB_NODE_NUMBER>
ctdb moveip 172.26.22.210 10

Quorum / Split-brain prevention

Server-side quorum is enabled.
https://access.redhat.com/documentation/en-US/Red_Hat_Storage/3.1/html/Administration_Guide/sect-Managing_Split-brain.html#Configuring_Server-Side_Quorum
cluster.server-quorum-ratio is 51%
"The quorum ratio setting of 51% means that more than half of the nodes in the trusted storage pool must be online and have network connectivity between them at any given time. If a network disconnect happens to the storage pool, then the bricks running on those nodes are stopped to prevent further writes."

gluster volume set cifs-vol1 cluster.server-quorum-type server
gluster volume set nfs-vol1 cluster.server-quorum-type server
gluster volume set gluster_shared_storage cluster.server-quorum-type server
gluster volume set ctdb-lock cluster.server-quorum-type server

Client-site quorum is enabled.
https://access.redhat.com/documentation/en-US/Red_Hat_Storage/3.1/html/Administration_Guide/sect-Managing_Split-brain.html#Configuring_Client-Side_Quorum
For each data volume, cluster.quorum-type is set to auto.
"This option allows writes to the file only if the percentage of active replicate bricks is more than 50% of the total number of bricks that constitute that replica. If there are only two bricks in the replica group, the first brick must be up and running to allow modifications."

gluster volume set cifs-vol1 cluster.quorum-type auto
gluster volume set nfs-vol1 cluster.quorum-type auto

IMPORTANT: Before performing any maintenance which will affect connectivity to or uptime of a brick (e.g. node reboot), client-side quorum will need to be temporarily disabled on each relevant volume to prevent the remaining replica brick from entering a read-only state. Once maintenance has completed then client-side quorum will need to be re-enabled on the relevant volumes.

To disable client-side quorum:

gluster volume set cifs-vol1 cluster.quorum-type none
gluster volume set nfs-vol1 cluster.quorum-type none

To re-enable client-side quorum:

gluster volume set cifs-vol1 cluster.quorum-type auto
gluster volume set nfs-vol1 cluster.quorum-type auto

NFS

Create a new NFS share

https://access.redhat.com/documentation/en-US/Red_Hat_Storage/3.1/html/Administration_Guide/sect-NFS.html#sect-NFS_Ganesha

Create sub-directory to use as export

Mount the Gluster NFS volume nfs-vol1 on the Gluster admin node (admin-gluster1-qh2):

mount -t glusterfs localhost:/nfs-vol1 /mnt/nfs-vol1

For a new faculty export, create a new directory for the faculty (if it does not already exist) prefixed with the faculty code.

mkdir /mnt/nfs-vol1/6000-science

Then create a sub-directory with the relevant unique project name/code:

mkdir /mnt/nfs-vol1/6000-science/6100-chem

For a VicNode allocation, create a new directory beneath the 9770-vicnode directory:

mkdir /mnt/nfs-vol1/9770-vicnode/9560-BionicsInstitute
Apply quota to project directory (enter the path relative to the volume)
gluster volume quota nfs-vol1 limit-usage /6000-science/6100-chem 50TB

Display quota information to confirm:

gluster volume quota nfs-vol1 list
Edit export conf file

Check out module puppet-melbourne and edit files/gluster/nfs/exports/$export_file.conf
Create additional stanza/s for new share/s at the end of the file.
Increment the Export_Id of the previous export and ensure that both the Export_Id and the Pseudo path are unique to each export.

EXPORT{
    Export_Id = 7;
    Path = "/nfs-vol1/6000-science";
 
    FSAL {
      name = GLUSTER;
      hostname = "localhost";
      volume = "nfs-vol1";
      volpath = "/6000-science";
    }
 
    Pseudo = "/6000-science";
    Protocols =  "3", "4";
    Transports = "UDP","TCP";
    SecType = "sys";
 
    CLIENT {
      Clients = XXX.XXX.XXX.XXX/XX, 128.250.116.160/27;
      Squash = None;
      Access_Type = RW;
    }
  }
 
  EXPORT{
      Export_Id = 8;
      Path = "/nfs-vol1/6000-science/6100-chem";
 
      FSAL {
        name = GLUSTER;
        hostname = "localhost";
        volume = "nfs-vol1";
        volpath = "/6000-science/6100-chem";
      }
 
      Pseudo = "/6100-chem";
      Protocols =  "3", "4";
      Transports = "UDP","TCP";
      SecType = "sys";
 
      CLIENT {
        Clients = XXX.XXX.XXX.XXX/XX, 128.250.116.160/27;
        Squash = None;
        Access_Type = RW;
      }
    }

Send it up for review.

Run puppet on all Gluster nodes to pull down the new export file
Dynamically add NFS export (without restarting services) on Gluster servers

pdsh -R ssh -w root@n[0-5]-gluster1-qh2 "/root/gluster_scripts/ganesha-dbus-exports.sh add nfs-vol1 6000-science 6100-chem"

Mount a NFS share (client-side)

Edit /etc/idmapd.conf and add under [General]:

Domain = storage.unimelb.edu.au

Clear idmap cache:

nfsidmap -c

Restart idmapd:

service rpcidmapd restart (RHEL / Centos) or restart idmapd (Ubuntu)

On Ubuntu / Debian, because the default options are slow:

Command line:

mount -t nfs -o rw,relatime,vers=4.0,rsize=1048576,wsize=1048576,namlen=255,hard,proto=tcp,port=0,timeo=600,retrans=2,sec=sys,local_lock=none research-nfs.storage.unimelb.edu.au:/6100-chem /mnt/6100-chem

/etc/fstab:

research-nfs.storage.unimelb.edu.au:/6100-chem /mnt/6100-chem nfs rw,relatime,vers=4.0,rsize=1048576,wsize=1048576,namlen=255,hard,proto=tcp,port=0,timeo=600,retrans=2,sec=sys,local_lock=none 0 2

On RHEL / Centos, because the default options are fast:

Command line:

mount -t nfs research-nfs.storage.unimelb.edu.au:/6100-chem /mnt/6100-chem

/etc/fstab:

research-nfs.storage.unimelb.edu.au:/6100-chem /mnt/6100-chem nfs defaults 0 2

Modify existing NFS shares (access & quota)

https://access.redhat.com/documentation/en-US/Red_Hat_Storage/3.1/html/Administration_Guide/sect-NFS.html#sect-NFS_Ganesha https://github.com/nfs-ganesha/nfs-ganesha/wiki/Dbusinterface https://github.com/nfs-ganesha/nfs-ganesha/wiki/DBusExports

Modify access to existing NFS export

Edit export conf file

Check out module puppet-melbourne and edit files/gluster/nfs/exports/$export_file.conf
Modify export (e.g. update client IP addresses) as required.

  EXPORT{
      Export_Id = 8;
      Path = "/nfs-vol1/6000-science/6100-chem";
 
      FSAL {
        name = GLUSTER;
        hostname = "localhost";
        volume = "nfs-vol1";
        volpath = "/6000-science/6100-chem";
      }
 
      Pseudo = "/6100-chem";
      Protocols =  "3", "4";
      Transports = "UDP","TCP";
      SecType = "sys";
 
      CLIENT {
        Clients = XXX.XXX.XXX.XXX/XX, 128.250.116.160/27;
        Squash = None;
        Access_Type = RW;
      }
    }

Send it up for review.

Run puppet on all Gluster nodes to pull down the new export file
Dynamically update NFS export (without restarting services) on Gluster servers

pdsh -R ssh -w root@n[0-5]-gluster1-qh2 "/root/gluster_scripts/ganesha-dbus-exports.sh update nfs-vol1 6000-science 6100-chem"

Remove NFS export

Edit export conf file

Check out module puppet-melbourne and edit files/gluster/nfs/exports/$export_file.conf
Remove relevant export stanza.

  EXPORT{
      Export_Id = 8;
      Path = "/nfs-vol1/6000-science/6100-chem";
 
      FSAL {
        name = GLUSTER;
        hostname = "localhost";
        volume = "nfs-vol1";
        volpath = "/6000-science/6100-chem";
      }
 
      Pseudo = "/6100-chem";
      Protocols =  "3", "4";
      Transports = "UDP","TCP";
      SecType = "sys";
 
      CLIENT {
        Clients = XXX.XXX.XXX.XXX/XX, 128.250.116.160/27;
        Squash = None;
        Access_Type = RW;
      }
    }

Send it up for review.

NB: Remove export using command below before running puppet on nodes

Dynamically remove NFS export (without restarting services)

Find Export_Id of share that is to be removed in the export conf file
Then substitute the ID number in the command below:
pdsh -R ssh -w root@n[0-5]-gluster1-qh2 "/root/gluster_scripts/ganesha-dbus-exports.sh del nfs-vol1 $export_id"

Run puppet on all Gluster nodes to pull down the new export file

Quota

https://access.redhat.com/documentation/en-US/Red_Hat_Storage/3.1/html/Administration_Guide/chap-Managing_Directory_Quotas.html

Display Quota Limit Information

gluster volume quota nfs-vol1 list

Set quota on share/sub-directory (enter the path relative to the volume)

gluster volume quota nfs-vol1 limit-usage /6000-science/6100-chem 50TB

CIFS

Create a new CIFS share

https://access.redhat.com/documentation/en-US/Red_Hat_Storage/3.1/html/Administration_Guide/sect-SMB.html#sect-Sharing_Volumes_over_SMB

Create sub-directory to use as share

Pre-requisite: Need to be member of Active Directory security group 9770-40GG-Gluster-FileShares-FullControl to Administer root of Gluster CIFS volume.

Windows Administration VM:

  • windows-devops.cloud.unimelb.edu.au (45.113.235.249)
  • Windows Server 2016
  • Joined to Unimelb domain
  • RDP accessible from 128.250.116.160/27
  • Active Directory Users & Computers installed
  • SetACL Studio installed

Create a new security group for the project share in Active Directory under:

unimelb.edu.au/Applications/Research/Security Groups/gluster/

e.g.

unimelb.edu.au/Applications/Research/Security Groups/gluster/9560gg-BionicsInstitute-Gluster-Admin

Add the relevant end-user groups (nominated by the project admin) to that. If the group needs to contain users/groups from student.unimelb.edu.au, select Group Scope: Universal

NB: Changing the NTFS permissions of a directory when it already contains a large number of files and sub-directories can take a very long time because the permissions are inherited, and so every file contained within needs to be updated. To avoid this, we create a new security group per project share so that if end-users decide that they need to change the user accounts or security groups that have access to their project directory then it only requires a change to the membership of the relevant security group and no change of NTFS permissions is required.

In Windows, mount \\research-cifs.storage.unimelb.edu.au\cifs-vol1

For a new faculty share, create a new directory for the faculty prefixed with the faculty code.
E.g. \\research-cifs.storage.unimelb.edu.au\cifs-vol1\6000-science

For a VicNode allocation, create a new directory beneath the 9770-vicnode directory: E.g. \\research-cifs.storage.unimelb.edu.au\cifs-vol1\9770-vicnode\5050-medproject

Newly created directories will have the following default NTFS permissions:
9770-40GG-Gluster-FileShares-FullControl - Full Control
9770us-40-Gluster - Full Control

Edit the new directory permissions and add the required Admin security group with Full Control permissions. Faculty/Project Admins can then set appropriate NTFS permissions for their users.
To set permissions using SetACL Studio follow the steps below.
NB: SetACL Studio has been very slow to respond recently. See further below for instructions for setting permissions using icacls in cmd.exe.
To configure SetAcl Studio:

  • Open 'SetACL'
  • Select the 'Add Computer' icon (top left)
  • Enter the name of a remote computer: research-cifs.storage.unimelb.edu.au
  • In left column, double-click Computer(research-cifs.storage.unimelb.edu.au)
  • Double-click File System
  • Double-click cifs-vol1 For a VicNode allocation:
  • Right-click 9770-vicnode, create sub-directory with required name. e.g. 5050-medproject
  • Select the newly created sub-directory and ensure that the only permissions are the inherited Full Control permissions for 9770us-40-gluster and 9770-40GG-Gluster-Fileshares-FullControl.
  • Then add the required permissions for the end user's admin group. Select 'add', search for the relevant AD security group, select 'Full Control' permissions, Applies to 'This folder, subfolders and files'.

To set permissions using icacls in cmd.exe (reference: https://ss64.com/nt/icacls.html):

  • Open cmd.exe
  • Map cifs-vol1 share:
net use z: \\research-cifs.storage.unimelb.edu.au\cifs-vol1
  • Add "faculty level admin security group" (Full Control access, This folder, subfolders and files):
icacls "Z:\7020-abp" /grant "7020gg-ABP-Gluster-admin@unimelb.edu.au":(OI)(CI)F
  • Add "Everyone" to faculty level share (Read & Execute, This folder only). This allows all users to traverse faculty level directories and only those with correct permissions can access the relevant project sub-directories.
icacls "Z:\7020-abp" /grant "Everyone":RX
  • Add "Project level admin security group" (Full Control access, This folder, subfolders and files):
icacls "Z:\7020-abp\7020-LEaRN" /grant "7020gg-LEaRN-Gluster-Admin@unimelb.edu.au":(OI)(CI)F
Apply quota to project directory (enter the path relative to the volume)

On Gluster Admin node, admin-gluster1-qh2:

gluster volume quota cifs-vol1 limit-usage /9770-vicnode/418-SocialNUI 40TB

Display quota information to confirm:

gluster volume quota cifs-vol1 list
Edit export conf file

Check out module puppet-melbourne.
For a new faculty share:
Edit templates/gluster/cifs/smb.erb
Add a new include line at the bottom of the file.
E.g.
include = /etc/samba/shares/4000-eng.conf
Then create a new share file:
/etc/samba/shares/4000-eng.conf

# Gluster Samba Shares for Engineering Faculty
 
[4000-eng]
comment = Samba share for Engineering Faculty on cifs-vol1
vfs objects = acl_xattr glusterfs
glusterfs:volume = cifs-vol1
glusterfs:logfile = /var/log/samba/glusterfs-cifs-vol1.%M.log
glusterfs:loglevel = 3
path = /4000-eng
read only = no
guest ok = no

For a Vicnode allocation, edit: files/gluster/cifs/shares/9770-vicnode.conf
Create additional stanza/s for new share/s at the end of the file.

# Gluster Samba Shares for VicNode Allocations
 
[9770-vicnode]
comment = Samba share for VicNode Allocations on cifs-vol1
vfs objects = acl_xattr glusterfs
glusterfs:volume = cifs-vol1
glusterfs:logfile = /var/log/samba/glusterfs-cifs-vol1.%M.log
glusterfs:loglevel = 3
path = /9770-vicnode
read only = no
guest ok = no
 
[418-SocialNUI]
comment = Samba share for SocialNUI on cifs-vol1
vfs objects = acl_xattr glusterfs
glusterfs:volume = cifs-vol1
glusterfs:logfile = /var/log/samba/glusterfs-cifs-vol1.%M.log
glusterfs:loglevel = 3
path = /9770-vicnode/418-SocialNUI
read only = no
guest ok = no
 

Send it up for review.

Run puppet on all Gluster nodes to pull down the new share file
Reload samba config on Gluster servers

pdsh -R ssh -w root@n[6-11]-gluster1-qh2 'smbcontrol all reload-config'

Mount a CIFS share (client-side)

 

Modify existing CIFS shares (access & quota)

https://access.redhat.com/documentation/en-US/Red_Hat_Storage/3.1/html/Administration_Guide/sect-SMB.html#sect-Sharing_Volumes_over_SMB

Quota

https://access.redhat.com/documentation/en-US/Red_Hat_Storage/3.1/html/Administration_Guide/chap-Managing_Directory_Quotas.html

Display Quota Limit Information

gluster volume quota cifs-vol1 list

Set quota on share/sub-directory (enter the path relative to the volume)

gluster volume quota cifs-vol1 limit-usage /6000-science/603-chem 20TB

Snapshot Configuration

https://access.redhat.com/documentation/en-US/Red_Hat_Storage/3.1/html/Administration_Guide/chap-Managing_Snapshots.html
Snapshot configuration has already been completed at a Gluster volume level.
When provisioning new shares on the existing Gluster volumes, the snapshot configuration and schedule will already be set and no further configuration is necessary. Notes below refer to steps taken to configure snapshots initially.
(Bricks need to have been created correctly to enable snapshot functionality. See /root/gluster_scripts/brick_config.sh script for brick provisioning details.)

Create test snapshot manually:

gluster snapshot create <snapname> <volname> [no-timestamp] [description <description>] [force]
gluster snapshot create snap1 cifs-vol1 description "Test snapshot of CIFS volume"

List snapshots:

gluster snapshot list

Enable activate-on-create so that snapshots are accessible immediately upon creation

gluster snapshot config activate-on-create enable

Limit total number of snapshots for volume to 30 (Since snap-max-soft-limit 90% is a global setting, set snap-max-hard-limit per Gluster volume at 34)

gluster snapshot config [VOLNAME] [snap-max-hard-limit <count>]
gluster snapshot config cifs-vol1 snap-max-hard-limit 34

Scheduled snapshots:

On all nodes, run this once to initialize scheduler:

snap_scheduler.py init

Then on a single node:

snap_scheduler.py enable
snap_scheduler.py status

Schedule daily snapshots at 12:10AM

snap_scheduler.py add "Job Name" "Schedule" "Volume Name"
snap_scheduler.py add "Daily-CIFS" "10 0 * * *" cifs-vol1
snap_scheduler.py add "Daily-NFS" "10 0 * * *" nfs-vol1

Confirm snapshot schedule

snap_scheduler.py list

Ensure snapshots are user accessible (.snaps dir):

NFS:
gluster volume set nfs-vol1 features.uss enable
CIFS:
gluster volume set cifs-vol1 features.uss enable
gluster volume set cifs-vol1 features.show-snapshot-directory on

Nothing to display