svc:/system/filesystem/local:default goes to maintenance status, find root cause and how to solve

svc:/system/filesystem/local handles filesystem mounting related. And it has many dependency services. When this service got a problem, many services will impacted with Offline status. For example network service, ssh service, etc. While service system/filesystem/local is not online status the other service status are offline.

Issue: Many services are not started with Offline status (ssh, .. etc)
Cause: svc:/system/filesystem/local:default service state is Maintenance
Impact: 42 Dependencies service is not running


Find out root caused :

For my issue, I have an example, Kernel can't find a file system mounted on /etc/vfstab during boot time when it will be checked with fsck. It caused service svc:/system/filesystem/local:default is goes to maintenance status. Operation of this service is stoped with failed to mount the missing filesystem. Bellow the error messages:


 
 WARNING - Unable to repair one or more filesystems.
 Run fsck manually (fsck filesystem...).
 checking ufs filesystems
 fsck: could not stat /dev/rdsk/c0t60060E801040B52004F28612000000000d0s0: No such file or directory  
 fsckall failed with exit code 1.  
   
 WARNING - Unable to repair one or more filesystems.  
 Run fsck manually (fsck filesystem...).  
   
 svc:/system/filesystem/local:default: WARNING: /usr/sbin/mountall -l failed: exit status 32  
 Aug 25 12:52:55 svc.startd[11]: svc:/system/filesystem/local:default: Method "/lib/svc/method/fs-local" failed with exit status 95.  
 Aug 25 12:52:55 svc.startd[11]: system/filesystem/local:default failed fatally: transitioned to maintenance (see 'svcs -xv' for details)  

Handling :

We have found that the missing filesystem is caused by svc:/system/filesystem/local service has scan and do fsck during OS boot. It read the /etc/vfstab file. It's was read and do fsck for missing LUN (external disk), then it failed with message "no such file or directory". So we should disabled it with put '#' character at the line of the missing LUN at /etc/vfstab. Then continued with mount all detected storages.



 # vi /etc/vfstab  
 #device     device     mount      FS   fsck  mount  mount  
 #to mount    to fsck     point      type  pass  at boot options  
 #
 /devices    -        /devices    devfs  -    no   -  
 /proc      -        /proc      proc  -    no   -  
 ctfs      -        /system/contract ctfs  -    no   -  
 objfs      -        /system/object objfs  -    no   -  
 sharefs     -        /etc/dfs/sharetab    sharefs -    no   -  
 fd       -        /dev/fd     fd   -   no   -  
 swap      -        /tmp      tmpfs  -    yes   -  
 /dev/zvol/dsk/rpool/swap    -        -        swap  -    no   -  
   
 #/dev/rdsk/c0t60060E801040B52004F28612000000000d0s0   /dev/rdsk  
 /c0t60060E801040B52004F28612000000000d0s0    /xdata    ufs   -    yes   -   

Remount all partition :


# mountall
mount: /tmp is already mounted or swap is busy

Restart service svc:/system/filesystem/local:default


# svcadm disable svc:/system/filesystem/local:default
# svcadm enable svc:/system/filesystem/local:default
Aug 25 13:00:58 sckmd: PF_KEY socket for IPsec load succeeded.

Result, service svc:/system/filesystem/local:default is back to normal and all dependencies
Check with command bellow :


# svcs -xv