[삽질] Pod 내 nfs연결시 오류

 

OS : Ubuntu 20.04

kubernetes : 1.26

nfs server : 

작업: Pod내 nfs 연결하는 yaml 실행

장애: pod 생성이 완료되지 않고 ContainerCreating 상태 지속됨

Error Message : MountVolume.SetUp failed for volume "html" : mount failed: exit status 32

Error Message : Unable to attach or mount volumes: unmounted volumes=[html]

Error Message : unattached volumes=[html kube-api-access-jmn62]: timed out waiting for the condition

 

 

[nfs.yaml]

apiVersion: v1
kind: Pod
metadata:
  name: web-nfs
spec:
  containers:
  - image: nginx:1.14
    name: nginx
    volumeMounts:
    - name: html
      mountPath: /usr/share/nginx/html
  volumes:
  - name: html
    nfs:
      server: 10.0.2.21
      path: /sharedir/k8s

 

• yaml 실행 결과는 정상

$ kubectl apply -f nfs.yaml
pod/web-nfs created

 

• pod는 아직도 생성중

$ kubectl get pods -o wide
NAME      READY   STATUS              RESTARTS   AGE     IP       NODE    NOMINATED NODE   READINESS GATES
web-nfs   0/1     ContainerCreating   0          2m11s   <none>   k8s-2   <none>           <none>

 

• pod 상태 확인 결과 mount fail 

$ kubectl describe pod web-nfs
Name:             web-nfs
Namespace:        default
Priority:         0
Service Account:  default
Node:             k8s-2/10.0.2.22
Start Time:       Tue, 11 Apr 2023 00:23:34 +0000
Labels:           <none>
Annotations:      <none>
Status:           Pending
IP:
IPs:              <none>
Containers:
  nginx:
    Container ID:
    Image:          nginx:1.14
    Image ID:
    Port:           <none>
    Host Port:      <none>
    State:          Waiting
      Reason:       ContainerCreating
    Ready:          False
    Restart Count:  0
    Environment:    <none>
    Mounts:
      /usr/share/nginx/html from html (rw)
      /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-jmn62 (ro)
Conditions:
  Type              Status
  Initialized       True
  Ready             False
  ContainersReady   False
  PodScheduled      True
Volumes:
  html:
    Type:      NFS (an NFS mount that lasts the lifetime of a pod)
    Server:    10.0.2.21
    Path:      /sharedir/k8s
    ReadOnly:  false
  kube-api-access-jmn62:
    Type:                    Projected (a volume that contains injected data from multiple sources)
    TokenExpirationSeconds:  3607
    ConfigMapName:           kube-root-ca.crt
    ConfigMapOptional:       <nil>
    DownwardAPI:             true
QoS Class:                   BestEffort
Node-Selectors:              <none>
Tolerations:                 node.kubernetes.io/not-ready:NoExecute op=Exists for 300s
                             node.kubernetes.io/unreachable:NoExecute op=Exists for 300s
Events:
  Type     Reason       Age                  From               Message
  ----     ------       ----                 ----               -------
  Normal   Scheduled    3m44s                default-scheduler  Successfully assigned default/web-nfs to k8s-2
  Warning  FailedMount  101s                 kubelet            Unable to attach or mount volumes: unmounted volumes=[html], unattached volumes=[html kube-api-access-jmn62]: timed out waiting for the condition
  Warning  FailedMount  96s (x9 over 3m44s)  kubelet            MountVolume.SetUp failed for volume "html" : mount failed: exit status 32
Mounting command: mount
Mounting arguments: -t nfs 10.0.2.21:/sharedir/k8s /var/lib/kubelet/pods/f9941be0-2389-4f96-a613-f9f07da47394/volumes/kubernetes.io~nfs/html
Output: mount: /var/lib/kubelet/pods/f9941be0-2389-4f96-a613-f9f07da47394/volumes/kubernetes.io~nfs/html: bad option; for several filesystems (e.g. nfs, cifs) you might need a /sbin/mount.<type> helper program.

 

• worker node의 os 로그 확인

$ vi /var/log/syslog

Apr 11 00:31:48 k8s-2 kubelet[689]: E0411 00:31:48.625157     689 mount_linux.go:231] Mount failed: exit status 32
Apr 11 00:31:48 k8s-2 kubelet[689]: Mounting command: mount
Apr 11 00:31:48 k8s-2 kubelet[689]: Mounting arguments: -t nfs 10.0.2.21:/sharedir/k8s /var/lib/kubelet/pods/f9941be0-2389-4f96-a613-f9f07da47394/volumes/kubernetes.io~nfs/html
Apr 11 00:31:48 k8s-2 kubelet[689]: Output: mount: /var/lib/kubelet/pods/f9941be0-2389-4f96-a613-f9f07da47394/volumes/kubernetes.io~nfs/html: bad option; for several filesystems (e.g. nfs, cifs) you might need a /sbin/mount.<type> helper program.
Apr 11 00:31:48 k8s-2 kubelet[689]: E0411 00:31:48.626583     689 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/nfs/f9941be0-2389-4f96-a613-f9f07da47394-html podName:f9941be0-2389-4f96-a613-f9f07da47394 nodeName:}" failed. No retries permitted until 2023-04-11 00:33:50.626561963 +0000 UTC m=+331523.109538115 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "html" (UniqueName: "kubernetes.io/nfs/f9941be0-2389-4f96-a613-f9f07da47394-html") pod "web-nfs" (UID: "f9941be0-2389-4f96-a613-f9f07da47394") : mount failed: exit status 32
Apr 11 00:31:48 k8s-2 kubelet[689]: Mounting command: mount
Apr 11 00:31:48 k8s-2 kubelet[689]: Mounting arguments: -t nfs 10.0.2.21:/sharedir/k8s /var/lib/kubelet/pods/f9941be0-2389-4f96-a613-f9f07da47394/volumes/kubernetes.io~nfs/html
Apr 11 00:31:48 k8s-2 kubelet[689]: Output: mount: /var/lib/kubelet/pods/f9941be0-2389-4f96-a613-f9f07da47394/volumes/kubernetes.io~nfs/html: bad option; for several filesystems (e.g. nfs, cifs) you might need a /sbin/mount.<type> helper program.

 

• nfs-common 설치

$ sudo apt install nfs-common

 

• 해결

$ kubectl get pods -o wide                                                                                                                                                                     k8s-1: Tue Apr 11 00:39:33 2023

NAME      READY   STATUS    RESTARTS   AGE   IP          NODE    NOMINATED NODE   READINESS GATES
web-nfs   1/1     Running   0          16m   10.44.0.1   k8s-2   <none>           <none>