need /sbin/reboot to work with RHEL6

[GoogleCodeExporter]

rebooting a node will currently just hang when the network shuts down. Any 
ideas on how to make a reboot happen?

Original issue reported on code.google.com by trentdho...@gmail.com on 28 Feb 2012 at 11:24

[GoogleCodeExporter]

in /etc/init.d/network I see this:


        # Don't shut the network down if root is on NFS or a network
        # block device.
        rootfs=$(awk '{ if ($1 !~ /^[ \t]*#/ && $2 == "/" && $3 != "rootfs") { print $3; }}' /proc/mounts)
        rootopts=$(awk '{ if ($1 !~ /^[ \t]*#/ && $2 == "/") { print $4; }}' /etc/mtab)

        if [[ "$rootfs" == nfs* || "$rootopts" =~ _r?netdev ]] ; then
                exit 1
        fi


Looking at the output from those awk commands, but printing $0 instead I see:


# hype201 /root > awk '{ if ($1 !~ /^[ \t]*#/ && $2 == "/" && $3 != "rootfs") { 
print $0; }}' /proc/mounts
192.168.81.156:/tftpboot/images/compute.x86_64 / nfs 
rw,relatime,vers=3,rsize=1048576,wsize=1048576,namlen=255,acregmin=600,acdirmin=
600,hard,nocto,nolock,nosharecache,proto=tcp,timeo=600,retrans=2,sec=sys,mountad
dr=192.168.81.156,mountvers=3,mountport=49727,mountproto=tcp,local_lock=all,addr
=192.168.81.156 0 0


awk '{ if ($1 !~ /^[ \t]*#/ && $2 == "/") { print $0; }}' /etc/mtab
/root / root rw 0 0


so /etc/init.d/network should exit 1, so this is likely not what is hanging the 
reboot

Original comment by trentdho...@gmail.com on 28 Feb 2012 at 11:41

[GoogleCodeExporter]

Here is what I see in the console:


Stopping cgconfig service: [  OK  ]
Sending all processes the TERM signal... [  OK  ]
Sending all processes the KILL signal... [  OK  ]
touch: cannot touch `/var/lib/random-seed': Read-only file system
chmod: cannot access `/var/lib/random-seed': No such file or directory
Saving random seed:  [FAILED]
Syncing hardware clock to system time [  OK  ]
Unmounting pipe file systems:  umount: /var/lib/nfs/rpc_pipefs: not mounted
[FAILED]
/vinit: rc main process (14261) killed by KILL signal

Original comment by trentdho...@gmail.com on 28 Feb 2012 at 11:56

[GoogleCodeExporter]

Adding bash -x to S01reboot I see:


+ '[' -n '' -a 3 -gt 0 ']'
+ __umount_loop '$3 ~ /^rpc_pipefs$/ || $3 ~ /^rpc_svc_gss_pipefs$/ {print $2}' 
/proc/mounts 'Unmounting pipe file systems: ' 'Unmounting pipe file systems 
(retry): ' -f
+ local remaining sig=
+ local retry=3 count
++ sort -r
++ LC_ALL=C
++ awk '/^#/ {next} $3 ~ /^rpc_pipefs$/ || $3 ~ /^rpc_svc_gss_pipefs$/ {print 
$2}' /proc/mounts
+ remaining='/var/lib/nfs/rpc_pipefs
/var/lib/nfs/rpc_pipefs'
+ '[' -n '/var/lib/nfs/rpc_pipefs
/var/lib/nfs/rpc_pipefs' -a 3 -gt 0 ']'
+ '[' 3 -eq 3 ']'
+ action 'Unmounting pipe file systems: ' fstab-decode umount 
/var/lib/nfs/rpc_pipefs /var/lib/nfs/rpc_pipefs
+ local STRING rc
+ STRING='Unmounting pipe file systems: '
+ echo -n 'Unmounting pipe file systems:  '
Unmounting pipe file systems:  + shift
+ fstab-decode umount /var/lib/nfs/rpc_pipefs /var/lib/nfs/rpc_pipefs
umount: /var/lib/nfs/rpc_pipefs: not mounted
+ failure 'Unmounting pipe file systems: '
+ local rc=1
+ '[' serial '!=' verbose -a -z '' ']'
+ echo_failure
+ '[' serial = color ']'
+ echo -n '['
[+ '[' serial = color ']'
+ echo -n FAILED
FAILED+ '[' serial = color ']'
+ echo -n ']'
]+ echo -ne '\r'
+ return 1
+ '[' -x /bin/plymouth ']'
+ /bin/plymouth --details
+ return 1
+ rc=1
+ echo

+ return 1
+ count=4
++ LC_ALL=C
++ awk '/^#/ {next} $3 ~ /^rpc_pipefs$/ || $3 ~ /^rpc_svc_gss_pipefs$/ {print 
$2}' /proc/mounts
++ sort -r
+ remaining=/var/lib/nfs/rpc_pipefs
+ '[' 4 -gt 0 ']'
+ '[' -z /var/lib/nfs/rpc_pipefs ']'
+ count=3
+ usleep 500000
++ LC_ALL=C
++ awk '/^#/ {next} $3 ~ /^rpc_pipefs$/ || $3 ~ /^rpc_svc_gss_pipefs$/ {print 
$2}' /proc/mounts
++ sort -r
+ remaining=/var/lib/nfs/rpc_pipefs
+ '[' 3 -gt 0 ']'
+ '[' -z /var/lib/nfs/rpc_pipefs ']'
+ count=2
+ usleep 500000
++ sort -r
++ LC_ALL=C
++ awk '/^#/ {next} $3 ~ /^rpc_pipefs$/ || $3 ~ /^rpc_svc_gss_pipefs$/ {print 
$2}' /proc/mounts
+ remaining=/var/lib/nfs/rpc_pipefs
+ '[' 2 -gt 0 ']'
+ '[' -z /var/lib/nfs/rpc_pipefs ']'
+ count=1
+ usleep 500000
++ LC_ALL=C
++ sort -r
++ awk '/^#/ {next} $3 ~ /^rpc_pipefs$/ || $3 ~ /^rpc_svc_gss_pipefs$/ {print 
$2}' /proc/mounts
+ remaining=/var/lib/nfs/rpc_pipefs
+ '[' 1 -gt 0 ']'
+ '[' -z /var/lib/nfs/rpc_pipefs ']'
+ count=0
+ usleep 500000
++ LC_ALL=C
++ sort -r
++ awk '/^#/ {next} $3 ~ /^rpc_pipefs$/ || $3 ~ /^rpc_svc_gss_pipefs$/ {print 
$2}' /proc/mounts
+ init: rc main process (14263) killed by KILL signal

Original comment by trentdho...@gmail.com on 29 Feb 2012 at 12:02

[GoogleCodeExporter]

adding ps to halt script I see:


2012-02-28 16:18:30 root      14317  0.0  0.0  11360  1324 ?        Ss   16:18  
 0:00 /bin/bash -x /etc/rc6.d/S01reboot start

Then at the hang you see this process get killed.


2012-02-28 16:18:35 + usleep 500000
2012-02-28 16:18:35 ++ LC_ALL=C
2012-02-28 16:18:35 ++ awk '/^#/ {next} $3 ~ /^rpc_pipefs$/ || $3 ~ 
/^rpc_svc_gss_pipefs$/ {print $2}' /proc/mounts
2012-02-28 16:18:35 ++ sort -init: rc main process (14317) killed by KILL signal

Original comment by trentdho...@gmail.com on 29 Feb 2012 at 12:20

[GoogleCodeExporter]

Playing around on a minimal rhel6 based system, I can get shutdown -h now to 
work if I edit this stanza in /etc/init.d/halt:

LANG=C __umount_loop '$2 ~ /^\/$|^\/proc|^\/dev/{next}
        $3 == "tmpfs" || $3 == "proc" {print $2 ; next}
        $3 ~ /(loopfs|autofs|nfs|cifs|smbfs|ncpfs|sysfs)/ {next}
        /(^none|^\/dev\/ram|^\/dev\/root$)/ {next}
        {print $2}' /proc/mounts \
        $"Unmounting file systems: " \
        $"Unmounting file systems (retry): " \
        -f

To remove the '$3 == "tmpfs" ||' part of the second line.

Original comment by garlick....@gmail.com on 6 Mar 2012 at 11:13

[GoogleCodeExporter]

Looking at the function __umount_loop in /etc/init.d/functions, it runs fuser 
-k on the file systems it is trying to unmount, so I think the shutdown script 
is killing itself (as you noted) because /etc is a tmpfs that it wants badly to 
unmount.

Original comment by garlick....@gmail.com on 6 Mar 2012 at 11:27

[GoogleCodeExporter]

Here's a script that will edit /etc/init.d/halt

Original comment by garlick....@gmail.com on 6 Mar 2012 at 11:28

Attachments:

• edithalt.sh

[GoogleCodeExporter]

Probably the nfsroot package shouldn't poke at the initscripts package.
Can this script or similar be incorporated into root image prep scripts?

Original comment by garlick....@gmail.com on 6 Mar 2012 at 11:41

• Changed state: Started

[GoogleCodeExporter]

Original comment by garlick....@gmail.com on 7 Mar 2012 at 11:21

[GoogleCodeExporter]

Script is checked into git under examples, but not distributed with package.
Calling it fixed.

Original comment by garlick....@gmail.com on 20 Mar 2012 at 6:40

• Changed state: Fixed

[GoogleCodeExporter]

Reopening as Trent says this doesn't work on llnl systems.

Original comment by garlick....@gmail.com on 22 Mar 2012 at 3:50

• Changed state: Started

[GoogleCodeExporter]

Re-testing various means of shutting down after running the proposed script,
on my test box != production systems Trent is testing:

-bash-4.1# ./edithalt.sh
-bash-4.1# /sbin/halt

Broadcast message from root@clarence
        (/dev/ttyS0) at 9:16 ...

The system is going down for halt NOW!
init: tty (/dev/tty1) main process (2597) killed by TERM signal
init: tty (/dev/tty2) main process (2599) killed by TERM signal
init: tty (/dev/tty3) main process (2602) killed by TERM signal
init: tty (/dev/tty4) main process (2604) killed by TERM signal
init: tty (/dev/tty5) main process (2606) killed by TERM signal
init: tty (/dev/tty6) main process (2608) killed by TERM signal
Stopping sshd: [  OK  ]
Stopping automount: [162167.915189] 9pnet: -- v9fs_session_cancel (2816): 
cancel session ffff88007aa8c400
[162167.922967] 9pnet: -- p9_conn_cancel (2816): mux ffff88007a0f8400 err -104
[  OK  ]
Shutting down ntpd: [  OK  ]
Stopping system message bus: [  OK  ]
init: Disconnected from system bus
Stopping rpcbind: [  OK  ]
Sending all processes the TERM signal... [  OK  ]
Sending all processes the KILL signal... [  OK  ]
Saving random seed:  [  OK  ]
Syncing hardware clock to system time [  OK  ]
Unmounting pipe file systems:  [  OK  ]
umount2: Device or resource busy
umount: /etc: device is busy.
        (In some cases useful info about processes that use
         the device is found by lsof(8) or fuser(1))
init: Re-executing /sbin/init
Halting system...
[162175.350266] md: stopping all md devices.
[162175.427068] pcieport 0000:00:1c.5: wake-up capability enabled by ACPI
[162175.444738] pcieport 0000:00:1c.4: wake-up capability enabled by ACPI
[162175.462330] ACPI: Preparing to enter system sleep state S5
[162175.468664] Disabling non-boot CPUs ...
[162175.479101] Power down.

system powered up again, then try#2:

-bash-4.1# ./edithalt.sh
-bash-4.1# /sbin/reboot

Broadcast message from root@clarence
        (/dev/ttyS0) at 9:22 ...

The system is going down for reboot NOW!
init: tty (/dev/tty1) main process (2610) killed by TERM signal
init: tty (/dev/tty2) main process (2612) killed by TERM signal
init: tty (/dev/tty3) main process (2615) killed by TERM signal
init: tty (/dev/tty4) main process (2617) killed by TERM signal
init: tty (/dev/tty5) main process (2619) killed by TERM signal
init: tty (/dev/tty6) main process (2621) killed by TERM signal
Stopping sshd: [  OK  ]
Stopping automount: [   54.214269] 9pnet: -- v9fs_session_cancel (2721): cancel 
session ffff88007a09ac00
[   54.221915] 9pnet: -- p9_conn_cancel (2721): mux ffff88007b67dc00 err -104
[  OK  ]
Shutting down ntpd: [  OK  ]
Stopping system message bus: [  OK  ]
init: Disconnected from system bus
Stopping rpcbind: [  OK  ]
Sending all processes the TERM signal... [  OK  ]
Sending all processes the KILL signal... [  OK  ]
Saving random seed:  [  OK  ]
Syncing hardware clock to system time [  OK  ]
Unmounting pipe file systems:  [  OK  ]
umount2: Device or resource busy
umount: /etc: device is busy.
        (In some cases useful info about processes that use
         the device is found by lsof(8) or fuser(1))
init: Re-executing /sbin/init
Please stand by while rebooting the system...
[   60.701750] md: stopping all md devices.
[   60.778448] Restarting system.
[   60.781585] machine restart
(system rebooted as expected)


.-bash-4.1# ./edithalt.sh
-bash-4.1# /sbin/shutdown -h now

Broadcast message from root@clarence
        (/dev/ttyS0) at 9:25init: tty (/dev/tty1) main process (2612) killed by TERM signal
init: tty (/dev/tty2) main process (2614) killed by TERM signal
init: tty (/dev/tty3) main process (2617) killed by TERM signal
init: tty (/dev/tty4) main process (2619) killed by TERM signal
init: tty (/dev/tty5) main process (2621) killed by TERM signal
init: tty (/dev/tty6) main process (2623) killed by TERM signal
Stopping sshd: [  OK  ]
Stopping automount: [  130.449715] 9pnet: -- v9fs_session_cancel (2722): cancel 
session ffff88007a14b000
[  130.457382] 9pnet: -- p9_conn_cancel (2722): mux ffff88007aa55200 err -104
[  OK  ]
Shutting down ntpd: [  OK  ]
Stopping system message bus: [  OK  ]
init: Disconnected from system bus
Stopping rpcbind: [  OK  ]
Sending all processes the TERM signal... [  OK  ]
Sending all processes the KILL signal... [  OK  ]
Saving random seed:  [  OK  ]
Syncing hardware clock to system time [  OK  ]
Unmounting pipe file systems:  [  OK  ]
umount2: Device or resource busy
umount: /etc: device is busy.
        (In some cases useful info about processes that use
         the device is found by lsof(8) or fuser(1))
init: Re-executing /sbin/init
Halting system...
[  137.489501] md: stopping all md devices.
[  137.565800] pcieport 0000:00:1c.5: wake-up capability enabled by ACPI
[  137.583630] pcieport 0000:00:1c.4: wake-up capability enabled by ACPI
[  137.601254] ACPI: Preparing to enter system sleep state S5
[  137.607551] Disabling non-boot CPUs ...
[  137.617927] Power down.
(system sounded like it powered off but came on again and rebooted...hmm)


-bash-4.1# ./edithalt.sh
-bash-4.1# /sbin/shutdown -r now

Broadcast message from root@clarence
        (/dev/ttyS0) at 9:27init: tty (/dev/tty1) main process (2606) killed by TERM signal
init: tty (/dev/tty2) main process (2608) killed by TERM signal
init: tty (/dev/tty3) main process (2611) killed by TERM signal
init: tty (/dev/tty4) main process (2613) killed by TERM signal
init: tty (/dev/tty5) main process (2615) killed by TERM signal
init: tty (/dev/tty6) main process (2617) killed by TERM signal
Stopping sshd: [  OK  ]
Stopping automount: [   53.138600] 9pnet: -- v9fs_session_cancel (2716): cancel 
session ffff88007abf0000
[   53.146283] 9pnet: -- p9_conn_cancel (2716): mux ffff88007ab43000 err -104
[  OK  ]
Shutting down ntpd: [  OK  ]
Stopping system message bus: [  OK  ]
init: Disconnected from system bus
Stopping rpcbind: [  OK  ]
Sending all processes the TERM signal... [  OK  ]
Sending all processes the KILL signal... [  OK  ]
Saving random seed:  [  OK  ]
Syncing hardware clock to system time [  OK  ]
Unmounting pipe file systems:  [  OK  ]
umount2: Device or resource busy
umount: /etc: device is busy.
        (In some cases useful info about processes that use
         the device is found by lsof(8) or fuser(1))
init: Re-executing /sbin/init
Please stand by while rebooting the system...
[   60.519054] md: stopping all md devices.
[   60.595451] Restarting system.
[   60.598583] machine restart

Original comment by garlick....@gmail.com on 22 Mar 2012 at 4:28