DevOps/Proxmox/can't lock file.md

How to: Fix Proxmox (PVE) “can’t lock file ‘/var/lock/qemu-server/lock-xxx.conf’ – got timeout” (Proxmox can’t shutdown/stop virtual machine) (Proxmox kill/force stop virtual machine)

The Issue

When trying to “Stop” or “Shutdown” virtual machine from Proxmox (PVE) web gui, the “Cluster log” shows

end task UPID:pve:xxxxxxxx:xxxxxxxx:xxxxxxx:qmstop:xxx:root@pam: can’t lock file ‘/var/lock/qemu-server/lock-xxx.conf’ -got timeout

end task UPID:pve:xxxxxxxx:xxxxxxxx:xxxxxxx:qmreboot:xxx:root@pam: VM quit/powerdown failed

Error: can’t lock file ‘/var/lock/qemu-server/lock-202.conf’ – got timeout

Related Question

How to manually (Using terminal/console to) unlock a VM

The Fix

Manually

We can manually delete the lock from following path

/run/lock/qemu-server
 
# The file will be
/var/lock/qemu-server/lock-100.conf
/var/lock/qemu-server/lock-102.conf
...
 
# Make sure only delete the correct one
 
# Manually unlock again (100 is the VM id)
qm unlock 100

Automated via bash script

1 Open Proxmox web gui

2 Click on “>_ Shell” to launch Shell for Proxmox

3 Type following command to create a “killvm.sh” file

nano killvm.sh

4 Select which script you want to use, copy paste the script to nano editor (Here we use the Simple Script as example, you can use Interactive script or Loop script based on your needs)

5 Use Ctrl + X, Y, Enter key to Exit and Save the file

6 Allow the script executing as program

chmod +x killvm.sh

7 Now we can use the bash script to remove locks easily by using following command

# ./killvm.sh VMID
#e.g.
./killvm.sh 202

8 If you would like to use other bash scripts or all of them, save each of them to separate .sh files e.g. killvm-select.sh killvm-loop.sh, the rest steps will be the same as above

Simple script is suitable to normal usage and can be used with other custom scripts

Interactive script is verbose

Copy one or all of following bash scripts to separate name.sh files to use them

Simple script

#!/bin/sh
echo
echo '-----AUTHOR: https://dannyda.com-----'
echo
echo '---Existing locks---'
qm unlock $1
ls -l /run/lock/qemu-server
rm -f /var/lock/qemu-server/lock-$1.conf
qm unlock $1
echo
echo '---Remaining locks---'
ls -l /run/lock/qemu-server

Usage: Assume saved as “killvm.sh” in “/root” folder on PVE, 202 is the VM ID we want to delete

./killvm.sh 202

Interactive script

#!/bin/sh
echo
echo '-----AUTHOR: https://dannyda.com-----'
echo
echo 'Existing lock files'
ls -l /run/lock/qemu-server
read -p 'Enter the VM ID here to delete corresponding lock e.g. 101: ' vmid
qm unlock $vmid
rm -f /var/lock/qemu-server/lock-$vmid.conf
qm unlock $vmid
echo
echo '---Remaining locks---'
ls -l /run/lock/qemu-server

Usage: Assume saved as “killvmi.sh” in “/root” folder on PVE, 203 is the VM ID we want to delete

./killvmi.sh
Enter the VM ID here to delete corresponding lock e.g. 101: 203

Loop script for deleting multiple lock files with ease (Double click on the following script to select all, then copy)

#!/bin/bash
echo
echo '-----AUTHOR: https://dannyda.com-----'
echo
echo 'Existing lock files'
ls -l /run/lock/qemu-server
while read -p 'Enter the VM ID here to delete corresponding lock, press Enter key to exit e.g. 101: ' vmid; do
    echo
    echo '-----AUTHOR: https://dannyda.com-----'
    echo
    echo 'Existing lock files'
    ls -l /run/lock/qemu-server
     
    if [[ "$vmid" = "" ]] || [[ "$vmid" = "q" ]] ;
    then       
        exit
    elif [[ "$vmid" -gt 0 ]] && [[ "$vmid" -lt 1000000000 ]];
    then
        qm unlock $vmid
        rm -f /var/lock/qemu-server/lock-$vmid.conf
        qm unlock $vmid
        ls -l /run/lock/qemu-server
    else
        echo 'Input error, please enter correct VM ID.'
    fi
done

Usage: Assume saved as “rmlock.sh” in “/root” folder on PVE, 201, 202 is the VM ID we want to delete

./rmlock.sh

# Enter the VM ID which we want to delete but locked, then hit Enter key to unlock it
 
# Note
# Press Enter key without entering anything to exit the script, or type q then press Enter key to exit.

Last step

Now we can stop/shutdown or reboot the virtual machine from Proxmox web gui without problem!

Bonus

Simple script (Remove the lock and stop the VM)

#!/bin/sh
echo
echo '-----AUTHOR: https://dannyda.com-----'
echo
echo '---Existing locks---'
qm unlock $1
ls -l /run/lock/qemu-server
rm -f /var/lock/qemu-server/lock-$1.conf
qm unlock $1
echo
echo '---Remaining locks---'
ls -l /run/lock/qemu-server
echo
qm stop $1 && qm status $1

Interactive script (Remove the lock and stop the VM)

#!/bin/sh
echo
echo '-----AUTHOR: https://dannyda.com-----'
echo
echo 'Existing lock files'
ls -l /run/lock/qemu-server
read -p 'Enter the VM ID here to delete corresponding lock e.g. 101: ' vmid
qm unlock $vmid
rm -f /var/lock/qemu-server/lock-$vmid.conf
qm unlock $vmid
echo
echo '---Remaining locks---'
ls -l /run/lock/qemu-server
echo
qm stop $vmid && qm status $vmid

Loop script for deleting multiple lock files with ease (Remove the lock and stop the VM)

#!/bin/bash
echo
echo '-----AUTHOR: https://dannyda.com-----'
echo
echo 'Existing lock files'
ls -l /run/lock/qemu-server
while read -p 'Enter the VM ID here to delete corresponding lock, press Enter key to exit e.g. 101: ' vmid; do
    echo
    echo '-----AUTHOR: https://dannyda.com-----'
    echo
    echo 'Existing lock files'
    ls -l /run/lock/qemu-server
     
    if [[ "$vmid" = "" ]] || [[ "$vmid" = "q" ]] ;
    then       
        exit
    elif [[ "$vmid" -gt 0 ]] && [[ "$vmid" -lt 1000000000 ]];
    then
        qm unlock $vmid
        rm -f /var/lock/qemu-server/lock-$vmid.conf
        qm unlock $vmid
        ls -l /run/lock/qemu-server
        qm stop $vmid && qm status $vmid
    else
        echo 'Input error, please enter correct VM ID.'
    fi
done

Using command to unlock VM

qm unlock <VMID>
 
# e.g.
qm unlock 100