Colima and docker instability: randomly freezing, requiring reset

[jacksonhull]

Description

I've been trying to troubleshoot an issue with Colima I've been seeing for weeks.

Colima will install and start fine, and I can start my ~20 containers without issue. However, mostly randomly, colima and docker will suddenly be completely unresponsive and containers will stop.

The only remedy is to restart the mac and run colima delete and start over.

I'm running on an Intel i9 with 32GB RAM on MacOS Sonoma 14.4.1 (23E224).

I'm hoping someone has been able to resolve the instability issues with colima running docker on a Mac. The performance of docker on colima is so much better than Docker Desktop and I'd rather not have to go back to it!

I'm happy to further troubleshoot, but hd.stderr.log isn't providing many clues.

Version

colima version 0.6.9
git commit: c3a31ed

runtime: docker
arch: x86_64
client: v26.1.3
server: v26.1.1
limactl version 0.22.0
qemu-img version 9.0.0

Operating System

• macOS Intel <= 13 (Ventura)
• macOS Intel >= 14 (Sonoma)
• Apple Silicon <= 13 (Ventura)
• Apple Silicon >= 14 (Sonoma)
• Linux

Output of colima status

INFO[0000] colima is running using macOS Virtualization.Framework
INFO[0000] arch: x86_64
INFO[0000] runtime: docker
INFO[0000] mountType: virtiofs
INFO[0000] socket: unix:///Users/johndoe/.colima/default/docker.sock

Reproduction Steps

1. Reset colima and start using a default.yaml
2. Runs fine for a few days
3. Containers stop and colima and docker freeze, or fail.

Expected behaviour

Run smoothly until manual restart.

Additional context

Default.yaml:

# Number of CPUs to be allocated to the virtual machine.
# Default: 2
cpu: 16

# Size of the disk in GiB to be allocated to the virtual machine.
# NOTE: changing this has no effect after the virtual machine has been created.
# Default: 60
disk: 120

# Size of the memory in GiB to be allocated to the virtual machine.
# Default: 2
memory: 24

# Architecture of the virtual machine (x86_64, aarch64, host).
# Default: host
arch: x86_64

# Container runtime to be used (docker, containerd).
# Default: docker
runtime: docker

# Set custom hostname for the virtual machine.
# Default: colima
#          colima-profile_name for other profiles
hostname: colima

# Kubernetes configuration for the virtual machine.
kubernetes:
  # Enable kubernetes.
  # Default: false
  enabled: false
  
  # Kubernetes version to use.
  # This needs to exactly match a k3s version https://github.com/k3s-io/k3s/releases
  # Default: latest stable release
  version: v1.28.3+k3s2
  
  # Additional args to pass to k3s https://docs.k3s.io/cli/server
  # Default: traefik is disabled
  k3sArgs:
    - --disable=traefik

# Auto-activate on the Host for client access.
# Setting to true does the following on startup
#  - sets as active Docker context (for Docker runtime).
#  - sets as active Kubernetes context (if Kubernetes is enabled).
# Default: true
autoActivate: true

# Network configurations for the virtual machine.
network:
  # Assign reachable IP address to the virtual machine.
  # NOTE: this is currently macOS only and ignored on Linux.
  # Default: false
  address: false
  
  # Custom DNS resolvers for the virtual machine.
  #
  # EXAMPLE
  # dns: [8.8.8.8, 1.1.1.1]
  #
  # Default: []
  dns: []
  
  # DNS hostnames to resolve to custom targets using the internal resolver.
  # This setting has no effect if a custom DNS resolver list is supplied above.
  # It does not configure the /etc/hosts files of any machine or container.
  # The value can be an IP address or another host.
  #
  # EXAMPLE
  # dnsHosts:
  #   example.com: 1.2.3.4
  dnsHosts: {}

# ===================================================================== #
# ADVANCED CONFIGURATION
# ===================================================================== #

# Forward the host's SSH agent to the virtual machine.
# Default: false
forwardAgent: false

# Docker daemon configuration that maps directly to daemon.json.
# https://docs.docker.com/engine/reference/commandline/dockerd/#daemon-configuration-file.
# NOTE: some settings may affect Colima's ability to start docker. e.g. `hosts`.
#
# EXAMPLE - disable buildkit
# docker:
#   features:
#     buildkit: false
#
# EXAMPLE - add insecure registries
# docker:
#   insecure-registries:
#     - myregistry.com:5000
#     - host.docker.internal:5000
#
# Colima default behaviour: buildkit enabled
# Default: {}
docker: {}

# Virtual Machine type (qemu, vz)
# NOTE: this is macOS 13 only. For Linux and macOS <13.0, qemu is always used.
#
# vz is macOS virtualization framework and requires macOS 13
#
# Default: qemu
vmType: vz

# Utilise rosetta for amd64 emulation (requires m1 mac and vmType `vz`)
# Default: false
rosetta: false

# Volume mount driver for the virtual machine (virtiofs, 9p, sshfs).
#
# virtiofs is limited to macOS and vmType `vz`. It is the fastest of the options.
#
# 9p is the recommended and the most stable option for vmType `qemu`.
#
# sshfs is faster than 9p but the least reliable of the options (when there are lots
# of concurrent reads or writes).
#
# Default: virtiofs (for vz), sshfs (for qemu)
mountType: virtiofs

# Propagate inotify file events to the VM.
# NOTE: this is experimental.
mountInotify: true

# The CPU type for the virtual machine (requires vmType `qemu`).
# Options available for host emulation can be checked with: `qemu-system-$(arch) -cpu help`.
# Instructions are also supported by appending to the cpu type e.g. "qemu64,+ssse3".
# Default: host
cpuType: ""

# Custom provision scripts for the virtual machine.
# Provisioning scripts are executed on startup and therefore needs to be idempotent.# Default: []
provision: []

# Modify ~/.ssh/config automatically to include a SSH config for the virtual machine.
# SSH config will still be generated in ~/.colima/ssh_config regardless.
# Default: true
sshConfig: true

# Configure volume mounts for the virtual machine.
# Colima mounts user's home directory by default to provide a familiar
# user experience.
# Colima default behaviour: $HOME and /tmp/colima are mounted as writable.
# Default: []
mounts:
  - location: /Volumes/Drive
    writable: true
# Environment variables for the virtual machine.
#
# EXAMPLE
# env:
#   KEY: value
#   ANOTHER_KEY: another value
#
# Default: {}
env: {}`

I note the following is almost always present in ha.stderr.log but I can't correlate it directly to the freezes.

{"error":"failed to run [ssh -F /dev/null -o IdentityFile=\"/Users/johndoe/.colima/_lima/_config/user\" -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null -o NoHostAuthenticationForLocalhost=yes -o GSSAPIAuthentication=no -o PreferredAuthentications=publickey -o Compression=no -o BatchMode=yes -o IdentitiesOnly=yes -o Ciphers=\"^aes128-gcm@openssh.com,aes256-gcm@openssh.com\" -o User=johndoe -o ControlMaster=auto -o ControlPath=\"/Users/johndoe/.colima/_lima/colima/ssh.sock\" -o ControlPersist=yes -T -O forward -L 0.0.0.0:8009:[::]:8009 -N -f -p 49304 127.0.0.1 --]: \"\": exit status 255","level":"warning","msg":"failed to set up forwarding tcp port 8009 (negligible if already forwarded)","time":"2024-05-12T09:55:03+01:00"}