Docker pull fails rootless mode | ApplyLayer exit status 1 stdout: stderr: invalid argument

Hi, I am trying to run docker without root access Run the Docker daemon as a non-root user (Rootless mode) | Docker Documentation
I encountered the issue on both centos 7 and centos 8

Here’s what I did (Steps to reproduce)

useradd datadog -d /l-n/data/datadog
passwd datadog
ssh datadog@p2ctelkl0070

curl -fsSL https://get.docker.com/rootless | sh
# Installing stable version 20.10.7
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100 66.4M  100 66.4M    0     0  51.4M      0  0:00:01  0:00:01 --:--:-- 51.4M
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100 19.1M  100 19.1M    0     0  33.5M      0 --:--:-- --:--:-- --:--:-- 33.5M
+ PATH=/l-n/data/datadog/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin
+ /l-n/data/datadog/bin/dockerd-rootless-setuptool.sh install
[INFO] systemd not detected, dockerd-rootless.sh needs to be started manually:

PATH=/l-n/data/datadog/bin:/sbin:/usr/sbin:$PATH dockerd-rootless.sh

[INFO] Creating CLI context "rootless"
Successfully created context "rootless"

[INFO] Make sure the following environment variables are set (or add them to ~/.bashrc):

# WARNING: systemd not found. You have to remove XDG_RUNTIME_DIR manually on every logout.
export XDG_RUNTIME_DIR=/l-n/data/datadog/.docker/run
export PATH=/l-n/data/datadog/bin:$PATH
export DOCKER_HOST=unix:///l-n/data/datadog/.docker/run/docker.sock

Followed the instructions regarding export and ran dockerd-rootless.sh as regular user

export XDG_RUNTIME_DIR=/l-n/data/datadog/.docker/run
export PATH=/l-n/data/datadog/bin:$PATH
export DOCKER_HOST=unix:///l-n/data/datadog/.docker/run/docker.sock

sh dockerd-rootless.sh
+ case "$1" in
+ '[' -w /l-n/data/datadog/.docker/run ']'
+ '[' -w /l-n/data/datadog ']'
+ rootlesskit=
+ for f in docker-rootlesskit rootlesskit
+ command -v docker-rootlesskit
+ for f in docker-rootlesskit rootlesskit
+ command -v rootlesskit
+ rootlesskit=rootlesskit
+ break
+ '[' -z rootlesskit ']'
+ : ''
+ : ''
+ : builtin
+ : auto
+ : auto
+ net=
+ mtu=
+ '[' -z ']'
+ command -v slirp4netns
+ '[' -z ']'
+ command -v vpnkit
+ net=vpnkit
+ '[' -z ']'
+ mtu=1500
+ '[' -z ']'
+ _DOCKERD_ROOTLESS_CHILD=1
+ export _DOCKERD_ROOTLESS_CHILD
++ id -u
+ '[' 1002 = 0 ']'
+ exec rootlesskit --net=vpnkit --mtu=1500 --slirp4netns-sandbox=auto --slirp4netns-seccomp=auto --disable-host-loopback --port-driver=builtin --copy-up=/etc --copy-up=/run --propagation=rslave dockerd-rootless.sh
+ case "$1" in
+ '[' -w /l-n/data/datadog/.docker/run ']'
+ '[' -w /l-n/data/datadog ']'
+ rootlesskit=
+ for f in docker-rootlesskit rootlesskit
+ command -v docker-rootlesskit
+ for f in docker-rootlesskit rootlesskit
+ command -v rootlesskit
+ rootlesskit=rootlesskit
+ break
+ '[' -z rootlesskit ']'
+ : ''
+ : ''
+ : builtin
+ : auto
+ : auto
+ net=
+ mtu=
+ '[' -z ']'
+ command -v slirp4netns
+ '[' -z ']'
+ command -v vpnkit
+ net=vpnkit
+ '[' -z ']'
+ mtu=1500
+ '[' -z 1 ']'
+ '[' 1 = 1 ']'
+ rm -f /run/docker /run/containerd /run/xtables.lock
+ exec dockerd
INFO[2021-07-17T04:37:47.419954253-04:00] Starting up
WARN[2021-07-17T04:37:47.420020372-04:00] Running in rootless mode. This mode has feature limitations.
INFO[2021-07-17T04:37:47.420025811-04:00] Running with RootlessKit integration
WARN[2021-07-17T04:37:47.428513666-04:00] could not change group /l-n/data/datadog/.docker/run/docker.sock to docker: group docker not found
INFO[2021-07-17T04:37:47.432504582-04:00] libcontainerd: started new containerd process  pid=2303
INFO[2021-07-17T04:37:47.432685186-04:00] parsed scheme: "unix"                         module=grpc
INFO[2021-07-17T04:37:47.432708981-04:00] scheme "unix" not registered, fallback to default scheme  module=grpc
INFO[2021-07-17T04:37:47.432788889-04:00] ccResolverWrapper: sending update to cc: {[{unix:///l-n/data/datadog/.docker/run/docker/containerd/containerd.sock  <nil> 0 <nil>}] <nil> <nil>}  module=grpc
INFO[2021-07-17T04:37:47.432830693-04:00] ClientConn switching balancer to "pick_first"  module=grpc
INFO[2021-07-17T04:37:47.446475757-04:00] starting containerd                           revision=d71fcd7d8303cbf684402823e425e9dd2e99285d version=v1.4.6
INFO[2021-07-17T04:37:47.464791196-04:00] loading plugin "io.containerd.content.v1.content"...  type=io.containerd.content.v1
INFO[2021-07-17T04:37:47.464885878-04:00] loading plugin "io.containerd.snapshotter.v1.aufs"...  type=io.containerd.snapshotter.v1
INFO[2021-07-17T04:37:47.467803678-04:00] skip loading plugin "io.containerd.snapshotter.v1.aufs"...  error="aufs is not supported (modprobe aufs failed: exit status 1 \"modprobe: FATAL: Module aufs not found.\\n\"): skip plugin" type=io.containerd.snapshotter.v1
INFO[2021-07-17T04:37:47.467869081-04:00] loading plugin "io.containerd.snapshotter.v1.btrfs"...  type=io.containerd.snapshotter.v1
INFO[2021-07-17T04:37:47.468529903-04:00] skip loading plugin "io.containerd.snapshotter.v1.btrfs"...  error="path /l-n/data/datadog/.local/share/docker/containerd/daemon/io.containerd.snapshotter.v1.btrfs (xfs) must be a btrfs filesystem to be used with the btrfs snapshotter: skip plugin" type=io.containerd.snapshotter.v1
INFO[2021-07-17T04:37:47.468574277-04:00] loading plugin "io.containerd.snapshotter.v1.devmapper"...  type=io.containerd.snapshotter.v1
WARN[2021-07-17T04:37:47.468660488-04:00] failed to load plugin io.containerd.snapshotter.v1.devmapper  error="devmapper not configured"
INFO[2021-07-17T04:37:47.468684444-04:00] loading plugin "io.containerd.snapshotter.v1.native"...  type=io.containerd.snapshotter.v1
INFO[2021-07-17T04:37:47.468789664-04:00] loading plugin "io.containerd.snapshotter.v1.overlayfs"...  type=io.containerd.snapshotter.v1
INFO[2021-07-17T04:37:47.485062918-04:00] loading plugin "io.containerd.snapshotter.v1.zfs"...  type=io.containerd.snapshotter.v1
INFO[2021-07-17T04:37:47.485775343-04:00] skip loading plugin "io.containerd.snapshotter.v1.zfs"...  error="path /l-n/data/datadog/.local/share/docker/containerd/daemon/io.containerd.snapshotter.v1.zfs must be a zfs filesystem to be used with the zfs snapshotter: skip plugin" type=io.containerd.snapshotter.v1
INFO[2021-07-17T04:37:47.485819674-04:00] loading plugin "io.containerd.metadata.v1.bolt"...  type=io.containerd.metadata.v1
WARN[2021-07-17T04:37:47.485898406-04:00] could not use snapshotter devmapper in metadata plugin  error="devmapper not configured"
INFO[2021-07-17T04:37:47.485927278-04:00] metadata content store policy set             policy=shared
INFO[2021-07-17T04:37:47.488329638-04:00] loading plugin "io.containerd.differ.v1.walking"...  type=io.containerd.differ.v1
INFO[2021-07-17T04:37:47.488381633-04:00] loading plugin "io.containerd.gc.v1.scheduler"...  type=io.containerd.gc.v1
INFO[2021-07-17T04:37:47.488474237-04:00] loading plugin "io.containerd.service.v1.introspection-service"...  type=io.containerd.service.v1
INFO[2021-07-17T04:37:47.488564913-04:00] loading plugin "io.containerd.service.v1.containers-service"...  type=io.containerd.service.v1
INFO[2021-07-17T04:37:47.488592835-04:00] loading plugin "io.containerd.service.v1.content-service"...  type=io.containerd.service.v1
INFO[2021-07-17T04:37:47.488637929-04:00] loading plugin "io.containerd.service.v1.diff-service"...  type=io.containerd.service.v1
INFO[2021-07-17T04:37:47.488672289-04:00] loading plugin "io.containerd.service.v1.images-service"...  type=io.containerd.service.v1
INFO[2021-07-17T04:37:47.488701608-04:00] loading plugin "io.containerd.service.v1.leases-service"...  type=io.containerd.service.v1
INFO[2021-07-17T04:37:47.488732448-04:00] loading plugin "io.containerd.service.v1.namespaces-service"...  type=io.containerd.service.v1
INFO[2021-07-17T04:37:47.488758960-04:00] loading plugin "io.containerd.service.v1.snapshots-service"...  type=io.containerd.service.v1
INFO[2021-07-17T04:37:47.488791643-04:00] loading plugin "io.containerd.runtime.v1.linux"...  type=io.containerd.runtime.v1
INFO[2021-07-17T04:37:47.488989003-04:00] loading plugin "io.containerd.runtime.v2.task"...  type=io.containerd.runtime.v2
INFO[2021-07-17T04:37:47.489159828-04:00] loading plugin "io.containerd.monitor.v1.cgroups"...  type=io.containerd.monitor.v1
INFO[2021-07-17T04:37:47.489994269-04:00] loading plugin "io.containerd.service.v1.tasks-service"...  type=io.containerd.service.v1
INFO[2021-07-17T04:37:47.490083321-04:00] loading plugin "io.containerd.internal.v1.restart"...  type=io.containerd.internal.v1
INFO[2021-07-17T04:37:47.490230737-04:00] loading plugin "io.containerd.grpc.v1.containers"...  type=io.containerd.grpc.v1
INFO[2021-07-17T04:37:47.490268021-04:00] loading plugin "io.containerd.grpc.v1.content"...  type=io.containerd.grpc.v1
INFO[2021-07-17T04:37:47.490294483-04:00] loading plugin "io.containerd.grpc.v1.diff"...  type=io.containerd.grpc.v1
INFO[2021-07-17T04:37:47.490319232-04:00] loading plugin "io.containerd.grpc.v1.events"...  type=io.containerd.grpc.v1
INFO[2021-07-17T04:37:47.490343975-04:00] loading plugin "io.containerd.grpc.v1.healthcheck"...  type=io.containerd.grpc.v1
INFO[2021-07-17T04:37:47.490372224-04:00] loading plugin "io.containerd.grpc.v1.images"...  type=io.containerd.grpc.v1
INFO[2021-07-17T04:37:47.490434503-04:00] loading plugin "io.containerd.grpc.v1.leases"...  type=io.containerd.grpc.v1
INFO[2021-07-17T04:37:47.490460462-04:00] loading plugin "io.containerd.grpc.v1.namespaces"...  type=io.containerd.grpc.v1
INFO[2021-07-17T04:37:47.490484758-04:00] loading plugin "io.containerd.internal.v1.opt"...  type=io.containerd.internal.v1
WARN[2021-07-17T04:37:47.490543903-04:00] failed to load plugin io.containerd.internal.v1.opt  error="mkdir /opt/containerd: permission denied"
INFO[2021-07-17T04:37:47.490566922-04:00] loading plugin "io.containerd.grpc.v1.snapshots"...  type=io.containerd.grpc.v1
INFO[2021-07-17T04:37:47.490595204-04:00] loading plugin "io.containerd.grpc.v1.tasks"...  type=io.containerd.grpc.v1
INFO[2021-07-17T04:37:47.490638465-04:00] loading plugin "io.containerd.grpc.v1.version"...  type=io.containerd.grpc.v1
INFO[2021-07-17T04:37:47.490661778-04:00] loading plugin "io.containerd.grpc.v1.introspection"...  type=io.containerd.grpc.v1
INFO[2021-07-17T04:37:47.491068356-04:00] serving...                                    address=/l-n/data/datadog/.docker/run/docker/containerd/containerd-debug.sock
INFO[2021-07-17T04:37:47.491190060-04:00] serving...                                    address=/l-n/data/datadog/.docker/run/docker/containerd/containerd.sock.ttrpc
INFO[2021-07-17T04:37:47.491293639-04:00] serving...                                    address=/l-n/data/datadog/.docker/run/docker/containerd/containerd.sock
INFO[2021-07-17T04:37:47.491331747-04:00] containerd successfully booted in 0.046443s
WARN[2021-07-17T04:37:47.499438972-04:00] Could not set may_detach_mounts kernel parameter  error="error opening may_detach_mounts kernel config file: open /proc/sys/fs/may_detach_mounts: permission denied"
INFO[2021-07-17T04:37:47.500468218-04:00] parsed scheme: "unix"                         module=grpc
INFO[2021-07-17T04:37:47.500507354-04:00] scheme "unix" not registered, fallback to default scheme  module=grpc
INFO[2021-07-17T04:37:47.500537601-04:00] ccResolverWrapper: sending update to cc: {[{unix:///l-n/data/datadog/.docker/run/docker/containerd/containerd.sock  <nil> 0 <nil>}] <nil> <nil>}  module=grpc
INFO[2021-07-17T04:37:47.500555863-04:00] ClientConn switching balancer to "pick_first"  module=grpc
INFO[2021-07-17T04:37:47.501867141-04:00] parsed scheme: "unix"                         module=grpc
INFO[2021-07-17T04:37:47.501905568-04:00] scheme "unix" not registered, fallback to default scheme  module=grpc
INFO[2021-07-17T04:37:47.501936094-04:00] ccResolverWrapper: sending update to cc: {[{unix:///l-n/data/datadog/.docker/run/docker/containerd/containerd.sock  <nil> 0 <nil>}] <nil> <nil>}  module=grpc
INFO[2021-07-17T04:37:47.501959940-04:00] ClientConn switching balancer to "pick_first"  module=grpc
ERRO[2021-07-17T04:37:47.503723336-04:00] failed to mount overlay: operation not permitted  storage-driver=overlay2
ERRO[2021-07-17T04:37:47.503826120-04:00] exec: "fuse-overlayfs": executable file not found in $PATH  storage-driver=fuse-overlayfs
ERRO[2021-07-17T04:37:47.506433088-04:00] AUFS cannot be used in non-init user namespace  storage-driver=aufs
ERRO[2021-07-17T04:37:47.507091066-04:00] failed to mount overlay: operation not permitted  storage-driver=overlay
ERRO[2021-07-17T04:37:47.507116499-04:00] Failed to built-in GetDriver graph devicemapper /l-n/data/datadog/.local/share/docker
INFO[2021-07-17T04:37:47.531738152-04:00] Loading containers: start.
WARN[2021-07-17T04:37:47.536833733-04:00] Running modprobe bridge br_netfilter failed with message: modprobe: ERROR: could not insert 'bridge': Operation not permitted
modprobe: ERROR: could not insert 'br_netfilter': Operation not permitted
insmod /lib/modules/3.10.0-1160.15.2.el7.x86_64/kernel/net/llc/llc.ko.xz
insmod /lib/modules/3.10.0-1160.15.2.el7.x86_64/kernel/net/llc/llc.ko.xz
, error: exit status 1
INFO[2021-07-17T04:37:47.679150594-04:00] Default bridge (docker0) is assigned with an IP address 172.17.0.0/16. Daemon option --bip can be used to set a preferred IP address
INFO[2021-07-17T04:37:47.792480976-04:00] Loading containers: done.
INFO[2021-07-17T04:37:47.803225171-04:00] Docker daemon                                 commit=b0f5bc3 graphdriver(s)=vfs version=20.10.7
INFO[2021-07-17T04:37:47.803392653-04:00] Daemon has completed initialization
INFO[2021-07-17T04:37:47.848306293-04:00] API listen on /l-n/data/datadog/.docker/run/docker.sock

Here’s where I get an error. Pulling nginx, busybox, mysql, hello-world worked fine except for datadog

docker pull nginx

Using default tag: latest
latest: Pulling from library/nginx
b4d181a07f80: Pull complete
66b1c490df3f: Pull complete
d0f91ae9b44c: Pull complete
baf987068537: Pull complete
6bbc76cbebeb: Pull complete
32b766478bc2: Pull complete
Digest: sha256:353c20f74d9b6aee359f30e8e4f69c3d7eaea2f610681c4a95849a2fd7c497f9
Status: Downloaded newer image for nginx:latest
docker.io/library/nginx:latest

docker pull hello-world

Using default tag: latest
latest: Pulling from library/hello-world
b8dfde127a29: Pull complete
Digest: sha256:df5f5184104426b65967e016ff2ac0bfcd44ad7899ca3bbcf8e44e4461491a9e
Status: Downloaded newer image for hello-world:latest
docker.io/library/hello-world:latest

docker pull mysql

Using default tag: latest
latest: Pulling from library/mysql
b4d181a07f80: Already exists
a462b60610f5: Pull complete
578fafb77ab8: Pull complete
524046006037: Pull complete
d0cbe54c8855: Pull complete
aa18e05cc46d: Pull complete
32ca814c833f: Pull complete
9ecc8abdb7f5: Pull complete
ad042b682e0f: Pull complete
71d327c6bb78: Pull complete
165d1d10a3fa: Pull complete
2f40c47d0626: Pull complete
Digest: sha256:52b8406e4c32b8cf0557f1b74517e14c5393aff5cf0384eff62d9e81f4985d4b
Status: Downloaded newer image for mysql:latest
docker.io/library/mysql:latest

docker pull busybox

Using default tag: latest
latest: Pulling from library/busybox
b71f96345d44: Pull complete
Digest: sha256:0f354ec1728d9ff32edcd7d1b8bbdfc798277ad36120dc3dc683be44524c8b60
Status: Downloaded newer image for busybox:latest
docker.io/library/busybox:latest

docker pull datadog/synthetics-private-location-worker

Using default tag: latest
latest: Pulling from datadog/synthetics-private-location-worker
f219f9fec81c: Pull complete
9189d0b37a86: Pull complete
bebec8789df0: Pull complete
c86c9b0c843c: Pull complete
908abd1fef24: Pull complete
526a7cb48cad: Pull complete
b9fa463cb987: Extracting [==================================================>]  67.66MB/67.66MB
dc3df4a5b015: Download complete
3de86ae225ca: Download complete
501d8d9cee8b: Download complete
eebd08781363: Download complete
f7377177dba1: Download complete
35bf2693fa0b: Download complete
0cf69c2f2fc2: Download complete
f1fb35cd721f: Download complete
f565dbcac4c7: Download complete
823c561aa5c3: Download complete
8056a69f9ff7: Download complete
e9471e8aa13a: Download complete
8e9a00d021ba: Download complete
4f4bf9f6fe4e: Download complete
37395fbe56b0: Download complete
ede13761d4ab: Download complete
a35c977559db: Download complete
7dee4b6ed62a: Download complete
d07da47084df: Download complete
76a98f4593a4: Download complete
2edac6c6ead2: Download complete
8cbb1be9c0fc: Download complete
42d9c8f9b1d7: Download complete
failed to register layer: ApplyLayer exit status 1 stdout:  stderr: invalid argument

docker info

Client:
 Context:    default
 Debug Mode: false

Server:
 Containers: 4
  Running: 0
  Paused: 0
  Stopped: 0
 Images: 0
 Server Version: 20.10.7
 Storage Driver: vfs
 Logging Driver: json-file
 Cgroup Driver: none
 Cgroup Version: 1
 Plugins:
  Volume: local
  Network: bridge host ipvlan macvlan null overlay
  Log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog
 Swarm: inactive
 Runtimes: io.containerd.runc.v2 io.containerd.runtime.v1.linux runc
 Default Runtime: runc
 Init Binary: docker-init
 containerd version: d71fcd7d8303cbf684402823e425e9dd2e99285d
 runc version: b9ee9c6314599f1b4a7f497e1f1f856fe433d3b7
 init version: de40ad0
 Security Options:
  seccomp
   Profile: default
  rootless
 Kernel Version: 3.10.0-1160.15.2.el7.x86_64
 Operating System: CentOS Linux 7 (Core)
 OSType: linux
 Architecture: x86_64
 CPUs: 8
 Total Memory: 62.76GiB
 Name: p2ctelkl0070
 ID: WWZN:2FVB:HYW6:KIWQ:B27Y:BJMZ:3C7K:U5XQ:H4VG:KC4I:REKP:LVYJ
 Docker Root Dir: /l-n/data/datadog/.local/share/docker
 Debug Mode: false
 Registry: https://index.docker.io/v1/
 Labels:
 Experimental: false
 Insecure Registries:
  127.0.0.0/8
 Live Restore Enabled: false
 Product License: Community Engine

WARNING: Running in rootless-mode without cgroups. To enable cgroups in rootless-mode, you need to boot the system in cgroup v2 mode.
WARNING: bridge-nf-call-iptables is disabled
WARNING: bridge-nf-call-ip6tables is disabled

docker version

Client:
 Version:           20.10.7
 API version:       1.41
 Go version:        go1.13.15
 Git commit:        f0df350
 Built:             Wed Jun  2 11:51:04 2021
 OS/Arch:           linux/amd64
 Context:           default
 Experimental:      true

Server: Docker Engine - Community
 Engine:
  Version:          20.10.7
  API version:      1.41 (minimum version 1.12)
  Go version:       go1.13.15
  Git commit:       b0f5bc3
  Built:            Wed Jun  2 11:55:29 2021
  OS/Arch:          linux/amd64
  Experimental:     false
 containerd:
  Version:          v1.4.6
  GitCommit:        d71fcd7d8303cbf684402823e425e9dd2e99285d
 runc:
  Version:          1.0.0-rc95
  GitCommit:        b9ee9c6314599f1b4a7f497e1f1f856fe433d3b7
 docker-init:
  Version:          0.19.0
  GitCommit:        de40ad0

lsb_release

LSB Version:    :core-4.1-amd64:core-4.1-noarch

uname -a

Linux p2ctelkl0070 3.10.0-1160.15.2.el7.x86_64 #1 SMP Wed Feb 3 15:06:38 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux

cat /etc/*release

CentOS Linux release 7.9.2009 (Core)
NAME="CentOS Linux"
VERSION="7 (Core)"
ID="centos"
ID_LIKE="rhel fedora"
VERSION_ID="7"
PRETTY_NAME="CentOS Linux 7 (Core)"
ANSI_COLOR="0;31"
CPE_NAME="cpe:/o:centos:centos:7"
HOME_URL="https://www.centos.org/"
BUG_REPORT_URL="https://bugs.centos.org/"

CENTOS_MANTISBT_PROJECT="CentOS-7"
CENTOS_MANTISBT_PROJECT_VERSION="7"
REDHAT_SUPPORT_PRODUCT="centos"
REDHAT_SUPPORT_PRODUCT_VERSION="7"

CentOS Linux release 7.9.2009 (Core)
CentOS Linux release 7.9.2009 (Core)

I also tried this on Ubuntu 20.04 and still got the same error.

$ docker pull datadog/synthetics-private-location-worker

Using default tag: latest
latest: Pulling from datadog/synthetics-private-location-worker
f219f9fec81c: Pulling fs layer
9189d0b37a86: Pull complete
bebec8789df0: Pull complete
c86c9b0c843c: Pull complete
908abd1fef24: Pull complete
526a7cb48cad: Pull complete
b9fa463cb987: Extracting [==================================================>]  67.66MB/67.66MB
dc3df4a5b015: Download complete
3de86ae225ca: Download complete
501d8d9cee8b: Download complete
eebd08781363: Download complete
f7377177dba1: Download complete
35bf2693fa0b: Download complete
0cf69c2f2fc2: Download complete
f1fb35cd721f: Download complete
f565dbcac4c7: Download complete
823c561aa5c3: Download complete
8056a69f9ff7: Download complete
e9471e8aa13a: Download complete
8e9a00d021ba: Download complete
4f4bf9f6fe4e: Download complete
37395fbe56b0: Download complete
ede13761d4ab: Download complete
a35c977559db: Download complete
7dee4b6ed62a: Download complete
d07da47084df: Download complete
76a98f4593a4: Download complete
2edac6c6ead2: Download complete
8cbb1be9c0fc: Download complete
42d9c8f9b1d7: Download complete
failed to register layer: ApplyLayer exit status 1 stdout:  stderr: invalid argument

$ cat /etc/*release

DISTRIB_ID=Ubuntu
DISTRIB_RELEASE=20.04
DISTRIB_CODENAME=focal
DISTRIB_DESCRIPTION="Ubuntu 20.04.2 LTS"
NAME="Ubuntu"
VERSION="20.04.2 LTS (Focal Fossa)"
ID=ubuntu
ID_LIKE=debian
PRETTY_NAME="Ubuntu 20.04.2 LTS"
VERSION_ID="20.04"
HOME_URL="https://www.ubuntu.com/"
SUPPORT_URL="https://help.ubuntu.com/"
BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/"
PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy"
VERSION_CODENAME=focal
UBUNTU_CODENAME=focal

$ systemctl --user status docker

● docker.service - Docker Application Container Engine (Rootless)
     Loaded: loaded (/home/testuser1/.config/systemd/user/docker.service; enabled; vendor preset: enabled)
     Active: active (running) since Mon 2021-07-19 11:02:57 PST; 10min ago
       Docs: https://docs.docker.com/go/rootless/
   Main PID: 7525 (rootlesskit)
     CGroup: /user.slice/user-1001.slice/user@1001.service/docker.service
             ├─7525 rootlesskit --net=vpnkit --mtu=1500 --slirp4netns-sandbox=auto --slirp4netns-seccomp=auto --disable-host-loopback --port-driver=builtin --copy-up=/etc --copy-up=/run --propagation=rslave />
             ├─7530 /proc/self/exe --net=vpnkit --mtu=1500 --slirp4netns-sandbox=auto --slirp4netns-seccomp=auto --disable-host-loopback --port-driver=builtin --copy-up=/etc --copy-up=/run --propagation=rslav>
             ├─7538 vpnkit --ethernet /tmp/rootlesskit985879234/vpnkit-ethernet.sock --mtu 1500 --host-ip 0.0.0.0
             ├─7559 dockerd
             └─7568 containerd --config /run/user/1001/docker/containerd/containerd.toml --log-level info

Jul 19 11:03:02 DESKTOP-57483 dockerd-rootless.sh[7559]: time="2021-07-19T11:03:02.315046928+08:00" level=info msg="Docker daemon" commit=b0f5bc3 graphdriver(s)=overlay2 version=20.10.7
Jul 19 11:03:02 DESKTOP-57483 dockerd-rootless.sh[7559]: time="2021-07-19T11:03:02.315336927+08:00" level=info msg="Daemon has completed initialization"
Jul 19 11:03:02 DESKTOP-57483 dockerd-rootless.sh[7559]: time="2021-07-19T11:03:02.563730699+08:00" level=info msg="API listen on /run/user/1001/docker.sock"
Jul 19 11:06:55 DESKTOP-57483 dockerd-rootless.sh[7559]: time="2021-07-19T11:06:55.326194936+08:00" level=info msg="Attempting next endpoint for pull after error: failed to register layer: ApplyLayer exit sta>
Jul 19 11:06:55 DESKTOP-57483 dockerd-rootless.sh[7559]: time="2021-07-19T11:06:55.331567115+08:00" level=info msg="Layer sha256:0ecac0c0759eea69137be4ba7a347b92345af1f92c04cd0ae7db10ae9e37f73f cleaned up"
Jul 19 11:06:55 DESKTOP-57483 dockerd-rootless.sh[7559]: time="2021-07-19T11:06:55.334186304+08:00" level=info msg="Layer sha256:93541167fb70ecb11cc23ed53da19f15f4543dec9b10f73e43bd6a26428a6964 cleaned up"
Jul 19 11:06:55 DESKTOP-57483 dockerd-rootless.sh[7559]: time="2021-07-19T11:06:55.352020732+08:00" level=info msg="Layer sha256:c2b16d0662e186ab74c3ae310efffef8be038a841748c53a30bfcd35e2ae1870 cleaned up"
Jul 19 11:06:55 DESKTOP-57483 dockerd-rootless.sh[7559]: time="2021-07-19T11:06:55.506976608+08:00" level=info msg="Layer sha256:81d8f00421b98eaba303b603f04273469c08fbb657203b571c530f73678aa6e1 cleaned up"
Jul 19 11:06:55 DESKTOP-57483 dockerd-rootless.sh[7559]: time="2021-07-19T11:06:55.508855101+08:00" level=info msg="Layer sha256:6968f95899c00fb6c948c7fec763ce4ca6f49bbd0c73cdf578c036f90dc60412 cleaned up"
Jul 19 11:06:55 DESKTOP-57483 dockerd-rootless.sh[7559]: time="2021-07-19T11:06:55.609265096+08:00" level=info msg="Layer sha256:b7ba449969d62d3789f243e84ba8a6fdc1521d9570f1c4459b9a1a70612ed468 cleaned up"

As per the Datadog Helpdesk “recommend not using Docker rootless as it increases the attack surface for local privilege escalation given its reliance on `unprivileged user namespace”.

On investigation, it turned out the issue is tied to the user namespaces of linux and permissions on privileged files. Unfortunately, as we have no workaround for this, it means we don’t currently support Docker Rootless. On a side note, our team recommend not using Docker rootless as it increases the attack surface for local privilege escalation given its reliance on unprivileged user namespace ​.

@arislawrence this is a CRAZY answer, running docker without rootless, gives docker users running containers root access on the box where they are running. Rootless is new and is being designed to help alleviate that flaw in docker root install mode.
I have been running rootless for a while now, and just encountered this error, something new has happened with latest centos kernel update and other security update files.

We are still looking for a real answer to this problem.

I had just installed mcafee and found one other article that said mcafee was the culprit, so we are going down that path to resolve, no solution yet, but this IS progress.