I was running my program in the background of doing work, and I looked over to see my program had stopped in my terminal and logged me out of the server. I reconnected back to the server (DGX) and then attempted to remount my docker and got this error, which I have not run into before:
docker: Error response from daemon: OCI runtime create failed: container_linux.go:380: starting container process caused: process_linux.go:545: container init caused: Running hook #0:: error running hook: exit status 1, stdout: , stderr: nvidia-container-cli: initialization error: nvml error: driver not loaded: unknown.
ERRO error waiting for container: context canceled
Any tips and help would be much appreciated due to my job being on a timeline to get brain scans mapped and analyzed! Also, as much explanation as possible is appreciated as well. Thank you!