Filesystem errors while creating directories #25419

altendky · 2020-10-22T01:40:32Z

altendky
Oct 22, 2020

Starting in the past day or so, when trying to destination.parent.mkdir(parents=True, exist_ok=True) I’m getting intermittent failures in GitHub Actions on both Windows and Linux (1 in 4’ish jobs, so every build with the present limited matrix). Retrying 5x doesn’t seem to help. Thoughts on debugging or fixing?

        try:
            destination.parent.mkdir(parents=True, exist_ok=True)
        except (NotADirectoryError, FileExistsError):
            print('destination_root', destination_root)
            print('self.source', self.source)
            print('self.destination', self.destination)
            print('destination', destination)
            print('destination.parent', destination.parent, destination.parent.is_dir(), destination.parent.is_file())
            print('destination.parent.iterdir()', list(destination.parent.iterdir()))
            raise

https://github.com/altendky/pyqt5-tools/runs/1288207768?check_suite_focus=true

2020-10-21T17:35:25.9404094Z     ----<==== Execute Copy Actions ====>----
2020-10-21T17:35:25.9406041Z Traceback (most recent call last):
2020-10-21T17:35:25.9407455Z destination_root /__w/pyqt5-tools/pyqt5-tools/build/lib/pyqtplugins/Qt
2020-10-21T17:35:25.9408958Z   File "./build.py", line 171, in copy
2020-10-21T17:35:25.9410180Z self.source /__w/pyqt5-tools/pyqt5-tools/build/pyqtplugins-rtp6hwhg/qt/5.15.1/gcc_64/plugins/platforms/libqxcb.so
2020-10-21T17:35:25.9412149Z     destination.parent.mkdir(parents=True, exist_ok=True)
2020-10-21T17:35:25.9413024Z self.destination plugins/platforms/libqxcb.so
2020-10-21T17:35:25.9414483Z   File "/usr/local/lib/python3.8/pathlib.py", line 1287, in mkdir
2020-10-21T17:35:25.9415709Z destination /__w/pyqt5-tools/pyqt5-tools/build/lib/pyqtplugins/Qt/plugins/platforms/libqxcb.so
2020-10-21T17:35:25.9417338Z     self._accessor.mkdir(self, mode)
2020-10-21T17:35:25.9418575Z destination.parent /__w/pyqt5-tools/pyqt5-tools/build/lib/pyqtplugins/Qt/plugins/platforms False False
2020-10-21T17:35:25.9421006Z NotADirectoryError: [Errno 20] Not a directory: '/__w/pyqt5-tools/pyqt5-tools/build/lib/pyqtplugins/Qt/plugins/platforms'
2020-10-21T17:35:25.9422402Z destination_root /__w/pyqt5-tools/pyqt5-tools/build/lib/pyqtplugins/Qt
2020-10-21T17:35:25.9423673Z 
2020-10-21T17:35:25.9424801Z self.source /__w/pyqt5-tools/pyqt5-tools/build/pyqtplugins-rtp6hwhg/qt/5.15.1/gcc_64/plugins/platforms/libqxcb.so
2020-10-21T17:35:25.9426680Z During handling of the above exception, another exception occurred:
2020-10-21T17:35:25.9427501Z self.destination plugins/platforms/libqxcb.so
2020-10-21T17:35:25.9429527Z 
2020-10-21T17:35:25.9429941Z Traceback (most recent call last):
2020-10-21T17:35:25.9430490Z   File "./build.py", line 47, in build_packages
2020-10-21T17:35:25.9430896Z     main(
2020-10-21T17:35:25.9431270Z   File "./build.py", line 611, in main
2020-10-21T17:35:25.9432068Z     return build(configuration=configuration)
2020-10-21T17:35:25.9432667Z   File "./build.py", line 784, in build
2020-10-21T17:35:25.9433235Z     action.copy(destination_root=reference)
2020-10-21T17:35:25.9434571Z   File "/tmp/pep517-build-env-3ubiri1j/lib/python3.8/site-packages/tenacity/__init__.py", line 329, in wrapped_f
2020-10-21T17:35:25.9435475Z     return self.call(f, *args, **kw)
2020-10-21T17:35:25.9436614Z   File "/tmp/pep517-build-env-3ubiri1j/lib/python3.8/site-packages/tenacity/__init__.py", line 409, in call
2020-10-21T17:35:25.9437513Z     do = self.iter(retry_state=retry_state)
2020-10-21T17:35:25.9438668Z   File "/tmp/pep517-build-env-3ubiri1j/lib/python3.8/site-packages/tenacity/__init__.py", line 368, in iter
2020-10-21T17:35:25.9439534Z     raise retry_exc.reraise()
2020-10-21T17:35:25.9440695Z   File "/tmp/pep517-build-env-3ubiri1j/lib/python3.8/site-packages/tenacity/__init__.py", line 186, in reraise
2020-10-21T17:35:25.9441643Z     raise self.last_attempt.result()
2020-10-21T17:35:25.9442363Z   File "/usr/local/lib/python3.8/concurrent/futures/_base.py", line 432, in result
2020-10-21T17:35:25.9443009Z     return self.__get_result()
2020-10-21T17:35:25.9443668Z   File "/usr/local/lib/python3.8/concurrent/futures/_base.py", line 388, in __get_result
2020-10-21T17:35:25.9444329Z     raise self._exception
2020-10-21T17:35:25.9445428Z   File "/tmp/pep517-build-env-3ubiri1j/lib/python3.8/site-packages/tenacity/__init__.py", line 412, in call
2020-10-21T17:35:25.9446273Z     result = fn(*args, **kwargs)
2020-10-21T17:35:25.9446721Z   File "./build.py", line 178, in copy
2020-10-21T17:35:25.9447911Z     print('destination.parent.iterdir()', list(destination.parent.iterdir()))
2020-10-21T17:35:25.9449001Z   File "/usr/local/lib/python3.8/pathlib.py", line 1121, in iterdir
2020-10-21T17:35:25.9449687Z     for name in self._accessor.listdir(self):
2020-10-21T17:35:25.9451335Z NotADirectoryError: [Errno 20] Not a directory: '/__w/pyqt5-tools/pyqt5-tools/build/lib/pyqtplugins/Qt/plugins/platforms'
2020-10-21T17:35:25.9452920Z error: [Errno 20] Not a directory: '/__w/pyqt5-tools/pyqt5-tools/build/lib/pyqtplugins/Qt/plugins/platforms'
2020-10-21T17:35:25.9454317Z destination /__w/pyqt5-tools/pyqt5-tools/build/lib/pyqtplugins/Qt/plugins/platforms/libqxcb.so
2020-10-21T17:35:25.9455924Z destination.parent /__w/pyqt5-tools/pyqt5-tools/build/lib/pyqtplugins/Qt/plugins/platforms False False
2020-10-21T17:35:25.9457231Z destination_root /__w/pyqt5-tools/pyqt5-tools/build/lib/pyqtplugins/Qt
2020-10-21T17:35:25.9458643Z self.source /__w/pyqt5-tools/pyqt5-tools/build/pyqtplugins-rtp6hwhg/qt/5.15.1/gcc_64/plugins/platforms/libqxcb.so
2020-10-21T17:35:25.9459724Z self.destination plugins/platforms/libqxcb.so
2020-10-21T17:35:25.9460928Z destination /__w/pyqt5-tools/pyqt5-tools/build/lib/pyqtplugins/Qt/plugins/platforms/libqxcb.so
2020-10-21T17:35:25.9462420Z destination.parent /__w/pyqt5-tools/pyqt5-tools/build/lib/pyqtplugins/Qt/plugins/platforms False False
2020-10-21T17:35:25.9463737Z destination_root /__w/pyqt5-tools/pyqt5-tools/build/lib/pyqtplugins/Qt
2020-10-21T17:35:25.9465151Z self.source /__w/pyqt5-tools/pyqt5-tools/build/pyqtplugins-rtp6hwhg/qt/5.15.1/gcc_64/plugins/platforms/libqxcb.so
2020-10-21T17:35:25.9466233Z self.destination plugins/platforms/libqxcb.so
2020-10-21T17:35:25.9467442Z destination /__w/pyqt5-tools/pyqt5-tools/build/lib/pyqtplugins/Qt/plugins/platforms/libqxcb.so
2020-10-21T17:35:25.9468921Z destination.parent /__w/pyqt5-tools/pyqt5-tools/build/lib/pyqtplugins/Qt/plugins/platforms False False
2020-10-21T17:35:25.9470217Z destination_root /__w/pyqt5-tools/pyqt5-tools/build/lib/pyqtplugins/Qt
2020-10-21T17:35:25.9471633Z self.source /__w/pyqt5-tools/pyqt5-tools/build/pyqtplugins-rtp6hwhg/qt/5.15.1/gcc_64/plugins/platforms/libqxcb.so
2020-10-21T17:35:25.9474448Z self.destination plugins/platforms/libqxcb.so
2020-10-21T17:35:25.9475716Z destination /__w/pyqt5-tools/pyqt5-tools/build/lib/pyqtplugins/Qt/plugins/platforms/libqxcb.so
2020-10-21T17:35:25.9487394Z destination.parent /__w/pyqt5-tools/pyqt5-tools/build/lib/pyqtplugins/Qt/plugins/platforms False False

More log at https://gist.github.com/altendky/b873e07c9a538102414c1ea42290dd4e

Answered by altendky

Oct 25, 2020

This is embarrasing, but so it goes. I was building a collection of copy actions then batch executing them. One of them incorrectly specified the target directory path rather than file path. If it ran ‘first’ then the file was copied where I wanted the directory. Otherwise the file was copied into the directory as desired. The inconsistency came in because I stored the actions in a set to deduplicate them. I have corrected the target paths and also sort the copy actions prior to executing them to avoid the inconsistency.

Sorry for the noise.

View full answer

altendky · 2020-10-22T03:18:36Z

altendky
Oct 22, 2020
Author

Adding a df -h call while handling the exception shows seemingly available disk space, though I don’t know if any restrictions are implemented via quotas or… whatever else.

https://github.com/altendky/pyqt5-tools/pull/68/checks?check_run_id=1290116682

Filesystem      Size  Used Avail Use% Mounted on
overlay          84G   64G   20G  77% /
tmpfs            64M     0   64M   0% /dev
tmpfs           3.4G     0  3.4G   0% /sys/fs/cgroup
shm              64M     0   64M   0% /dev/shm
/dev/sdb1        84G   64G   20G  77% /__t
tmpfs           696M  764K  695M   1% /run/docker.sock
tmpfs           3.4G     0  3.4G   0% /proc/acpi
tmpfs           3.4G     0  3.4G   0% /proc/scsi
tmpfs           3.4G     0  3.4G   0% /sys/firmware

0 replies

BrightRan · 2020-10-23T06:02:12Z

BrightRan
Oct 23, 2020

@altendky,

Generally, each GitHub-hosted runner has 7 GB of RAM memory and 14 GB of SSD disk space available for users. See here.

When you run the job in a container, there may be also some limitation for the available space in the container.

0 replies

altendky · 2020-10-23T13:06:57Z

altendky
Oct 23, 2020
Author

Is there some way to check string the run if I’m close to that limit? I’m just guessing here since the error seems not representative if the actual issue.

0 replies

altendky · 2020-10-24T02:15:59Z

altendky
Oct 24, 2020
Author

Welp, I finally managed to recreate this one time locally. Maybe it’s a real bug somehow.

0 replies

altendky · 2020-10-25T23:27:22Z

altendky
Oct 25, 2020
Author

This is embarrasing, but so it goes. I was building a collection of copy actions then batch executing them. One of them incorrectly specified the target directory path rather than file path. If it ran ‘first’ then the file was copied where I wanted the directory. Otherwise the file was copied into the directory as desired. The inconsistency came in because I stored the actions in a set to deduplicate them. I have corrected the target paths and also sort the copy actions prior to executing them to avoid the inconsistency.

Sorry for the noise.

0 replies

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

GitHub Community

Filesystem errors while creating directories #25419

{{title}}

Replies: 5 comments

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

Select a reply

Navigation Menu

GitHub Community

Filesystem errors while creating directories #25419

altendky Oct 22, 2020

Replies: 5 comments

altendky Oct 22, 2020 Author

BrightRan Oct 23, 2020

altendky Oct 23, 2020 Author

altendky Oct 24, 2020 Author

altendky Oct 25, 2020 Author

altendky
Oct 22, 2020

altendky
Oct 22, 2020
Author

BrightRan
Oct 23, 2020

altendky
Oct 23, 2020
Author

altendky
Oct 24, 2020
Author

altendky
Oct 25, 2020
Author