Are tag-less container images deleted? #26716

airtower-luna · 2020-09-18T21:04:36Z

airtower-luna
Sep 18, 2020

I’ve been trying out the GHCR with a very simple container, including build on Actions. So far I have the build always push to the beta tag. I noticed that after a second git push that triggered the container build and push to GHCR there is no trace left of the image that was previously tagged beta (i.e., no untagged version in the versions list).

Is this intentional? I’m not complaining about the behavior, I just want to know if I can rely on it to e.g. push a nightly tag or similar without worrying about cleaning up old versions. 😉

Answered by clarkbw

Sep 18, 2020

There should be an untagged version left behind assuming the new build is different. If there was no difference then you’d see this behaviour. We’re not deleting tag-less containers, they stick around until you delete them.

View full answer

clarkbw · 2020-09-18T21:50:33Z

clarkbw
Sep 18, 2020

There should be an untagged version left behind assuming the new build is different. If there was no difference then you’d see this behaviour. We’re not deleting tag-less containers, they stick around until you delete them.

0 replies

airtower-luna · 2020-09-18T21:53:45Z

airtower-luna
Sep 18, 2020
Author

Thanks for the explanation! Content-wise the containers are identical, just different labels, so I suppose that explains it. 🙂

In that case I will have to look into cleanup for nightly builds and the like.

0 replies

airtower-luna · 2020-09-19T14:01:05Z

airtower-luna
Sep 19, 2020
Author

I experimented with this a bit more and confirmed that tag-less images are still accessible if you have the digest. However, there seems to be no way to discover them: They don’t show up on the Github package page (only tagged versions there), and the Docker HTTP API V2 documentation doesn’t mention any way to find them either.

Is there any other way I’m missing?

0 replies

clarkbw · 2020-09-21T22:31:31Z

clarkbw
Sep 21, 2020

Can you take a look at the versions page? This is part of the settings page. I thought it had all the versions available and you could click into them.

GitHub

Build software better, together

GitHub is where people build software. More than 50 million people use GitHub to discover, fork, and contribute to over 100 million projects.

0 replies

airtower-luna · 2020-09-22T06:54:45Z

airtower-luna
Sep 22, 2020
Author

Evidently it shows only tagged versions. Two digests I still have saved for that image, both formerly tagged beta:

sha256:8090e11e86f6a5be8138cfc5026e170123497bfa1e72ec75a6cb831135bba36b
sha256:676d25701f6af4f8e3b958392c30e18af0e71a48fcf3ccd4ef3621ab6b041ba3

I just tested that I can still pull the images using those digests (yes), but they don’t show up on the versions page, or anywhere else that I’m aware of.

0 replies

jcansdale · 2020-09-22T11:41:24Z

jcansdale
Sep 22, 2020

airtower-luna:

I just tested that I can still pull the images using those digests (yes), but they don’t show up on the versions page, or anywhere else that I’m aware of.

This is also how it works on Docker Hub.

How would you want the un-tagged images to appear? Would you want to see the tags the image was previous tagged with?

0 replies

airtower-luna · 2020-09-22T18:43:53Z

airtower-luna
Sep 22, 2020
Author

jcansdale:

How would you want the un-tagged images to appear?

Preferably listed via an API call (rationale below), although on the versions page (maybe behind an “show untagged images” button or something) would be nice.

jcansdale:

Would you want to see the tags the image was previous tagged with?

Can’t hurt, but I’d be more interested in the timestamp of when the image was pushed.

What I’m thinking of is cleaning up images that aren’t needed after a while, e.g.:

Nightly (or even per-commit) builds that could be used for integration testing, or development on related projects.
Build environments: I’d like to prepare environments for use that with the Actions jobs.<job_id>.container keyword so dependencies don’t have to be installed during every build. That takes about 2 min each time, per distro tested in a matrix build (well, except Alpine which needs just a few seconds). To get updates of the underlying distro and dependencies I’d want to rebuild those environments frequently, e.g. during the first CI build of the day.

The latter would accumulate in the order of gigabytes pretty quickly, and would hardly be useful for anyone else in old versions, so I’d like to clean up after myself. Basically a sort of docker image prune. 😉

As long as packages storage is free for open source projects it would not hurt me to leave them around, but for people with private images it could quickly become a billing issue. And evidently it became an issue for Docker Hub, too, given their recent announcement to delete unused images on free accounts after 6 months (that’d do the trick for me, too, but maybe not for people who need to pay for storage). That post also says:

In addition, Docker will also be providing tooling, in the form of a UI and APIs, that will allow users to more easily manage their images.

I haven’t seen any specifics on that yet, but to me it sound like “tools to help you clean up”, so a compatible API on GHCR might be a thing to consider.

0 replies

jcansdale · 2020-09-23T16:06:25Z

jcansdale
Sep 23, 2020

@airtower-luna thanks for the link to the announcement.

Example Welcome to GitHub product feedback #1: Molly, a free Docker Hub user, pushed a tagged image molly/hello-world:v1 to Docker Hub on January 1, 2019. The image was never pulled since it was pushed. This tagged image will be considered inactive beginning November 1, 2020 when the new policy takes effect. The image and any tag pointing to hit, will be subject to deletion on November 1, 2020.

Wow, this is some pretty agressive cleanup of tagged images!

Example Feature Request: Select a column when adding an issue to a project #2 : Molly has another untagged image molly/myapp@sha256:c0ffee that was first pushed on January 1, 2018. This image was last pulled on August 1, 2020. This image will be considered an active image and will not be subject to deletion on November 1, 2020.

For untagged images this seems pretty reasonable.

What I’m thinking of is cleaning up images that aren’t needed after a while

So you’d want to know when they were pushed, whether they’re tagged and maybe when they where last downloaded?

0 replies

airtower-luna · 2020-09-23T20:52:42Z

airtower-luna
Sep 23, 2020
Author

jcansdale:

So you’d want to know when they were pushed, whether they’re tagged and maybe when they where last downloaded?

Yes, exactly. I guess the most flexible solution would be to include a list of tags for the image, if any. With that information I could easily set up e.g. a scheduled Actions workflow to clean up old images that are either untagged or tagged in a certain way. 🙂

And I agree that the Docker Hub policy is rather aggressive for tagged images, in principle I’d like to keep release versions around indefinitely. On the other hand depending on the image content I might worry about people using such an old image anyway, at least without vulnerability scanning (which I’d really like to see in GHCR, but that’s a separate thing). I wouldn’t want to keep track of which of the included libraries might have gotten security patches in the last half year, and whether any of the issues might be exploitable in the context of the image. 😅

0 replies

airtower-luna · 2021-02-18T21:25:11Z

airtower-luna
Feb 18, 2021
Author

I’ve written the kind of cleanup script I was thinking of here using the new packages delete/restore API. 🙂

github.com

airtower-luna/hello-ghcr/blob/e17ebc574711f4bcf85f8bd6b2380e226901778d/ghcr-prune.py#L76-L85


# prune old untagged images if requested
if del_before is not None and created < del_before \
   and len(metadata['tags']) == 0:
    if args.dry_run:
        print(f'would delete {v["id"]}')
    else:
        r = s.delete(f'https://api.github.com/user/packages/'
                     f'container/{args.container}/versions/{v["id"]}')
        r.raise_for_status()
        print(f'deleted {v["id"]}')

0 replies

jcansdale · 2021-03-17T11:33:44Z

jcansdale
Mar 17, 2021

Hi @airtower-luna,

I’ve written the kind of cleanup script I was thinking of here using the new packages delete/restore API . 🙂

I had a similar thought. 😃

Here’s a GitHub Action script that can be used to delete untagged images for a specified container package:

      - uses: actions/github-script@v3
        with:
          github-token: ${{ secrets.DELETE_PACKAGES_TOKEN }}
          script: |
            const response = await github.request("GET /${{ env.OWNER }}/packages/container/${{ env.PACKAGE_NAME }}/versions",
              { per_page: ${{ env.PER_PAGE }}
            });
            for(version of response.data) {
                if (version.metadata.container.tags.length == 0) {
                    console.log("delete " + version.id)
                    const deleteResponse = await github.request("DELETE /${{ env.OWNER }}/packages/container/${{ env.PACKAGE_NAME }}/versions/" + version.id, { });
                    console.log("status " + deleteResponse.status)
                }
            }
        env:
          OWNER: user # or orgs/<org name>
          PACKAGE_NAME: <package name>
          PER_PAGE: 100

OWNER should be either a user name or orgs/ORG_NAME.
DELETE_PACKAGES_TOKEN is a PAT with the delete:packages and write:packages scopes.

If anyone tries this, please let me know if you have any problems running the script!

3 replies

lakinduakash Sep 8, 2022

@jcansdale

Hi @airtower-luna,

I’ve written the kind of cleanup script I was thinking of here using the new packages delete/restore API . 🙂

I had a similar thought. 😃

Here’s a GitHub Action script that can be used to delete untagged images for a specified container package:
      - uses: actions/github-script@v3
        with:
          github-token: ${{ secrets.DELETE_PACKAGES_TOKEN }}
          script: |
            const response = await github.request("GET /${{ env.OWNER }}/packages/container/${{ env.PACKAGE_NAME }}/versions",
              { per_page: ${{ env.PER_PAGE }}
            });
            for(version of response.data) {
                if (version.metadata.container.tags.length == 0) {
                    console.log("delete " + version.id)
                    const deleteResponse = await github.request("DELETE /${{ env.OWNER }}/packages/container/${{ env.PACKAGE_NAME }}/versions/" + version.id, { });
                    console.log("status " + deleteResponse.status)
                }
            }
        env:
          OWNER: user # or orgs/<org name>
          PACKAGE_NAME: <package name>
          PER_PAGE: 100
OWNER should be either a user name or orgs/ORG_NAME. DELETE_PACKAGES_TOKEN is a PAT with the delete:packages and write:packages scopes.

If anyone tries this, please let me know if you have any problems running the script!

OH, thanks! I tried every old package deletion GitHub action packages. None worked. This worked!

RiccardoLincetto Sep 28, 2022

OWNER should be either a user name or orgs/ORG_NAME. DELETE_PACKAGES_TOKEN is a PAT with the delete:packages and write:packages scopes.

@jcansdale thanks for the workflow, it's great. By chance have you tried to avoid using a PAT and using instead jobs-level permissions?

mering Aug 21, 2023

I would be interested in a solution using jobs-level permissions as well!

ain-soph · 2021-04-29T04:41:39Z

ain-soph
Apr 29, 2021

Just want to clarify that OWNER needn’t change to be your username, just keeping it to be "user". I’ve been stuck on that for quite some time.

0 replies

jcansdale · 2021-04-29T08:16:43Z

jcansdale
Apr 29, 2021

ain-soph:

Just want to clarify that OWNER needn’t change to be your username, just keeping it to be "user" . I’ve been stuck on that for quite some time.

Sorry about that! I forgot and fell into this trap as well. 😊

1 reply

pkatul Sep 1, 2023

First of all, this script was really helpful,
For my I had to use orgs/ORG_NAME to make it working.

ghost · 2021-09-23T11:48:34Z

ghost
Sep 23, 2021

Hi, there.

Thank you for your script. It is working as intended.

However, if you build a multi-arch image with @latest tag looks like only one arch will be tagged which means the container of rest arch will be deleted. Could you tell me you have a solution for this case please?

2 replies

corinz Aug 31, 2022

@JonSun30 Did you ever figure this out?

b-ehlers Dec 23, 2022

Initially docker images supported only one architecture, so the structure to describe the image, named "manifest", supports only that. When multi-arch docker images were introduced, a "manifest list" structure was created, that contains the supported architecture names and references to manifests for each architecture. So if you create a multi-arch image supporting linux/amd64 and linux/arm64, three structures (with corresponding digests) are created: A manifest list, a manifest for the linux/amd64 architecture and another one for the linux/arm64 architecture.

Tags for single-arch images are tagging the manifest, for multi-arch images the manifest list gets tagged.

As an example look at the version page of turboscan-docker-images/mysql. You can't distingush if an item is a manifest or a manifest lists, even using the GitHub API won't help. The offical way is to load the contents of the digests. In the web interface you can follow the digest (or tag) link and you will notice a different page for the first item than that for the other ones. The first item is a manifest list, the other ones are manifests. In the manifest list of the first item in the "OS / Arch" tab the digests of the second and third item are referenced, so these three items belong to one multi-arch image. The forth and fifth item are not referenced by the manifest list, so they are two single-arch images.

Deleting manifests, that are referenced by a manifest list should be avoided. My solution (more a workaround) is to look at the creation dates. Multi-arch images are creating the manifests and the manifest list at the same time, so their creation date is within a short time span, for example here the information for the mysql images using the GitHub API:

$ curl -s -H "Authorization: token $TOKEN" 'https://api.github.com/orgs/github/packages/container/turboscan-docker-images%2Fmysql/versions' | jq -r '.[] | "\(.name) \(.created_at)"'
sha256:cdc0473e945049c9c32ccb10299c955d5be398a6a2a00ef2c060c538d5cd72d8 2022-03-17T13:18:15Z
sha256:c98b4ff5d7f4b81df412008da476e5fc6ea5fff71e7947a3ba80d2028362b74d 2022-03-17T13:18:14Z
sha256:ea49aee4acb5d1191114abfb7ffae73c785f7e37c9ba6f03a808e53ba154cb2f 2022-03-17T13:18:13Z
sha256:f0910a67381037cac57bfcef16645709cc113846ad826ba08da01716973d4a28 2022-03-17T13:17:32Z
sha256:11a5376741b847fb608ad362228b991fcaadd42af0371e2c7a25f4ad5ba94ab2 2022-03-17T13:12:35Z

The first three digests are created within a short time frame, they should be handled as a single entity. So either delete all three of them or none. The fourth and fifth digest don't belong to the manifest list and could be deleted at any time.

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

GitHub Community

Are tag-less container images deleted? #26716

{{title}}

Replies: 14 comments 6 replies

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{editor}}'s edit

{{editor}}'s edit

Select a reply

Are tag-less container images deleted? #26716

Replies: 14 comments · 6 replies

airtower-luna Sep 18, 2020 Author

airtower-luna Sep 19, 2020 Author

airtower-luna Sep 22, 2020 Author

airtower-luna Sep 22, 2020 Author

airtower-luna Sep 23, 2020 Author

airtower-luna Feb 18, 2021 Author

Replies: 14 comments 6 replies

airtower-luna
Sep 18, 2020
Author

airtower-luna
Sep 19, 2020
Author

airtower-luna
Sep 22, 2020
Author

airtower-luna
Sep 22, 2020
Author

airtower-luna
Sep 23, 2020
Author

airtower-luna
Feb 18, 2021
Author