How do I remove sensitive data from my repository?

Please assist a newbie? I need to remove google-services.json and a directory named environments. I don’t see them in the master branch but I can still view the history on previous commits, can see the contents of the files with diff. I want them GONE. Not even diffs.

So I found this document.

I used the BFG Repo-Cleaner, environment is Cygwin.

CBdeV@HP-6005 ~/SecureCoop_app
$ java -jar ../BFG\ Git\ Repo-Cleaner\ 1.13.0.jar --delete-files environment.ts .

Using repo : C:\cygwin64\home\CBdeV\SecureCoop_app\.\.git

Found 370 objects to protect
Found 4 commit-pointing refs : HEAD, refs/heads/master, refs/remotes/origin/HEAD, refs/remotes/origin/master

Protected commits
-----------------

These are your protected commits, and so their contents will NOT be altered:

 * commit 72f68244 (protected by 'HEAD')

Cleaning
--------

Found 1821 commits
Cleaning commits:       100% (1821/1821)
Cleaning commits completed in 1,665 ms.

Updating 2 Refs
---------------

        Ref                          Before     After
        ------------------------------------------------
        refs/heads/master          | 72f68244 | fdadfd13
        refs/remotes/origin/master | 72f68244 | fdadfd13

Updating references:    100% (2/2)
...Ref update completed in 45 ms.

Commit Tree-Dirt History
------------------------

        Earliest                                              Latest
        |                                                          |
        .....................................DDDDDDDDDDDDDDDDDDDDDDD

        D = drty commits (file tree fixed)
        m = modified commits (commit message or parents changed)
        . = clean commits (no changes to file tree)

                                Before     After
        -------------------------------------------
        First modified commit | a380eafb | 30c2dbd7
        Last drty commit     | 3d0d175c | 406b140d

Deleted files
-------------

        Filename         Git id
        ----------------------------------------------------
        environment.ts | 68ec7df6 (1.1 KB), ad419602 (742 B)


In total, 547 object ids were changed. Full details are logged here:

        C:\cygwin64\home\CBdeV\SecureCoop_app\..bfg-report\2020-10-04\21-12-55

BFG run is complete! When ready, run: git reflog expire --expire=now --all && git gc --prune=now --aggressive


--
You can rewrite history in Git - don't let Trump do it for real!
Trump's administration has lied consistently, to make people give up on ever
being told the truth. Don't give up: https://www.rescue.org/topic/refugees-america
--



CBdeV@HP-6005 ~/SecureCoop_app
$ git reflog expire --expire=now --all && git gc --prune=now --aggressive
Enumerating objects: 19253, done.
Counting objects: 100% (19253/19253), done.
Delta compression using up to 4 threads
Compressing objects: 100% (18417/18417), done.
Writing objects: 100% (19253/19253), done.
Total 19253 (delta 8738), reused 9743 (delta 0), pack-reused 0
Removing duplicate objects: 100% (256/256), done.

CBdeV@HP-6005 ~/SecureCoop_app
$ 

Then did a Push in GitHub desktop. It appears to have completed successfully, but when I load up the history on GitHub.com, I can still see the old commit, can still view the contents.

Also tried filter-branch. Same result after a push. Please assist.

CBdeV@HP-6005 ~/SecureCoop_app
$ git filter-branch --index-filter "git rm -rf --cached --ignore-unmatch src/environments/" HEAD
WARNING: git-filter-branch has a glut of gotchas generating mangled history
         rewrites.  Hit Ctrl-C before proceeding to abort, then use an
         alternative filtering tool such as 'git filter-repo'
         (https://github.com/newren/git-filter-repo/) instead.  See the
         filter-branch manual page for more details; to squelch this warning,
         set FILTER_BRANCH_SQUELCH_WARNING=1.
Proceeding with filter-branch...

Rewrite 7d2dbbc994283027447520fb46ebefb729c838a8 (901/1179) (709 seconds passed, remaining 218 predicted)    rm 'src/environments/index.ts'
rm 'src/environments/prod.ts'
rm 'src/environments/schema.ts'
Rewrite c28a5283bb9a6481384d08ee6d25937587c6872b (903/1179) (710 seconds passed, remaining 217 predicted)    rm 'src/environments/index.ts'
rm 'src/environments/prod.ts'
rm 'src/environments/schema.ts'
(..)
Rewrite 643420098451a4da0f71911dced7672b8afd7e1d (1173/1179) (918 seconds passed, remaining 4 predicted)    rm 'src/environments/environment.prod.ts'
rm 'src/environments/environment.ts'
Rewrite 500ad169f81199389347fb5df25ee4a73bcff55f (1175/1179) (920 seconds passed, remaining 3 predicted)    rm 'src/environments/environment.prod.ts'
rm 'src/environments/environment.ts'
Rewrite 3d0d175c8b824b26146d9a1b924d28525831124d (1177/1179) (921 seconds passed, remaining 1 predicted)    rm 'src/environments/environment.ts'
Rewrite 2ee349a291a53f1bb0a1009e4d507e27b7f41fb9 (1179/1179) (924 seconds passed, remaining 0 predicted)
Ref 'refs/heads/master' was rewritten



CBdeV@HP-6005 ~/SecureCoop_app
$ 

Are you doing a forced push or regular push?
How did you clone the repo to start with?

1 Like

Forced on CLI and regular on GitHub Desktop.

First cloned using GitHub Desktop.

It is a young repository and I can re-create it without too much headache, if there is no other option. Strange though that there is apparently no way to do this.

Do you mean that it’s still part of the commit history of the branch, or that a direct link to the commit still works? In the former case you probably missed something while cleaning. The latter case is to be expected, that’s why the documentation you linked recommends contacting Github support after cleaning up the history so they can remove the commits from any caches.

Note that you should consider the data compromised anyway, so you should change any passwords, tokens, API keys and the like. :warning:

1 Like

Yes, a direct link to the commit still works. So I will ask Github support to clean up. Didn’t notice that the first time I read the docs, even though it’s in red. My wife says I miss things right in front of me, too.

1 Like