Help
cancel
Showing results for 
Search instead for 
Did you mean: 
Copilot Lvl 3
Message 1 of 12

Training ML Model and pushing it back to repository

Solved! Go to Solution.

Hey! I'm currently facing an issue I can't seem to get around. I have a Github action which install the associated repository and the required dependencies, pull some training data from other sources and then train the repositorys ML model. After training, the model files are overwritten and my intention is to push the new model back to the repo.

 

If I train the model for less than 100 epochs (less than 1 hours) then the action works fine, it trains the model and then pushes the newly trained model data back to the repository. If I train for more than 100 epochs then when I reach the push step it says :

Push to branch automatic-retraining
remote: Invalid username or password.
fatal: Authentication failed for 'https://github.com/thenotcompany/formula-generator-model.git/'

 

I find it really weird, same code, same model, just different amount of epochs (and thus runtime) and I get this error. Any ideas?

Thanks in advance!

11 Replies
Highlighted
Commander Lvl 1
Message 2 of 12

Re: Training ML Model and pushing it back to repository

Hey @amorenocb, you are not hitting the job timeout but the VM you are running on must forget the git credentials after an hour. Can you provide the yml file you are using for the workflow (making sure to omit any sensitive info)?



*Mark helpful posts with Accept as Solution to help other users locate important info. Don't forget to give Kudos for great content!*
Highlighted
Copilot Lvl 3
Message 3 of 12

Re: Training ML Model and pushing it back to repository

Hey @logankilpatrick! Thanks for you reply, I have the same feeling but haven't been able to find any reference to "Github runner forgeting credentinals". Below is my yaml file. 

 

name: Model re-training

on:
  # schedule:
  #   # Train once a week on Mondays at 0:00
  #  - cron: '0 00 * * 1'
  [push]

jobs:
  re-train:
    name: Re-training model
    runs-on: ubuntu-latest

    steps:

    # Setup SSH to clone private dependencies.
    - uses: webfactory/ssh-agent@v0.1.1
      with:
        ssh-private-key: ${{ secrets.A_PRIVATE_KEY }}

    - name: Checkout
      uses: actions/checkout@v2

    - name: Checkout a database repository
      uses: actions/checkout@v2
      with:
        repository: ---
        token: ${{ secrets.A_PAT }}
        path: ---

    - name: Checkout another database repository
      uses: actions/checkout@v2
      with:
        repository: ---
        token: ${{ secrets.A_PAT }}
        path: ---

    - name: Setup Python
      uses: actions/setup-python@v1
      with:
        python-version: '3.6' # Version range or exact version of a Python version to use, using SemVer's version range syntax

    - name: Install DVC
      run: |
        pip install 'dvc[gs]==0.86.5'

    - name: Setup DVC credentials
      id: dvc_setup
      uses: timheuer/base64-to-file@v1.0.3
      with:
        fileName: '---'
        encodedString: ${{ secrets.SOME_CREDENTIALS }}

    - name: Get training data
      env:
        [ ... ]
      run: |
        [ ...]

    # Install formula generator and run re-training command line utility.
    - name: Re-train
      run: |
        pip install --upgrade setuptools
        pip install -e .
        train-model [...]

    - name: Commit newly trained model
      run: |
        git config --local user.email "action@github.com"
        git config --local user.name "Github Action"
        git commit -m "Model retraining" -a
    - name: Push newly trained model
      uses: ad-m/github-push-action@v0.5.0
      with:
        branch: automatic-retraining
        github_token: ${{ secrets.A_PAT }}
Highlighted
Copilot Lvl 3
Message 4 of 12

Re: Training ML Model and pushing it back to repository

Hey @logankilpatrick thanks for you reply! I have the same feeling, somehow the GH runner is forgetting the credentials after a while. Ill attach a the yml file below.

 

name: Model re-training

on:
  [push]

jobs:
  re-train:
    name: Re-training model
    runs-on: ubuntu-latest

    steps:
    
    # Setup SSH to clone private dependencies.
    - uses: webfactory/ssh-agent@v0.1.1
    [ ... ]
    - name: Checkout
      uses: actions/checkout@v2
 
    - name: Checkout db repository
      uses: actions/checkout@v2
      with:
        repository: [ ... ]
        token: [ ... ]
        path: [ ... ]

    - name: Checkout another db
      uses: actions/checkout@v2
      with:
        repository: [ ... ]
        token: [ ... ]
        path:  [ ... ]

    - name: Setup Python
      uses: actions/setup-python@v1
      with:
        python-version: '3.6' 

    - name: Install DVC
      run: |
        pip install 'dvc[gs]==0.86.5'

    - name: Setup DVC credentials
      id: dvc_setup
      uses: timheuer/base64-to-file@v1.0.3
      with:
        fileName:  [ ... ]
        encodedString:  [ ... ]

    - name: Get training data 
      env:
        CREDENTIALS:  [ ... ]
      run: |
        [ ... ]

    # Install repository model and run re-training command line utility.
    - name: Re-train
      run: |
        pip install --upgrade setuptools
        pip install -e .
        train-model --epochs 120
    - name: Commit newly trained model
      run: |
        git config --local user.email "action@github.com"
        git config --local user.name "Github Action"
        git commit -m "Model retraining" -a
    - name: Push newly trained model
      uses: ad-m/github-push-action@v0.5.0
      with:
        branch: automatic-retraining
        github_token:  [ ... ]

Thanks again!

Highlighted
GitHub Partner
Message 5 of 12

Re: Training ML Model and pushing it back to repository

Hi @amorenocb ,

 

This is by designed, the access token expires after 60 minutes. GitHub fetches a token for each job, before the job begins. When a workflow run or its jobs are queued for more than one hour, the token may expire before the job starts. You can find more details here and here

 

Thanks.

Highlighted
Copilot Lvl 3
Message 6 of 12

Re: Training ML Model and pushing it back to repository

Hi @weide-zhou! Is there any way to get around this? Maybe using a personal access token to pull and then push?

Highlighted
Commander Lvl 1
Message 7 of 12

Re: Training ML Model and pushing it back to repository

If there is a way to get around this, it's using your PAT! I would give it a shot. 



*Mark helpful posts with Accept as Solution to help other users locate important info. Don't forget to give Kudos for great content!*
Highlighted
Copilot Lvl 3
Message 8 of 12

Re: Training ML Model and pushing it back to repository

Hey! I managed to get it working by manually doing the git pull/push instructions using the PAT. I'll post it here if its of any use for someone in the future :)

name: Model re-training

on:
  # schedule:
  #    # Train once a week on Mondays at 0:00
  #   - cron: '0 00 * * 5'
  push

jobs:
  re-train:
    name: Re-training model
    runs-on: ubuntu-latest

    steps:

    # Setup SSH to clone private dependencies.
    - uses: webfactory/ssh-agent@v0.1.1
      with:
        ssh-private-key: ${{ secrets.SSH_PRIVATE_KEY }}

    # Get some parsed info of the repository
    - name: Get repository metadata
      id: r_m
      uses: amorenocb/repository-metadata@v0.1.0

    - name: Checkout repository
      env:
        GIT_CLONE: ${{ format('{0} --depth 1 https://{1}@github.com/{2}.git', steps.r_m.outputs.branch, secrets.PAT, github.repository) }}
      run: |
        git clone --single-branch --branch $GIT_CLONE

    # Install model and run re-training command line utility.
    - name: Re-train
      env:
        REPO_NAME: ${{ format('{0}',steps.r_m.outputs.name) }}
      run: |
        pip install --upgrade setuptools
        cd $GITHUB_WORKSPACE/$REPO_NAME
        pip install -e .
        train-model

    - name: Commit newly trained model
      env:
        REPO_NAME: ${{ format('{0}',steps.r_m.outputs.name) }}
      run: |
        cd $GITHUB_WORKSPACE/$REPO_NAME
        git config --local user.email "trainer@github.com"
        git config --local user.name "Trainer"
        git commit -m "Model retraining" -a
        git push --force https://${{ secrets.PAT }}@github.com/${{ github.repository }}.git

 

Highlighted
Solution
GitHub Partner
Message 9 of 12

Re: Training ML Model and pushing it back to repository

Hi @amorenocb ,

 

Check on my side, i used PAT , and add parameter 'persist-credentials: false' to actions/checkout@v2 to successfully complete the push. Code sample as below:

name: tokenexpire
on: [push]
  
jobs:
  job1:
    runs-on: [ubuntu-latest]
    steps:
      - name: checkout 2nd
        uses: actions/checkout@v2
        with:
          repository: weide-zhou/ticket19
          token: ${{ secrets.PAT }}
          path: ticket19
          persist-credentials: false
      - name: checkout 1st
        uses: actions/checkout@v2
        with:
          persist-credentials: false
          clean: false
      - name: touch file
        run: |
          touch filetest1.txt
      - name: sleep 1hour
        run: |
          sleep 1h
      - name: Commit newly trained model
        run: |
          git config --local user.email "action@github.com"
          git config --local user.name "Github Action"
          git add .
          git commit -m "Model retraining" -a
      - name: Push newly trained model
        uses: ad-m/github-push-action@v0.5.0
        with:
          branch: B66
          github_token: ${{ secrets.PAT }}

pushPAT.png

Highlighted
Copilot Lvl 3
Message 10 of 12

Re: Training ML Model and pushing it back to repository

Hey @weide-zhou it works! Although I noticed that now that I don't use the github token after I push the trained model the action triggers itself again and again. Is there a way to specify a parameter with the push command so it does not trigger the action again? 

It's not going to be a problem anyway, because the trigger of this action will be a schedule, not a push. But anyways it would be nice to know. 

 

Thank you for your response!