set-output Truncates Multiline Strings

Hi,

I am using this code to create a release for one of our repositories:

- shell: bash
        id: release_description
        run: |
          description=$(./resources/get_release_description.sh ${{ steps.versioning.outputs.cli_version }})
          echo $description
          echo "::set-output name=description::$description"
        if: startsWith(steps.commit_message.outputs.commit_message, 'Version change')
      - shell: bash
        run: |
          echo ${{ steps.release_description.outputs.description }}
        if: startsWith(steps.commit_message.outputs.commit_message, 'Version change')
      - uses: csexton/create-release@add-body
        id: create_release
        with:
          tag_name: ${{ steps.versioning.outputs.cli_version }}
          release_name: safe-cli
          draft: false
          prerelease: false
          body: ${{ steps.release_description.outputs.description }}

This works without any errors. However, the problem is, the description (the body parameter of the create_release action) is basically a small markdown document, so it’s a multiline string. The problem is that ::set-output is truncating everything except the first line. I can confirm that by echoing the description in the same action where it’s assigned and then referencing the description output variable in the next action.

Could someone please advise how I can get this to work with a multiline string?

Thanks. 

8 Likes

I have reported your question to the appropriate engineering team for further evaluation. The team will review the feedback and notify me about the next steps. I will update here in time. Thank you for your understanding.

1 Like

I got response from the team. 

% and \n and \r can be escaped like below, the runner will unescape in reverse.

content="${content//'%'/'%25'}"
content="${content//$'\n'/'%0A'}"
content="${content//$'\r'/'%0D'}"

Please try to add next three red lines to your yml, kindly let me know whether this could help.

- shell: bash
        id: release_description
        run: |
          description=$(./resources/get_release_description.sh ${{ steps.versioning.outputs.cli_version }})
          echo $description
<font color="#FF0000"> description="${description//'%'/'%25'}"
          description="${description//$'\n'/'%0A'}"
          description="${description//$'\r'/'%0D'}"</font>
echo "::set-output name=description::$description"
if: startsWith(steps.commit_message.outputs.commit_message, 'Version change')

 I tested in my side, after adding these lines, I can use echo " ${{ steps.release_description.outputs.description1 }} " to output multiple line value. Please pay attention to “”

multiline.png

23 Likes

Sorry, I just noticed your reply!

Thanks very much for this, it resolves the issue.

It took me an hour to realize that multi-line strings were the issue for me, and then a few more hours to find this solution.

It would be great if multi-line strings as values could be better supported, and documented.

Here are a few observation. First, it is important to suppress word-splitting upon expansion in bash. This is easiest done by enclosing with double quotes:

REPORT="$(cat logfile)"

Similarly, word splitting also has to be suppressed when reading back:

echo "$REPORT"

The double quotes are necessary.

However, Workflows gets confused/truncates when dealing with an expanded multi-line value:

echo "::set-env name=REPORT::$REPORT"

This does not work for multi-line variables.

The solution above uses a standard bash feature to expand and substitute characters:

REPORT="${REPORT//'%'/'%25'}"
REPORT="${REPORT//$'\n'/'%0A'}"
REPORT="${REPORT//$'\r'/'%0D'}"

This essentially makes the value into a single line string.

The github engineers then also knew that when reading back the variable in the workflow these escape values get substituted back to the actual characters:

${{ env.REPORT }}

This results actually in a multi-line string. Unfortunately, this is not documented anywhere and comes across a bit as magic. If there are other substitutions it would be good to also document those.

7 Likes

Thanks so much posting this. I came across the same issue and found this rather obscure solution, after a lot of trial and error.

A couple observation dealing with multiline values, mostly as a reminder on how this works.

If using bash, it is important to suppress word splitting upon parameter expansion, to keep the line breaks. It is easiest with double quotes:

REPORT="$(cat log.out)"

Accessing the variable also invokes word splitting, which needs to be suppressed:

echo "$REPORT"

However, set-output or set-env does not work with expanded multiline values:

echo "::set-env name=REPORT::$REPORT"

This does does not work, as Workflows somehow truncates or ignores the value.

The solution above is to escape the newlines and other apparently unhandled characters using bash expansion and substitution. This makes the value effectively single line.

The thing to notice is that Workflows substitutes the escaped characters back when the parameters is used in ${{ }}:

body: |
  eslint and diff report:
  ```
  ${{ env.REPORT }}
  ```

Here the final value includes actual newlines.

It would be great if this behaviour cold be documented somewhere. For example, are there other substitutions happening ?

1 Like

@andreasplesch  Thank you for your further investigation. I would recommand you to create an issue in the action toolkit repo . You could ask for adding an example for set-env and set-output with multiline values in this document. 

Thanks. Good to know that there is more in depth documentation available in the toolkit repo. I think a link from the main help page at https://help.github.com/en/actions to this documentation could be helpful.

Hm, it looks like newlines should already be escaped automatically:

https://github.com/actions/toolkit/blob/master/packages/core/src/command.ts#L76

Anyways, here is the new issue in the repo:

https://github.com/actions/toolkit/issues/403

and a related issue:

https://github.com/actions/toolkit/issues/193

Escaping of special characters already happened for javascript actions but does not for bash scripts. There is a plan to update documentation.

I have bumped into this exact issue, and found another solution: we can use jq to do the conversion into a valid JSON string for us, and then use fromJSON to unpack it again.

For example: the variable $mystr holds the (multiline) contents of file myfile; we convert to a one-line JSON string with jq --raw-input --slurp '.', and then unpack in a later step.

- id: mystep
  run: |
    mystr=$(< myfile)
    msg=$(printf '%s' "$mystr" | jq --raw-input --slurp '.')
    echo "::set-output name=msg::$msg"

- run: |
    echo "${{ fromJSON(steps.mystep.outputs.msg) }}"

Notice that this isn’t fool-proof, either: if the expanded string contains double quotes, the echo in the second step might get confused.

3 Likes

I also have an issue where I would like to output a list of paths and other data only if the action processes multiple files (to push them to nuget.org or any nuget feeds).

Here is yet another potential work-around.

This is working in a CI Action job of mine, which takes the content of a text file that contains test results and posts the whole content (which is multiline content) back to the relevant PR as a comment. The resulting comment accurately reflects the original newlines of the test result output.

name: build_for_PR

on:
  pull_request:
    branches:
      - main

jobs:
  linux_publish_test_output:

    runs-on: ubuntu-18.04

    steps:
    - uses: actions/checkout@v1

    - name: run_all_linux_ci
      run: ./linux_build_all_and_test.sh

    - name: get_legacy_gui_test_output
      id: legacy_out_01
      run: echo "::set-output name=testconfout::$(cat path_from_my_git_root/to/build/artifacts/db_mgr_test_output.txt | python3 -c 'import sys; print(sys.stdin.read().replace("\\","\\\\").replace("\"","\\\"").replace("\n","\\n"))' )\n"

    - uses: actions/github-script@v3
      with:
        script: |
          var theContent = "${{join(steps.legacy_out_01.outputs.*, '\n')}}";
          github.issues.createComment({
          issue_number: context.issue.number,
          owner: context.repo.owner,
          repo: context.repo.repo,
          body: 'LEGACY GUI TEST OUTPUT (from most recent CI job on this PR):\n\n' + theContent
          })

This was inspired by the approach posted by @ bewuethr. As far as I can tell in a couple days of usage, the python-based solution I am sharing does manage to handle double-quotes in the text (in addition to newlines).

2 Likes

Thanks a lot for sharing, this unblocked me :slight_smile:

1 Like

For completeness, I want to point out that you can also use an environment variable via environment files for multiline text: Workflow commands for GitHub Actions - GitHub Docs

steps:
  - name: Set the value
    shell: bash
    run: |
      echo 'JSON_RESPONSE<<EOF' >> $GITHUB_ENV
      curl https://httpbin.org/json >> $GITHUB_ENV
      echo 'EOF' >> $GITHUB_ENV
  - name: Print multiline response
    shell: bash
    run: |
      echo "$JSON_RESPONSE"
2 Likes

Might be technically possible, but so far the only working result I could get was from this answer

Has this been fixed? I built a function to work around this, and now it displays the escape sequences.

Here’s my multi-line output solution which requires no changing of the data, however you do have to write it to a text file first.

In this example I’m using it for stderr, but you can adapt it to any kind of multi-line output too

# on windows, powershell fails on stderr output and improperly
# handles putting it in the file. It's required to run as cmd /c
# if not using windows, then change this command appropriately
- name: build
  id: build
  continue-on-error: true
  run: cmd /c "build 2> errors.txt"

- name: Set error-log to var
  uses: actions/github-script@v4
  id: error-log
  if: steps.build.outcome != 'success'
  with:
    script: |
      const fs = require('fs');
      return fs.readFileSync('errors.txt','utf8').toString();
    result-encoding: string

# you can access the output variable in like so
- name: Some action
  uses: whateveryouwant/whatever@4
  if: steps.build.outcome != 'success'
  with:
    something: ${{ steps.error-log.outputs.result }}

# print errors.txt to stdout + fail workflow since build failed
# use `cat` if not using windows
- name: Show build errors
  if: steps.build.outcome != 'success'
  run: |
    type errors.txt
    exit 1
1 Like

Note: You can change the output variable name by using core.setOutput as well.

For any lost souls in the future: If you’re setting output in a Docker container (like if your action is a .NET binary), not in the action YML, by just printing ::set-output name=whatever::whatever, all you need to do to fix this is replace any \n you have with %0A