Skip to content

fix: stream artifact downloads to disk to avoid OOM on large artifacts#1

Merged
yeicor merged 4 commits into
masterfrom
copilot/fix-artifact-download-memory-issue
Apr 17, 2026
Merged

fix: stream artifact downloads to disk to avoid OOM on large artifacts#1
yeicor merged 4 commits into
masterfrom
copilot/fix-artifact-download-memory-issue

Conversation

Copilot AI commented Apr 15, 2026

Copy link
Copy Markdown
  • Remove getArtifactDownloadUrl (broken: relies on Octokit throwing on 302 which doesn't happen with native fetch in @actions/github v9)
  • Remove streamUrlToFile (no longer needed)
  • Remove node:https and node:http imports (only used by the removed functions)
  • Add Readable from node:stream and pipeline from node:stream/promises
  • Add downloadArtifactToFile(client, owner, repo, artifactId, destPath) using Octokit request with custom fetch that pipes response body directly to disk
  • Update download flow in main() to use downloadArtifactToFile for both skip_unpack and extraction paths
Original prompt

Problem

In main.js, every artifact ZIP is currently downloaded by calling Octokit's downloadArtifact, which follows the redirect and fetches the entire binary response body into a zip.data ArrayBuffer that lives in Node.js heap memory. That Buffer.from(zip.data) is then passed either to AdmZip (which also keeps everything in RAM) or written to a temp file for unzip.

For large artifacts this can easily exhaust the runner's memory and kill the process with an OOM error.

Root-cause locations in main.js

Line(s) Problem
269-274 client.rest.actions.downloadArtifact buffers the whole ZIP in zip.data
285 Buffer.from(zip.data) copies the in-memory buffer to disk (skip_unpack path)
296 Buffer.from(zip.data) copies the in-memory buffer to a temp file before unzip
300 new AdmZip(Buffer.from(zip.data)) loads the whole ZIP into AdmZip in-memory

Required changes to main.js

Replace all four in-memory paths with a streaming download written directly to disk. Here is the complete target implementation:

import * as core from '@actions/core'
import * as exec from '@actions/exec'
import * as github from '@actions/github'
import * as artifact from '@actions/artifact'
import AdmZip from 'adm-zip'
import { filesize } from 'filesize'
import pathname from 'node:path'
import fs from 'node:fs'
import https from 'node:https'
import http from 'node:http'
import os from 'node:os'

async function downloadAction(name, path) {
    const artifactClient = artifact.create()
    const downloadOptions = {
        createArtifactFolder: false
    }
    const downloadResponse = await artifactClient.downloadArtifact(
        name,
        path,
        downloadOptions
    )
    core.setOutput("found_artifact", true)
}

async function getWorkflow(client, owner, repo, runID) {
    const run = await client.rest.actions.getWorkflowRun({
        owner: owner,
        repo: repo,
        run_id: runID || github.context.runId,
    })
    return run.data.workflow_id
}

/**
 * Resolves the final pre-signed download URL for an artifact by letting
 * Octokit attempt the request with redirects disabled, then following the
 * Location header ourselves so we never buffer the ZIP body in JS memory.
 */
async function getArtifactDownloadUrl(client, owner, repo, artifactId) {
    try {
        // Ask Octokit NOT to follow the redirect so we get the Location header.
        await client.rest.actions.downloadArtifact({
            owner,
            repo,
            artifact_id: artifactId,
            archive_format: "zip",
            request: { redirect: "manual" },
        })
    } catch (error) {
        // Octokit throws on 302 when redirects are disabled; grab the URL.
        if (error.status === 302 && error.response?.headers?.location) {
            return error.response.headers.location
        }
        // Some Octokit versions expose it differently.
        if (error.response?.headers?.location) {
            return error.response.headers.location
        }
        throw error
    }
    throw new Error("Expected a redirect response from downloadArtifact but got none")
}

/**
 * Streams a URL (following up to one redirect) straight to a file on disk.
 * Never holds the full body in Node.js heap memory.
 */
function streamUrlToFile(url, destPath) {
    return new Promise((resolve, reject) => {
        const dest = fs.createWriteStream(destPath)

        const handleResponse = (res) => {
            // Follow a single redirect (the pre-signed Azure Blob URL may itself redirect).
            if (res.statusCode >= 300 && res.statusCode < 400 && res.headers.location) {
                res.resume() // drain the redirect body
                const redirectUrl = new URL(res.headers.location)
                const mod = redirectUrl.protocol === 'https:' ? https : http
                mod.get(res.headers.location, handleResponse).on('error', reject)
                return
            }
            if (res.statusCode !== 200) {
                reject(new Error(`Unexpected HTTP ${res.statusCode} while downloading artifact`))
                res.resume()
                return
            }
            res.pipe(dest)
            dest.on('finish', resolve)
            dest.on('error', reject)
        }

        const parsedUrl = new URL(url)
        const mod = parsedUrl.protocol === 'https:' ? https : http
        mod.get(url, handleResponse).on('error', reject)
    })
}

async function main() {
    try {
        const token = core.getInput("github_token", { required: true })
        const [owner, repo] = core.getInput("repo", { required: true }).split("/")
        const path = core.getInput("path", { required: true })
        const name = core.getInput("name")
        const nameIsRegExp = core.getBooleanInput("name_is_regexp")
        const skipUnpack = core.getBooleanInput("skip_unpack")
        const ifNoArtifactFound = core.getInput("if_no_...

</details>



<!-- START COPILOT CODING AGENT SUFFIX -->

*This pull request was created from Copilot chat.*
>

Copilot AI changed the title [WIP] Fix memory issue when downloading large artifacts fix: stream artifact downloads to disk to avoid OOM on large artifacts Apr 15, 2026
Copilot AI requested a review from yeicor April 15, 2026 15:08
@yeicor yeicor marked this pull request as ready for review April 15, 2026 15:09
Copilot AI review requested due to automatic review settings April 15, 2026 15:09

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Updates the artifact download path in the GitHub Action to avoid buffering entire artifact ZIPs in Node.js heap memory by switching to a redirect-URL + streaming-to-disk approach.

Changes:

  • Adds getArtifactDownloadUrl() to resolve the pre-signed artifact URL without downloading the ZIP into memory.
  • Adds streamUrlToFile() to stream the ZIP directly to a temp file and then extract from disk.
  • Refactors the artifact download/extract loop to use the temp ZIP path for all unpacking modes and ensures temp cleanup via finally.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Comment thread main.js Outdated
Comment on lines +72 to +74
if (res.statusCode !== 200) {
reject(new Error(`Unexpected HTTP ${res.statusCode} while downloading artifact from ${url}`))
res.resume()

Copilot AI Apr 15, 2026

Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The error message includes the full url (Unexpected HTTP ... from ${url}), but this download URL is pre-signed and typically contains sensitive query parameters (SAS token). Emitting it to action logs can leak credentials. Prefer omitting the URL entirely or logging only a scrubbed form (e.g., origin + pathname without query).

Copilot uses AI. Check for mistakes.
Comment thread main.js Outdated
Comment on lines +31 to +52
async function getArtifactDownloadUrl(client, owner, repo, artifactId) {
try {
// Ask Octokit NOT to follow the redirect so we get the Location header.
await client.rest.actions.downloadArtifact({
owner,
repo,
artifact_id: artifactId,
archive_format: "zip",
request: { redirect: "manual" },
})
} catch (error) {
// Octokit throws on 302 when redirects are disabled; grab the URL.
if (error.status === 302 && error.response?.headers?.location) {
return error.response.headers.location
}
// Some Octokit versions expose it differently.
if (error.response?.headers?.location) {
return error.response.headers.location
}
throw error
}
throw new Error("Expected a redirect response from downloadArtifact but got none")

Copilot AI Apr 15, 2026

Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

getArtifactDownloadUrl assumes downloadArtifact(..., request: { redirect: "manual" }) will throw and expose the 302 Location on error.response. In the Octokit client used elsewhere (and in @actions/artifact), this call typically returns { status: 302, headers } without throwing; in that case this function will always hit throw new Error("Expected a redirect...") and break artifact downloads. Consider reading status/headers.location from the successful response and only throwing when status is not 302 or the Location header is missing.

Copilot uses AI. Check for mistakes.
Comment thread main.js Outdated
Comment on lines +62 to +85

const handleResponse = (res) => {
// Follow a single redirect (the pre-signed Azure Blob URL may itself redirect).
if (res.statusCode >= 300 && res.statusCode < 400 && res.headers.location) {
res.resume() // drain the redirect body
const redirectUrl = new URL(res.headers.location)
const mod = redirectUrl.protocol === 'https:' ? https : http
mod.get(res.headers.location, handleResponse).on('error', reject)
return
}
if (res.statusCode !== 200) {
reject(new Error(`Unexpected HTTP ${res.statusCode} while downloading artifact from ${url}`))
res.resume()
return
}
res.on('error', reject)
res.pipe(dest)
dest.on('finish', resolve)
dest.on('error', reject)
}

const parsedUrl = new URL(url)
const mod = parsedUrl.protocol === 'https:' ? https : http
mod.get(url, handleResponse).on('error', reject)

Copilot AI Apr 15, 2026

Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

streamUrlToFile claims to follow "up to one redirect", but the implementation will follow an unbounded number of redirects because the redirect branch recursively calls handleResponse with no counter/limit. This can hang on redirect loops and makes the behavior diverge from the comment. Add an explicit redirect limit (e.g., max 1 as documented) and fail with a clear error once exceeded.

Suggested change
const handleResponse = (res) => {
// Follow a single redirect (the pre-signed Azure Blob URL may itself redirect).
if (res.statusCode >= 300 && res.statusCode < 400 && res.headers.location) {
res.resume() // drain the redirect body
const redirectUrl = new URL(res.headers.location)
const mod = redirectUrl.protocol === 'https:' ? https : http
mod.get(res.headers.location, handleResponse).on('error', reject)
return
}
if (res.statusCode !== 200) {
reject(new Error(`Unexpected HTTP ${res.statusCode} while downloading artifact from ${url}`))
res.resume()
return
}
res.on('error', reject)
res.pipe(dest)
dest.on('finish', resolve)
dest.on('error', reject)
}
const parsedUrl = new URL(url)
const mod = parsedUrl.protocol === 'https:' ? https : http
mod.get(url, handleResponse).on('error', reject)
const maxRedirects = 1
const makeRequest = (currentUrl, redirectsRemaining) => {
const parsedUrl = new URL(currentUrl)
const mod = parsedUrl.protocol === 'https:' ? https : http
mod.get(currentUrl, (res) => {
// Follow a single redirect (the pre-signed Azure Blob URL may itself redirect).
if (res.statusCode >= 300 && res.statusCode < 400 && res.headers.location) {
if (redirectsRemaining <= 0) {
res.resume()
reject(new Error(`Too many redirects while downloading artifact from ${url}; exceeded maximum of ${maxRedirects} redirect`))
return
}
res.resume() // drain the redirect body
const redirectUrl = new URL(res.headers.location, currentUrl)
makeRequest(redirectUrl.toString(), redirectsRemaining - 1)
return
}
if (res.statusCode !== 200) {
reject(new Error(`Unexpected HTTP ${res.statusCode} while downloading artifact from ${url}`))
res.resume()
return
}
res.on('error', reject)
res.pipe(dest)
dest.on('finish', resolve)
dest.on('error', reject)
}).on('error', reject)
}
makeRequest(url, maxRedirects)

Copilot uses AI. Check for mistakes.
Comment thread main.js Outdated
Comment on lines +62 to +85

const handleResponse = (res) => {
// Follow a single redirect (the pre-signed Azure Blob URL may itself redirect).
if (res.statusCode >= 300 && res.statusCode < 400 && res.headers.location) {
res.resume() // drain the redirect body
const redirectUrl = new URL(res.headers.location)
const mod = redirectUrl.protocol === 'https:' ? https : http
mod.get(res.headers.location, handleResponse).on('error', reject)
return
}
if (res.statusCode !== 200) {
reject(new Error(`Unexpected HTTP ${res.statusCode} while downloading artifact from ${url}`))
res.resume()
return
}
res.on('error', reject)
res.pipe(dest)
dest.on('finish', resolve)
dest.on('error', reject)
}

const parsedUrl = new URL(url)
const mod = parsedUrl.protocol === 'https:' ? https : http
mod.get(url, handleResponse).on('error', reject)

Copilot AI Apr 15, 2026

Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

On non-200 responses or request/response errors, streamUrlToFile rejects but leaves the destination write stream open and may leave a partial file on disk. To avoid FD leaks and confusing subsequent runs, ensure you destroy()/close() the write stream on any error path and remove the partially written destPath before rejecting.

Suggested change
const handleResponse = (res) => {
// Follow a single redirect (the pre-signed Azure Blob URL may itself redirect).
if (res.statusCode >= 300 && res.statusCode < 400 && res.headers.location) {
res.resume() // drain the redirect body
const redirectUrl = new URL(res.headers.location)
const mod = redirectUrl.protocol === 'https:' ? https : http
mod.get(res.headers.location, handleResponse).on('error', reject)
return
}
if (res.statusCode !== 200) {
reject(new Error(`Unexpected HTTP ${res.statusCode} while downloading artifact from ${url}`))
res.resume()
return
}
res.on('error', reject)
res.pipe(dest)
dest.on('finish', resolve)
dest.on('error', reject)
}
const parsedUrl = new URL(url)
const mod = parsedUrl.protocol === 'https:' ? https : http
mod.get(url, handleResponse).on('error', reject)
let settled = false
const fail = (error, res) => {
if (settled) {
return
}
settled = true
if (res && !res.destroyed) {
res.destroy()
}
if (!dest.destroyed) {
dest.destroy()
}
fs.unlink(destPath, (unlinkError) => {
if (unlinkError && unlinkError.code !== 'ENOENT') {
reject(unlinkError)
return
}
reject(error)
})
}
const succeed = () => {
if (settled) {
return
}
settled = true
dest.close((error) => {
if (error) {
reject(error)
return
}
resolve()
})
}
const handleResponse = (res) => {
// Follow a single redirect (the pre-signed Azure Blob URL may itself redirect).
if (res.statusCode >= 300 && res.statusCode < 400 && res.headers.location) {
res.resume() // drain the redirect body
const redirectUrl = new URL(res.headers.location)
const mod = redirectUrl.protocol === 'https:' ? https : http
mod.get(res.headers.location, handleResponse).on('error', (error) => fail(error))
return
}
if (res.statusCode !== 200) {
res.resume()
fail(new Error(`Unexpected HTTP ${res.statusCode} while downloading artifact from ${url}`), res)
return
}
res.on('error', (error) => fail(error, res))
dest.on('finish', succeed)
dest.on('error', (error) => fail(error, res))
res.pipe(dest)
}
const parsedUrl = new URL(url)
const mod = parsedUrl.protocol === 'https:' ? https : http
mod.get(url, handleResponse).on('error', (error) => fail(error))

Copilot uses AI. Check for mistakes.
@yeicor yeicor merged commit 0e902bf into master Apr 17, 2026
20 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants