Downloading Large Files and Uploading to Salesforce Static Resources

Question:

I need to download a large .gz file and upload it to Salesforce as a Static Resource. The file is significantly larger than Salesforce’s HTTP response size limit (6 MB), making direct download via an Apex HTTP callout infeasible. My goal is to:

1. Download the .gz file programmatically using an HTTP GET request.
2. Decompress the .gz file in Apex.
3. Either save the decompressed file as a Static Resource or store its content elsewhere in Salesforce, such as in Files or Attachments.

Answer:

I attempted an HTTP callout using Apex but encountered the size limitation. While I am considering external solutions such as using Java Spring Boot or splitting the file, I prefer a solution that remains within the Salesforce ecosystem. What is the best approach to achieve this while adhering to Salesforce limits?
Enroll now for a free demo of Salesforce training at CRS Info Solutions, Salesforce Online training and kickstart your career with expert-led sessions.

Handling large files in Salesforce requires creative workarounds due to its strict governor limits. Below are several approaches you can use to accomplish the task, depending on your constraints and resources.

1. Using an External Service for File Processing

Since Salesforce has an HTTP response size limit of 6 MB, downloading a 250 MB file directly is impossible. You can use an external service, such as AWS Lambda, Google Cloud Functions, or a custom Java Spring Boot application, to fetch the file, decompress it, and split it into manageable chunks. The chunks can then be sent to Salesforce using REST or SOAP APIs.

Here’s a high-level workflow:

• Your external service downloads the .gz file and decompresses it.
• The decompressed file is split into smaller parts (e.g., 1 MB chunks).
• Each chunk is sent to Salesforce using the REST API, where it can be stored as individual Static Resources, Attachments, or ContentVersion records.

2. Using Salesforce Apex and Remote Site Setup

Although Salesforce cannot handle the file size directly, you can use an external service to perform the initial download and store the decompressed file on a server that allows partial file retrieval using HTTP range requests. From Salesforce, you can make sequential HTTP callouts to fetch the file in chunks.

For example:

1. Set up the remote site URL in Salesforce for the external server hosting the file.
2. Make HTTP callouts with the Range header to retrieve chunks of the file.

Here’s a sample Apex code snippet for chunked file downloads:

public class LargeFileDownloader {
    public static void downloadAndSaveFile(String fileUrl) {
        Integer chunkSize = 1000000; // 1 MB
        Integer offset = 0;
        Boolean isMoreData = true;
        Blob fileBlob = Blob.valueOf('');

        while (isMoreData) {
            Http http = new Http();
            HttpRequest request = new HttpRequest();
            request.setEndpoint(fileUrl);
            request.setMethod('GET');
            request.setHeader('Range', 'bytes=' + offset + '-' + (offset + chunkSize - 1));

            HttpResponse response = http.send(request);

            if (response.getStatusCode() == 206 || response.getStatusCode() == 200) {
                fileBlob = Blob.concat(fileBlob, response.getBodyAsBlob());
                offset += chunkSize;
                isMoreData = (response.getBodyAsBlob().size() == chunkSize);
            } else {
                isMoreData = false;
            }
        }

        saveFileToStaticResource(fileBlob, 'MyLargeFile');
    }

    private static void saveFileToStaticResource(Blob fileBlob, String resourceName) {
        StaticResource resource = new StaticResource(Name=resourceName, ContentType='application/octet-stream', Body=fileBlob);
        insert resource;
    }
}

Step-by-Step Breakdown

1. Class Definition

public class LargeFileDownloader {
    public static void downloadAndSaveFile(String fileUrl) {

The class LargeFileDownloader contains a method downloadAndSaveFile that accepts the file URL as a parameter. This method orchestrates the process of downloading the file in chunks and saving it as a Static Resource.

2. Chunk Size and Offset Initialization

Integer chunkSize = 1000000; // 1 MB
Integer offset = 0;
Boolean isMoreData = true;
Blob fileBlob = Blob.valueOf('');

• chunkSize: The size of each chunk to be downloaded (1 MB in this case).
• offset: Tracks the starting byte of the current chunk.
• isMoreData: A flag indicating whether there is more data to download.
• fileBlob: A variable to concatenate the downloaded chunks into a single file.

3. Download File in Chunks

while (isMoreData) {
    Http http = new Http();
    HttpRequest request = new HttpRequest();
    request.setEndpoint(fileUrl);
    request.setMethod('GET');
    request.setHeader('Range', 'bytes=' + offset + '-' + (offset + chunkSize - 1));

• A loop iterates until the entire file is downloaded.
• Each iteration makes an HTTP GET request with a Range header specifying the byte range for the current chunk.
  • Range: Requests only a specific range of bytes from the file (e.g., bytes=0-999999 for the first chunk).

    HttpResponse response = http.send(request);

• Sends the HTTP request and stores the response in response.

4. Process the HTTP Response

if (response.getStatusCode() == 206 || response.getStatusCode() == 200) {
    fileBlob = Blob.concat(fileBlob, response.getBodyAsBlob());
    offset += chunkSize;
    isMoreData = (response.getBodyAsBlob().size() == chunkSize);
} else {
    isMoreData = false;
}

• response.getStatusCode(): Checks the HTTP status code:
  • 206 (Partial Content): Indicates a successful range request.
  • 200 (OK): Indicates the file is fully downloaded (for smaller files).
• Blob.concat: Appends the current chunk (response.getBodyAsBlob()) to fileBlob.
• offset += chunkSize: Updates the starting byte for the next chunk.
• isMoreData: If the chunk size is less than the specified chunkSize, it means the file is fully downloaded, and the loop exits.

5. Save the Downloaded File as a Static Resource

saveFileToStaticResource(fileBlob, 'MyLargeFile');

Once all chunks are downloaded, the saveFileToStaticResource method is called to save the concatenated file as a Static Resource in Salesforce.

6. Static Resource Insertion

private static void saveFileToStaticResource(Blob fileBlob, String resourceName) {
    StaticResource resource = new StaticResource(Name=resourceName, ContentType='application/octet-stream', Body=fileBlob);
    insert resource;
}

• StaticResource: Represents a Salesforce Static Resource.
• Name: Sets the name of the Static Resource (e.g., MyLargeFile).
• ContentType: Specifies the file type (application/octet-stream for binary data).
• Body: Contains the file content (the concatenated Blob).
• insert resource: Saves the Static Resource to Salesforce.

Key Considerations

1. Governor Limits: The final Blob size and Static Resource insertion must remain within Salesforce’s 6 MB limit for file uploads.
2. Partial File Retrieval: This approach depends on the server supporting HTTP Range requests. Not all servers provide this capability.
3. Error Handling: For production use, implement proper error handling for failed HTTP requests, timeouts, and server errors.

This code is a robust example of downloading large files programmatically in chunks within Salesforce’s constraints, offering a scalable solution for handling sizable external files.

3. Using Salesforce’s Platform Events or Batch Apex

For extremely large files, consider splitting the download into multiple asynchronous transactions using Platform Events or Batch Apex. You can create a scheduler that handles chunks of the file in separate transactions.

4. Leverage Salesforce’s Integration Tools

If keeping the process within Salesforce is not mandatory, you could use tools like MuleSoft or Salesforce Functions to handle the download, decompression, and upload outside Apex, avoiding governor limits. These tools allow for seamless integration with Salesforce.

Summing Up

Downloading large files and uploading them to Salesforce Static Resources can be challenging due to Salesforce’s strict governor limits, particularly the 6 MB HTTP response size limit. However, by downloading the file in manageable chunks using HTTP Range requests, you can bypass these limitations. The file is downloaded progressively, with each chunk being appended to a Blob variable, and once all parts are retrieved, the combined content is saved as a Static Resource. This approach allows you to handle large files within Salesforce while ensuring performance and scalability, and is particularly useful when working within the Salesforce ecosystem without relying on external services.

Best Salesforce Training in Hyderabad

Elevate your Salesforce skills with our expert-led program that offers certification guidance, rigorous interview preparation, and well-structured modules tailored for Admin, Developer, and AI tracks. Salesforce training in Hyderabad With detailed class notes and hands-on learning, we ensure you gain practical knowledge to excel in the industry and secure your dream job. Experience the quality of our training by joining our free demo class today!