Compression plugin

[perseoGI]

Changelog: Feature: Allow externalization of compression/decompression tasks to a user defined plugin
Docs: conan-io/docs#4107

Close #18259

Related issues:

• [feature] Multithreaded conan cache save #18185
• [feature] Support zstd compression for conan cache save #18255

Related PRs:

• implementing cache save/restore with xz #18276
• Add support for zstd compression #14706

With the aim of simplicity this PR keeps the current behaviour in some compression/decompression tasks such as:

• Cache save/restore
• File uploads
  Adding an extra layer which will be in charge of determining if the compression.py plugin is defined falling back to existing behaviour if it doesn't.

The compression.py plugin interface may follow the following structure:

def tar_extract(archive_path, dest_dir, conf=None, *args, **kwargs) -> None:
    pass
def tar_compress(archive_path, files, recursive, conf=None, ref=None, *args, **kwargs) -> None:
    pass

Pluging example with zstd:

# File: compression.py

import os
import tarfile

import zstandard

from conan.api.output import ConanOutput

# zstd compression
# Original author https://github.com/conan-io/conan/pull/14706
def tar_extract(archive_path, dest_dir, conf=None, *args, **kwargs):
    dctx = zstandard.ZstdDecompressor()
    ConanOutput().info(f"Decompressing {os.path.basename(archive_path)} with compression plugin (ZSTD)")
    with open(archive_path, "rb") as tarfile_obj:
        with dctx.stream_reader(tarfile_obj) as stream_reader:
            # The choice of bufsize=32768 comes from profiling decompression at various
            # values and finding that bufsize value consistently performs well.
            with tarfile.open(
                fileobj=stream_reader, bufsize=32768, mode="r|"
            ) as the_tar:
                the_tar.extractall(
                    path=dest_dir, filter=lambda tarinfo, _: tarinfo
                )


def tar_compress(archive_path, files, recursive, conf=None, ref=None, *args, **kwargs):
    ConanOutput(scope=str(ref or "")).info(
        f"Compressing {os.path.basename(archive_path)} with compression plugin (ZSTD)"
    )
    compresslevel = conf.get("user.zstd:compresslevel", check_type=int) if conf else None
    with open(archive_path, "wb") as tarfile_obj:
        # Only provide level if it was overridden by config.
        zstd_kwargs = {}
        if compresslevel is not None:
            zstd_kwargs["level"] = compresslevel

        dctx = zstandard.ZstdCompressor(write_checksum=True, threads=-1, **zstd_kwargs)

        # Create a zstd stream writer so tarfile writes uncompressed data to
        # the zstd stream writer, which in turn writes compressed data to the
        # output tar.zst file.
        with dctx.stream_writer(tarfile_obj) as stream_writer:
            # The choice of bufsize=32768 comes from profiling compression at various
            # values and finding that bufsize value consistently performs well.
            # The variance in compression times at bufsize<=64KB is small. It is only
            # when bufsize>=128KB that compression times start increasing.
            with tarfile.open(
                mode="w|",
                fileobj=stream_writer,
                bufsize=32768,
                format=tarfile.PAX_FORMAT,
            ) as tar:
                current_frame_bytes = 0
                for filename, abs_path in sorted(files.items()):
                    tar.add(abs_path, filename, recursive=recursive)

                    # Flush the current frame if it has reached a large enough size.
                    # There is no required size, but 128MB is a good starting point
                    # because it allows for faster random access to the file.
                    current_frame_bytes += os.path.getsize(abs_path)
                    if current_frame_bytes >= 134217728:
                        stream_writer.flush(zstandard.FLUSH_FRAME)
                        current_frame_bytes = 0

[memsharded]

Looking much better, easier to assess that the risk is low.
Just some small concern about the pkglist.json of the conan cache restore

[memsharded]

It is looking good.

Before proceeding, I'd like to see a poc of using zstd and how the possible fallbacks and error handling is done, when mixing packages conan_package.tgz compressed with gz and with zstd.

[AbrilRBS]

A comment about not re-checking the file, otherwise this looks good :)

[Copilot]

Pull Request Overview

Introduces a plugin hook for compression/decompression by loading a user-supplied compression.py and delegating to it when present, with fallback to the builtin behavior.

• Loaded compression.py via a new config.compression_plugin property and propagated it through all compress/uncompress calls.
• Updated compress_files and uncompress_file to invoke the plugin if available.
• Added integration tests for plugin loading, fallback behavior, and default output.

Reviewed Changes

Copilot reviewed 9 out of 9 changed files in this pull request and generated 2 comments.

Show a summary per file

File	Description
test/integration/extensions/test_compression_plugin.py	added tests for plugin validity and end-to-end usage
test/integration/command/cache/test_cache_save_restore.py	asserted default compress/decompress outputs when no plugin
conan/internal/rest/rest_client_local_recipe_index.py	passed config_api into LocalRecipesIndexApp
conan/internal/rest/remote_manager.py	propagated compression_plugin into uncompress_file
conan/internal/cache/home_paths.py	added compression_plugin_path property
conan/internal/api/uploader.py	forwarded compression_plugin to compress_files
conan/internal/conan_app.py	injected config_api into RemoteManager and LocalRecipesIndexApp
conan/api/subapi/config.py	implemented compression_plugin loader property
conan/api/subapi/cache.py	integrated plugin hook into cache save/restore

Comments suppressed due to low confidence (1)

conan/api/subapi/config.py:248

• The code references os.path but there’s no import os at the top of this module. Add import os to avoid a NameError.

if not os.path.exists(compression_plugin_path):

[Copilot]

[nitpick] Add a docstring explaining that this property returns the path to the compression.py plugin under the extensions directory.

Suggested change

	def compression_plugin_path(self):
	def compression_plugin_path(self):
	"""Returns the path to the `compression.py` plugin under the extensions directory."""

[Copilot]

[nitpick] Instead of matching the newline, assert on "Compressing conan_cache_save.tgz" in c.out to make the test more robust across platforms and output formats.

Suggested change

	assert "Compressing conan_cache_save.tgz\n" in c.out
	assert "Compressing conan_cache_save.tgz" in c.out