Custom Transport Adapter in HTTP Request

Quick Recap

This writing’s such as a small note for myself (or at least) hopefully its also useful for anyone who reads it. For a brief information why I’m ended up writing custom Transport Adapter when sending HTTP request based on several reasons:

• consideration when uploading the file with large of size, which we need chunked the operation to split up into several parts
• other reason was just a quirks intuition that Transport Adapter become useful when finding edge of case like this

Cited on the requests documentation : Transport Adapters provide a mechanism to define interaction methods for an HTTP service. In particular, they allow you to apply per-service configuration, so basically the issues that im facing on can be solved by making a custom adapter according to its functionality.

Here’s a snippet of the code that i’m using to send HTTP request :

class AppGalleryAdapter(BaseAdapter):
    def __init__(self, file_object) -> None:
        super(AppGalleryAdapter, self).__init__()
        self.file_object = file_object
 
    def send(self, request, **kwargs) -> requests.Response:
        if request.method not in ["POST", "PUT"]:
            raise ValueError("Invalid HTTP verb method to perform request")
 
        # response = Response()
 
        file_name = str(self.file_object)
        file_path = os.path.abspath(self.file_object)
        file_size = os.stat(file_path).st_size
 
        if file_name is not None:
            print(file_name, file_path, file_size)
 
        with open(file=file_path) as file:
            for chunk in read_in_chunks(file_object=file):
                headers_payload = {"Content-Length": file_size}
                try:
                    response = requests.request(
                        method=request.method,
                        url=request.url,
                        data=chunk,
                        headers=headers_payload,
                    )
                except Exception as e:
                    raise Exception(f"Uncaught exception error {e}")
                file.close()
 
        if response.status_code == 200:
            response.status_code = response.ok
            response.url = request.url
            response.encoding = "utf-8"
 
        return response
 
    def close(self) -> None:
        return super().close()

the code snippet above eventually only works and will execute when we want to upload a file from a directory or absolute path to a specific HTTP target. For my own case, the upload process will be directed to the Huawei AppGallery service where the process is only acceptable using the POST and PUT methods. The upload process will go through iterations using the read_in_chunks() method where the maximum uploaded file size is only set to 5 GB (based on AppGallery requirement). In that code, there are also custom headers which are intended to set the content-length based on the file size in the directory or absolute path, if the request process is successful it will be returned as an encoded response with OK status along with the destination URL. Noteworthy information in here is the implementation of the send() method which is aims to construct arguments from the predefined __init__ class and ensure that the process will run on the underlying requests.session.

Once you’ve defined the custom adapter, you need to instantiate this new class and mount it into requests.session for the request process to run. For example :

# only mounted to HTTPS protocol since the domain url of AppGallery using HTTPS
# so as not to confuse you, this snippet i've exclude from the wrapper class named 
# "send_request" which is used for HTTP configuration to performing HTTP request
chunked_file_adapter = AppGalleryAdapter(file_object=files)
self.session.mount("https://", chunked_file_adapter)

Basically, whenever we wanted to doing some request for this chunked upload file, you can override the arguments in requests.session like this :

response = self.send_request(method="PUT", url=f"{BASE_URL}/publish/v2/upload-url", file_object="new-release.apk")

Additional Note

• i’m must say that the process for sending this request itself is not in a thread-safe manner
• a file that exceeds the maximum size, even though it has been uploaded successfully, there is a possibility that the files will not be updated as a the same file