Asynchronous invocations

For event-driven file processing, OSCAR automatically manages the creation and notification system of MinIO buckets. This allow the event-driven invocation of services using asynchronous requests for every file uploaded to the bucket, which generates a Kubernetes job for every file to be processed.

These jobs will be queued up in the Kubernetes scheduler and will be processed whenever there are resources available. If OSCAR cluster has been deployed as an elastic Kubernetes cluster (see Deployment with IM), then new Virtual Machines will be provisioned (up to the maximum number of nodes defined) in the underlying Cloud platform and seamlessly integrated in the Kubernetes clusters to proceed with the execution of jobs. These nodes will be terminated as the worload is reduced. Notice that the output files can be stores in MinIO or in any other storage back-end supported by the FaaS supervisor.

If you want to process a large number of data files, consider using OSCAR Batch, a tool designed to perform batch-based processing in OSCAR clusters. It includes a coordinator tool where the user provides a MinIO bucket containing files for processing. This service calculates the optimal number of parallel service invocations that can be accommodated within the cluster, according to its current status, and distributes the image processing workload accordingly among the service invocations. This is mainly intended to process large amounts of files, for example, historical data.