Moving towards Cloud storage

Previous versions of PhotoPaste used the file system as storage. It was straightforward and easy to work with. But I've been reaching the limits of this strategy lately:

1. A typical event of 50 people produces 1GB of media files. VPS providers offer only limited space. Hostinger doesn't have an option for add-on storage.
2. Using the file system left a bunch of blocking I/O calls all over the app, requiring many Mono.fromCallable() wrappers that make the code hard to read.

RClone

To mitigate the storage limit, I had been using RClone. This lets you use an S3 compatible Cloud Drive as if it's a hard disk. It's obviously a little slower than an actual disk, but fast enough for what I was doing. I had recently worked with CloudFlare and could easily define an R2 bucket there.

There were two problems with RClone that appeared as time went on:

1. RClone occasionally breaks, and I don't have the knowledge and skills to diagnose and solve this (only solution I have so far is rebooting).
2. Using cloud storage as a hard disk creates a lot of unnecessary Class A Operations (list, write), which goes over the free limit when there's a traffic spike.
3. I use none of the free bandwidth associated with CloudFlare R2 storage.

To solve a bunch of these problems, I'm preparing the app for direct cloud storage, without using RClone.

Generic storage service

To prepare the move, I created a generic StorageService interface that's suitable for both disk access and cloud access.

/**
 * Generic interface for storing files, modeled after cloud storage patterns
 */
public interface StorageService {

    /**
     * Store content under the given key/filename.
     *
     * @param key:     object key ('/events/myevent/photos/123123123123')
     * @param content: content to store as a reactive stream of DataBuffers
     * @return the final storage key
     */
    Mono<String> store(String key, Flux<DataBuffer> content);

    /**
     * Load content with key.
     *
     * @param key object key ('/events/myevent/photos/123123123123')
     * @return content as a reactive stream of DataBuffers
     */
    Flux<DataBuffer> load(String key);

    /**
     * Load content as resource. Useful when returning files in API calls.
     *
     * @param key object key ('/events/myevent/photos/123123123123')
     * @return content as a Resource
     */
    Mono<Resource> loadAsResource(String key);

    /**
     * Delete an object. Completes sucessfully even if object doesn't exist.
     *
     * @param key object key ('/events/myevent/photos/123123123123')
     * @return nothing
     */
    Mono<Void> delete(String key);

    /**
     * Check if object exists.
     *
     * @param key object key ('/events/myevent/photos/123123123123')
     * @return Mono<true> if object exists, else Mono<false>
     */
    Mono<Boolean> exists(String key);

    /**
     * List all keys that start with the given prefix.
     * The returned keys are relative ('/events/myevent/photos/123123123123').
     *
     * @param prefix: object prefix ('/events/myevent/photos/123', '/events/myevent')
     * @return Flux of object keys
     */
    Flux<String> list(String prefix);

    /**
     * Delete all keys that start with the given prefix.
     *
     * @param prefix: object prefix ('/events/myevent/photos/123')
     * @return nothing
     */
    default Mono<Void> deleteByPrefix(String prefix) {
        return list(prefix)
                .flatMap(this::delete)
                .then();
    }

}

Then, I created class FileSystemStorageService that implements StorageService. I found and replaced all direct file system access calls with equivalent StorageService calls. An added benefit is that this places all the ugly Mono.fromCallable() wrappers in one place.

Result

My I/O code is centralized, and the app is a lot cleaner now. In the future, I can create an S3StorageService class and drop that in place of the FileSystemStorageService, and the app will behave exactly the same.