Implement multipart tests (#4 )

Currently, tests do not test multipart uploads (since they use very small files). Implement some large (randomly generated) files as well. Reviewed-on: #4
Revert "CI broken notice"
2023-12-31 09:02:36 +00:00 · 2023-12-31 13:21:27 +06:00 · 2023-12-30 23:29:41 +06:00 · 2023-12-30 22:56:44 +06:00 · 2023-12-30 16:42:51 +00:00 · 2023-12-30 06:36:49 +00:00
20 changed files with 921 additions and 256 deletions
--- a/.editorconfig
+++ b/.editorconfig
@@ -0,0 +1,19 @@
+root = true
+
+[*]
+charset = utf-8
+end_of_line = lf
+indent_size = 4
+indent_style = space
+insert_final_newline = false
+max_line_length = 120
+tab_width = 4
+
+[{*.yaml,*.yml}]
+indent_size = 2
+
+[src/test/**/*]
+ktlint_standard_no-wildcard-imports = disabled
+
+[{*.kt,*.kts}]
+ij_kotlin_packages_to_use_import_on_demand = org.junit.jupiter.api,aws.sdk.kotlin.services.s3,kotlinx.coroutines,java.io,ziputils
--- a/.github/workflows/publish.yaml
+++ b/.github/workflows/publish.yaml
@@ -0,0 +1,50 @@
+name: Publish Workflow
+on:
+  push:
+    tags:
+      - v*
+jobs:
+  publish:
+    name: Publish library
+    runs-on: ubuntu-latest
+    steps:
+      - uses: actions/checkout@v4
+
+      - name: Set up Java
+        uses: actions/setup-java@v3
+        with:
+          distribution: adopt
+          java-version: 17
+
+      - name: Verify Gradle wrapper
+        uses: gradle/wrapper-validation-action@v1
+
+      - name: Setup Gradle
+        uses: gradle/gradle-build-action@v2
+
+      - name: Setup AWS Credentials
+        run: |
+          mkdir ~/.aws
+          echo ${{ secrets.aws_config }} | base64 -d > ~/.aws/config
+          echo ${{ secrets.aws_credentials }} | base64 -d > ~/.aws/credentials
+
+      - name: Run checks
+        run: ./gradlew check
+
+      - name: Parse parameters
+        id: parse
+        run: |
+          export VERSION="$(echo ${{ github.ref_name }} | cut -c2-)"
+          echo "Parsed version: '$VERSION'"
+          echo "backup_version=$VERSION" >> "$GITHUB_OUTPUT"
+
+      - name: Build
+        env:
+          BACKUP_VERSION: ${{ steps.parse.outputs.backup_version }}
+        run: ./gradlew shadowJar
+
+      - name: Upload artifacts
+        uses: actions/upload-artifact@v3
+        with:
+          name: ShadowJAR
+          path: build/libs/*-all.jar
--- a/.github/workflows/test.yaml
+++ b/.github/workflows/test.yaml
@@ -0,0 +1,35 @@
+name: Test Workflow
+on:
+  pull_request:
+    branches:
+      - main
+  push:
+    branches:
+      - main
+jobs:
+  lint-and-test:
+    name: Lint and test library
+    runs-on: ubuntu-latest
+    steps:
+      - uses: actions/checkout@v4
+
+      - name: Set up Java
+        uses: actions/setup-java@v3
+        with:
+          distribution: adopt
+          java-version: 17
+
+      - name: Verify Gradle wrapper
+        uses: gradle/wrapper-validation-action@v1
+
+      - name: Setup Gradle
+        uses: gradle/gradle-build-action@v2
+
+      - name: Setup AWS Credentials
+        run: |
+          mkdir ~/.aws
+          echo ${{ secrets.aws_config }} | base64 -d > ~/.aws/config
+          echo ${{ secrets.aws_credentials }} | base64 -d > ~/.aws/credentials
+
+      - name: Run checks
+        run: ./gradlew check
--- a/.idea/.name
+++ b/.idea/.name
@@ -0,0 +1 @@
+s3backup-tool
--- a/.idea/uiDesigner.xml
+++ b/.idea/uiDesigner.xml
@@ -0,0 +1,124 @@
+<?xml version="1.0" encoding="UTF-8"?>
+<project version="4">
+  <component name="Palette2">
+    <group name="Swing">
+      <item class="com.intellij.uiDesigner.HSpacer" tooltip-text="Horizontal Spacer" icon="/com/intellij/uiDesigner/icons/hspacer.svg" removable="false" auto-create-binding="false" can-attach-label="false">
+        <default-constraints vsize-policy="1" hsize-policy="6" anchor="0" fill="1" />
+      </item>
+      <item class="com.intellij.uiDesigner.VSpacer" tooltip-text="Vertical Spacer" icon="/com/intellij/uiDesigner/icons/vspacer.svg" removable="false" auto-create-binding="false" can-attach-label="false">
+        <default-constraints vsize-policy="6" hsize-policy="1" anchor="0" fill="2" />
+      </item>
+      <item class="javax.swing.JPanel" icon="/com/intellij/uiDesigner/icons/panel.svg" removable="false" auto-create-binding="false" can-attach-label="false">
+        <default-constraints vsize-policy="3" hsize-policy="3" anchor="0" fill="3" />
+      </item>
+      <item class="javax.swing.JScrollPane" icon="/com/intellij/uiDesigner/icons/scrollPane.svg" removable="false" auto-create-binding="false" can-attach-label="true">
+        <default-constraints vsize-policy="7" hsize-policy="7" anchor="0" fill="3" />
+      </item>
+      <item class="javax.swing.JButton" icon="/com/intellij/uiDesigner/icons/button.svg" removable="false" auto-create-binding="true" can-attach-label="false">
+        <default-constraints vsize-policy="0" hsize-policy="3" anchor="0" fill="1" />
+        <initial-values>
+          <property name="text" value="Button" />
+        </initial-values>
+      </item>
+      <item class="javax.swing.JRadioButton" icon="/com/intellij/uiDesigner/icons/radioButton.svg" removable="false" auto-create-binding="true" can-attach-label="false">
+        <default-constraints vsize-policy="0" hsize-policy="3" anchor="8" fill="0" />
+        <initial-values>
+          <property name="text" value="RadioButton" />
+        </initial-values>
+      </item>
+      <item class="javax.swing.JCheckBox" icon="/com/intellij/uiDesigner/icons/checkBox.svg" removable="false" auto-create-binding="true" can-attach-label="false">
+        <default-constraints vsize-policy="0" hsize-policy="3" anchor="8" fill="0" />
+        <initial-values>
+          <property name="text" value="CheckBox" />
+        </initial-values>
+      </item>
+      <item class="javax.swing.JLabel" icon="/com/intellij/uiDesigner/icons/label.svg" removable="false" auto-create-binding="false" can-attach-label="false">
+        <default-constraints vsize-policy="0" hsize-policy="0" anchor="8" fill="0" />
+        <initial-values>
+          <property name="text" value="Label" />
+        </initial-values>
+      </item>
+      <item class="javax.swing.JTextField" icon="/com/intellij/uiDesigner/icons/textField.svg" removable="false" auto-create-binding="true" can-attach-label="true">
+        <default-constraints vsize-policy="0" hsize-policy="6" anchor="8" fill="1">
+          <preferred-size width="150" height="-1" />
+        </default-constraints>
+      </item>
+      <item class="javax.swing.JPasswordField" icon="/com/intellij/uiDesigner/icons/passwordField.svg" removable="false" auto-create-binding="true" can-attach-label="true">
+        <default-constraints vsize-policy="0" hsize-policy="6" anchor="8" fill="1">
+          <preferred-size width="150" height="-1" />
+        </default-constraints>
+      </item>
+      <item class="javax.swing.JFormattedTextField" icon="/com/intellij/uiDesigner/icons/formattedTextField.svg" removable="false" auto-create-binding="true" can-attach-label="true">
+        <default-constraints vsize-policy="0" hsize-policy="6" anchor="8" fill="1">
+          <preferred-size width="150" height="-1" />
+        </default-constraints>
+      </item>
+      <item class="javax.swing.JTextArea" icon="/com/intellij/uiDesigner/icons/textArea.svg" removable="false" auto-create-binding="true" can-attach-label="true">
+        <default-constraints vsize-policy="6" hsize-policy="6" anchor="0" fill="3">
+          <preferred-size width="150" height="50" />
+        </default-constraints>
+      </item>
+      <item class="javax.swing.JTextPane" icon="/com/intellij/uiDesigner/icons/textPane.svg" removable="false" auto-create-binding="true" can-attach-label="true">
+        <default-constraints vsize-policy="6" hsize-policy="6" anchor="0" fill="3">
+          <preferred-size width="150" height="50" />
+        </default-constraints>
+      </item>
+      <item class="javax.swing.JEditorPane" icon="/com/intellij/uiDesigner/icons/editorPane.svg" removable="false" auto-create-binding="true" can-attach-label="true">
+        <default-constraints vsize-policy="6" hsize-policy="6" anchor="0" fill="3">
+          <preferred-size width="150" height="50" />
+        </default-constraints>
+      </item>
+      <item class="javax.swing.JComboBox" icon="/com/intellij/uiDesigner/icons/comboBox.svg" removable="false" auto-create-binding="true" can-attach-label="true">
+        <default-constraints vsize-policy="0" hsize-policy="2" anchor="8" fill="1" />
+      </item>
+      <item class="javax.swing.JTable" icon="/com/intellij/uiDesigner/icons/table.svg" removable="false" auto-create-binding="true" can-attach-label="false">
+        <default-constraints vsize-policy="6" hsize-policy="6" anchor="0" fill="3">
+          <preferred-size width="150" height="50" />
+        </default-constraints>
+      </item>
+      <item class="javax.swing.JList" icon="/com/intellij/uiDesigner/icons/list.svg" removable="false" auto-create-binding="true" can-attach-label="false">
+        <default-constraints vsize-policy="6" hsize-policy="2" anchor="0" fill="3">
+          <preferred-size width="150" height="50" />
+        </default-constraints>
+      </item>
+      <item class="javax.swing.JTree" icon="/com/intellij/uiDesigner/icons/tree.svg" removable="false" auto-create-binding="true" can-attach-label="false">
+        <default-constraints vsize-policy="6" hsize-policy="6" anchor="0" fill="3">
+          <preferred-size width="150" height="50" />
+        </default-constraints>
+      </item>
+      <item class="javax.swing.JTabbedPane" icon="/com/intellij/uiDesigner/icons/tabbedPane.svg" removable="false" auto-create-binding="true" can-attach-label="false">
+        <default-constraints vsize-policy="3" hsize-policy="3" anchor="0" fill="3">
+          <preferred-size width="200" height="200" />
+        </default-constraints>
+      </item>
+      <item class="javax.swing.JSplitPane" icon="/com/intellij/uiDesigner/icons/splitPane.svg" removable="false" auto-create-binding="false" can-attach-label="false">
+        <default-constraints vsize-policy="3" hsize-policy="3" anchor="0" fill="3">
+          <preferred-size width="200" height="200" />
+        </default-constraints>
+      </item>
+      <item class="javax.swing.JSpinner" icon="/com/intellij/uiDesigner/icons/spinner.svg" removable="false" auto-create-binding="true" can-attach-label="true">
+        <default-constraints vsize-policy="0" hsize-policy="6" anchor="8" fill="1" />
+      </item>
+      <item class="javax.swing.JSlider" icon="/com/intellij/uiDesigner/icons/slider.svg" removable="false" auto-create-binding="true" can-attach-label="false">
+        <default-constraints vsize-policy="0" hsize-policy="6" anchor="8" fill="1" />
+      </item>
+      <item class="javax.swing.JSeparator" icon="/com/intellij/uiDesigner/icons/separator.svg" removable="false" auto-create-binding="false" can-attach-label="false">
+        <default-constraints vsize-policy="6" hsize-policy="6" anchor="0" fill="3" />
+      </item>
+      <item class="javax.swing.JProgressBar" icon="/com/intellij/uiDesigner/icons/progressbar.svg" removable="false" auto-create-binding="true" can-attach-label="false">
+        <default-constraints vsize-policy="0" hsize-policy="6" anchor="0" fill="1" />
+      </item>
+      <item class="javax.swing.JToolBar" icon="/com/intellij/uiDesigner/icons/toolbar.svg" removable="false" auto-create-binding="false" can-attach-label="false">
+        <default-constraints vsize-policy="0" hsize-policy="6" anchor="0" fill="1">
+          <preferred-size width="-1" height="20" />
+        </default-constraints>
+      </item>
+      <item class="javax.swing.JToolBar$Separator" icon="/com/intellij/uiDesigner/icons/toolbarSeparator.svg" removable="false" auto-create-binding="false" can-attach-label="false">
+        <default-constraints vsize-policy="0" hsize-policy="0" anchor="0" fill="1" />
+      </item>
+      <item class="javax.swing.JScrollBar" icon="/com/intellij/uiDesigner/icons/scrollbar.svg" removable="false" auto-create-binding="true" can-attach-label="false">
+        <default-constraints vsize-policy="6" hsize-policy="0" anchor="0" fill="2" />
+      </item>
+    </group>
+  </component>
+</project>
--- a/README.md
+++ b/README.md
@@ -2,35 +2,92 @@
 This is a small backup utility for uploading/restoring a local directory to/from
 an AWS S3 bucket.

+## Usage
+This tool is released as a JAR in the [releases page](https://git.koval.net/cyclane/teamcity-executors-test-task/releases).
+Use `java -jar s3backup-tool-<version>.jar --help` for more detailed usage instructions.
+
+### --help
+```
+Usage: s3backup-tool [<options>] <command> [<args>]...
+
+  A simple AWS S3 backup tool. This tool assumes credentials are properly configured using aws-cli.
+
+Options:
+  -h, --help  Show this message and exit
+
+Commands:
+  create        Create a backup of a file or directory.
+  restore       Restore a backup from AWS S3.
+  restore-file  Restore a single file from a backup from AWS S3.
+```
+
+#### Subcommands
+```
+Usage: s3backup-tool create [<options>] <source> <bucket>
+
+  Create a backup of a file or directory.
+
+Options:
+  -h, --help  Show this message and exit
+
+Arguments:
+  <source>  File or directory to backup
+  <bucket>  Name of S3 bucket to backup to
+```
+
+```
+Usage: s3backup-tool restore [<options>] <bucket> <backupkey> <destination>
+
+  Restore a backup from AWS S3.
+
+Options:
+  -h, --help  Show this message and exit
+
+Arguments:
+  <bucket>       Name of S3 bucket to restore the backup from
+  <backupkey>    The S3 key of the backup to restore
+  <destination>  Directory to restore to
+```
+
+```
+Usage: s3backup-tool restore-file [<options>] <bucket> <backupkey> <filepath> <destination>
+
+  Restore a single file from a backup from AWS S3.
+
+Options:
+  -h, --help  Show this message and exit
+
+Arguments:
+  <bucket>       Name of S3 bucket to restore the backup from
+  <backupkey>    The S3 key of the backup to restore
+  <filepath>     File path within the backup
+  <destination>  Directory to restore to
+```
+
 ## Assumptions
-1. This test task is not interested in re-implementations of common libraries (AWS SDK, Clikt, Gradle Shadow, ...)
-2. The last part (restoration of a single file) should be optimised so that only the part of the blob required for this 
-   file is downloaded.
-3. Only this tool is ever used to create backups, so S3 object keys are in the expected format, and ZIP files do not have
-   a comment in the *end of central directory* record (making it a predictable length of 22 bytes).
-   - EOCD64 should similarly not have a comment.
+1. The test task is not interested in re-implementations of common libraries (AWS SDK, Clikt, Gradle Shadow, ...)
+2. The test task is more interested in Kotlin JVM (not Kotlin Native).

 ## Design decisions
- Backups may be large, so we want to use multipart uploads if possible (< 100mb is recommended).
+- Backups may be large, so we want to use **multipart uploads** if possible (< 100mb is recommended).
  https://docs.aws.amazon.com/AmazonS3/latest/userguide/mpuoverview.html
  - The Java SDK has high-level support for this via [S3TransferManager](https://sdk.amazonaws.com/java/api/latest/software/amazon/awssdk/transfer/s3/S3TransferManager.html),
    but unfortunately when the content is too small, the HTTP `Content-Length` is not automatically calculated resulting
    in an error response from the API.
-  - I'm not sure whether this is intended behaviour or a bug, but decided to manually implement multipart uploads using
-    the Kotlin SDK instead anyway.
-  - **Note**: I could have just used a temporary file (with a known `Content-Length`), but I wanted to play around with
-    streams and kotlin concurrency a bit, which is why I went with the more scalable way using streams.
- Zip files are used so that the backups can be stored in a very common format which also provides compression.
-  - Java zip specification: https://docs.oracle.com/en/java/javase/17/docs/api/java.base/java/util/zip/package-summary.html
+    - I'm not sure whether this is intended behaviour or a bug, but decided to manually implement multipart uploads using
+      the Kotlin SDK instead anyway.
+- **ZIP files** are used so that the backups can be stored in a very common format which also provides compression.
+  - Allows future expansion to allow for encryption as well.
+  - [Java ZIP specification](https://docs.oracle.com/en/java/javase/17/docs/api/java.base/java/util/zip/package-summary.html): 
    - ZIP64 implementation is optional, but possible, so we'll handle it.
  - The End of Central Directory record is also useful for locating the exact positions of files in the blob, so that
    single files can be downloaded using the HTTP `Range` header.
-  - End of Central Directory comment must be blank (assumption 3). Otherwise, the EOCD length is unpredictable and so we
+  - End of Central Directory comment must be blank. Otherwise, the EOCD length is unpredictable, and so we
    cannot use just a single request the HTTP `Range` header to get the entire EOCD.
-    - Alternative: use S3 object tags to store the EOCD offset, but this way the blob itself would no longer contain all
-      the data required by this backup tool.
-    - Alternative: store the EOCD offset in the EOCD comment or the beginning of the file, but this makes a similar, but
-      more strict assumption anyway.
+    - *Alternative*: use S3 object tags to store the EOCD size, fallback to 22 bytes otherwise. This could be
+      interesting if we want the backup tool to be able to import existing ZIPs (which could potentially have a comment),
+      but that is beyond the scope of the instructions.
+- Only the `BackupClient` is tested, since by testing it, all other (internal) classes are functions are tested as well.

 ## Instructions
 Create a backup utility that copies files to AWS S3. The utility should take a local directory with files and put it into AWS S3 in the form of one blob file. The reverse behavior should also be possible. We should be able to specify what backup we want to restore and where it should put the files on the local system. The utility should be able to restore one individual file from a backup.
--- a/build.gradle.kts
+++ b/build.gradle.kts
@@ -1,10 +1,12 @@
 plugins {
+    application
    kotlin("jvm") version "1.9.21"
+    id("org.jlleitschuh.gradle.ktlint") version "12.0.3"
    id("com.github.johnrengelman.shadow") version "8.1.1"
 }

-group = "net.koval"
-version = "1.0-SNAPSHOT"
+group = "net.koval.teamcity-executors-test-task"
+version = System.getenv("BACKUP_VERSION")

 repositories {
    mavenCentral()
@@ -12,7 +14,9 @@ repositories {

 dependencies {
    implementation("aws.sdk.kotlin:s3:1.0.25")
+    implementation("org.slf4j:slf4j-simple:2.0.9") // AWS SDK wants a SLF4J provider.
    implementation("org.jetbrains.kotlinx:kotlinx-coroutines-core:1.7.3")
+    implementation("com.github.ajalt.clikt:clikt:4.2.1")
    testImplementation("org.jetbrains.kotlin:kotlin-test")
 }

@@ -22,8 +26,6 @@ tasks.test {
 kotlin {
    jvmToolchain(17)
 }
-tasks.jar {
-    manifest {
-        attributes("Main-Class" to "backup.MainKt")
-    }
+application {
+    mainClass.set("MainKt")
 }
--- a/settings.gradle.kts
+++ b/settings.gradle.kts
@@ -1,5 +1,4 @@
 plugins {
    id("org.gradle.toolchains.foojay-resolver-convention") version "0.5.0"
 }
-rootProject.name = "teamcity-executors-test-task"
-
+rootProject.name = "s3backup-tool"
--- a/src/main/kotlin/Main.kt
+++ b/src/main/kotlin/Main.kt
@@ -0,0 +1,80 @@
+import aws.sdk.kotlin.services.s3.S3Client
+import backup.BackupClient
+import com.github.ajalt.clikt.core.CliktCommand
+import com.github.ajalt.clikt.core.subcommands
+import com.github.ajalt.clikt.parameters.arguments.argument
+import com.github.ajalt.clikt.parameters.arguments.help
+import com.github.ajalt.clikt.parameters.types.file
+import kotlinx.coroutines.runBlocking
+import kotlin.system.exitProcess
+
+fun main(args: Array<String>) =
+    runBlocking {
+        S3Client.fromEnvironment().use { s3 ->
+            S3BackupTool()
+                .subcommands(
+                    Create(s3),
+                    Restore(s3),
+                    RestoreFile(s3),
+                )
+                .main(args)
+        }
+    }
+
+class S3BackupTool : CliktCommand(
+    help = "A simple AWS S3 backup tool. This tool assumes credentials are properly configured using aws-cli.",
+) {
+    override fun run() {
+        shortHelp(currentContext)
+    }
+}
+
+class Create(val s3: S3Client) : CliktCommand(
+    help = "Create a backup of a file or directory.",
+) {
+    val source by argument().file(mustExist = true).help("File or directory to backup")
+    val bucket by argument().help("Name of S3 bucket to backup to")
+
+    override fun run() =
+        runBlocking {
+            val backupKey = BackupClient(s3, bucket).upload(source)
+            echo("Successfully created backup with key '$backupKey'")
+        }
+}
+
+class Restore(val s3: S3Client) : CliktCommand(
+    help = "Restore a backup from AWS S3.",
+) {
+    val bucket by argument().help("Name of S3 bucket to restore the backup from")
+    val backupKey by argument().help("The S3 key of the backup to restore")
+    val destination by argument().file(mustExist = true).help("Directory to restore to")
+
+    override fun run() =
+        runBlocking {
+            if (!destination.isDirectory) {
+                echo("Destination must be an existing directory", err = true)
+                exitProcess(1)
+            }
+            BackupClient(s3, bucket).restore(destination.toPath(), backupKey)
+            echo("Successfully restored backup '$backupKey' to '$destination'")
+        }
+}
+
+class RestoreFile(val s3: S3Client) : CliktCommand(
+    help = "Restore a single file from a backup from AWS S3.",
+) {
+    val bucket by argument().help("Name of S3 bucket to restore the backup from")
+    val backupKey by argument().help("The S3 key of the backup to restore")
+    val filePath by argument().help("File path within the backup")
+    val destination by argument().file(mustExist = true).help("Directory to restore to")
+
+    override fun run() =
+        runBlocking {
+            if (!destination.isDirectory) {
+                echo("Destination must be an existing directory", err = true)
+                exitProcess(1)
+            }
+            BackupClient(s3, bucket).restoreFile(destination.toPath(), backupKey, filePath)
+            echo("Successfully restored '$filePath' from backup '$backupKey' to '$destination'")
+        }
+}
--- a/src/main/kotlin/backup/BackupClient.kt
+++ b/src/main/kotlin/backup/BackupClient.kt
@@ -24,170 +24,218 @@ import java.util.zip.ZipInputStream
 import java.util.zip.ZipOutputStream
 import kotlin.io.path.createDirectory

+/**
+ * AWS S3 backup client.
+ */
 class BackupClient(
    private val s3: S3Client,
    private val bucketName: String,
-    private val bufSize: Int = 1024 * 1024 * 100
+    private val bufSize: Int = 1024 * 1024 * 32,
 ) {
-    suspend fun upload(file: File) = coroutineScope {
-        val backupKey = "${file.name}/${Instant.now()}.zip"
-        PipedInputStream().use { inputStream ->
-            val outputStream = PipedOutputStream(inputStream)
-            val zipper = launch(Dispatchers.IO) {
-                file.compressToZip(outputStream)
-            }
+    /**
+     * Upload a file/directory backup to AWS S3.
+     * @param file The File object for the file or directory.
+     */
+    suspend fun upload(file: File) =
+        coroutineScope {
+            val backupKey = "${file.canonicalFile.name}/${Instant.now()}.zip"
+            PipedInputStream().use { inputStream ->
+                val outputStream = PipedOutputStream(inputStream)
+                val zipper =
+                    launch(Dispatchers.IO) {
+                        file.compressToZip(outputStream)
+                    }

-            val data = ByteArray(bufSize)
-            val initialRead = inputStream.readNBytes(data, 0, bufSize)
-            if (initialRead == bufSize) {
-                // Large upload, use multipart
-                // TODO: multipart uploads can be asynchronous, which would improve
-                //       performance a little bit for big uploads.
-                val upload = s3.createMultipartUpload {
-                    bucket = bucketName
-                    key = backupKey
-                }
-                try {
-                    val uploadParts = mutableListOf<CompletedPart>()
-                    var number = 1
-                    var bytesRead = initialRead
-                    while (bytesRead > 0) {
-                        val part = s3.uploadPart {
+                val data = ByteArray(bufSize)
+                val initialRead = inputStream.readNBytes(data, 0, bufSize)
+                if (initialRead == bufSize) {
+                    // Large upload, use multipart
+                    // TODO: multipart uploads can be asynchronous, which would improve
+                    //       performance a little bit for big uploads.
+                    val upload =
+                        s3.createMultipartUpload {
                            bucket = bucketName
                            key = backupKey
-                            partNumber = number
-                            uploadId = upload.uploadId
-                            body = ByteStream.fromBytes(data.take(bytesRead))
-                        }.asCompletedPart(number)
-                        uploadParts.add(part)
-                        number++
-                        bytesRead = inputStream.readNBytes(data, 0, bufSize)
-                    }
-                    s3.completeMultipartUpload {
-                        bucket = bucketName
-                        key = backupKey
-                        uploadId = upload.uploadId
-                        multipartUpload = CompletedMultipartUpload {
-                            parts = uploadParts
                        }
+                    try {
+                        val uploadParts = mutableListOf<CompletedPart>()
+                        var number = 1
+                        var bytesRead = initialRead
+                        while (bytesRead > 0) {
+                            val part =
+                                s3.uploadPart {
+                                    bucket = bucketName
+                                    key = backupKey
+                                    partNumber = number
+                                    uploadId = upload.uploadId
+                                    body = ByteStream.fromBytes(data.take(bytesRead))
+                                }.toCompletedPart(number)
+                            uploadParts.add(part)
+                            number++
+                            bytesRead = inputStream.readNBytes(data, 0, bufSize)
+                        }
+                        s3.completeMultipartUpload {
+                            bucket = bucketName
+                            key = backupKey
+                            uploadId = upload.uploadId
+                            multipartUpload =
+                                CompletedMultipartUpload {
+                                    parts = uploadParts
+                                }
+                        }
+                    } catch (e: Exception) {
+                        s3.abortMultipartUpload {
+                            bucket = bucketName
+                            key = backupKey
+                            uploadId = upload.uploadId
+                        }
+                        throw e
                    }
-                } catch (e: Exception) {
-                    s3.abortMultipartUpload {
+                } else {
+                    // Small upload, use single request
+                    s3.putObject {
                        bucket = bucketName
                        key = backupKey
-                        uploadId = upload.uploadId
+                        body = ByteStream.fromBytes(data.take(initialRead))
                    }
-                    throw e
-                }
-            } else {
-                // Small upload, use single request
-                s3.putObject {
-                    bucket = bucketName
-                    key = backupKey
-                    body = ByteStream.fromBytes(data.take(initialRead))
                }
+                zipper.join() // Should be instant
            }
-            zipper.join() // Should be instant
+            backupKey
        }
-        backupKey
-    }

-    suspend fun restore(destination: Path, backupKey: String) = coroutineScope {
-        val req = GetObjectRequest {
-            bucket = bucketName
-            key = backupKey
-        }
+    /**
+     * Restore a backup from AWS S3.
+     * @param destination The destination directory path for the backup contents.
+     * @param backupKey The S3 key of the backup.
+     */
+    suspend fun restore(
+        destination: Path,
+        backupKey: String,
+    ) = coroutineScope {
+        val req =
+            GetObjectRequest {
+                bucket = bucketName
+                key = backupKey
+            }
        s3.getObject(req) { resp ->
            ZipInputStream(
                resp.body?.toInputStream()
-                    ?: throw IOException("S3 response is missing body")
+                    ?: throw IOException("S3 response is missing body"),
            ).use { zipStream ->
                zipStream.decompress { destination.resolve(it) }
            }
        }
    }

-    suspend fun restoreFile(destination: Path, backupKey: String, fileName: String) = coroutineScope {
+    /**
+     * Restore a single file from a backup from AWS S3.
+     * @param destination The destination directory path for the file from the backup.
+     * @param backupKey The S3 key of the backup.
+     * @param fileName The full name of the file to restore (including directories if it was under a subdirectory).
+     */
+    suspend fun restoreFile(
+        destination: Path,
+        backupKey: String,
+        fileName: String,
+    ) = coroutineScope {
        // For byte ranges refer to https://pkware.cachefly.net/webdocs/APPNOTE/APPNOTE-6.3.9.TXT
-        val eocdReq = GetObjectRequest {
-            bucket = bucketName
-            key = backupKey
-            // Assumption: EOCD has an empty comment
-            // Assumption: Backups are at least 22 + 20 (= 42) bytes. Only COMPLETELY empty backups can be smaller,
-            //             in which case this function would error anyway, so it should be fine to have this edge-case.
-            range = "bytes=-${EndOfCentralDirectoryRecord.SIZE + EndOfCentralDirectoryLocator.SIZE}"
-        }
-        val eocdBytes = s3.getObject(eocdReq) { resp ->
-            val bytes = resp.body?.toByteArray() ?: throw IOException("S3 response is missing body")
-            bytes
-        }
-        val eocd = EndOfCentralDirectoryRecord.fromByteArray(eocdBytes, EndOfCentralDirectoryLocator.SIZE)
-        val eocd64 = if (eocd.eocd64Required()) {
-            val locator = EndOfCentralDirectoryLocator.fromByteArray(eocdBytes, 0)
-            val eocd64Req = GetObjectRequest {
+        val eocdReq =
+            GetObjectRequest {
                bucket = bucketName
                key = backupKey
-                range = "bytes=${locator.endOfCentralDirectory64Offset}-"
+                // Assumption: EOCD has an empty comment
+                // Assumption: Backups are at least 22 + 20 (= 42) bytes. Only COMPLETELY empty backups can be smaller,
+                //             in which case this function would error anyway, so it should be fine to have this edge-case.
+                range = "bytes=-${EndOfCentralDirectoryRecord.SIZE + EndOfCentralDirectoryLocator.SIZE}"
            }
-            s3.getObject(eocd64Req) { resp ->
+        val eocdBytes =
+            s3.getObject(eocdReq) { resp ->
                val bytes = resp.body?.toByteArray() ?: throw IOException("S3 response is missing body")
-                EndOfCentralDirectoryRecord64.fromByteArray(bytes, 0)
+                bytes
            }
-        } else null
-        val cenOffset = if (eocd.centralDirectoryOffset == 0xffffffffU && eocd64 != null) {
-            eocd64.centralDirectoryOffset
-        } else eocd.centralDirectoryOffset.toULong()
-        val censReq = GetObjectRequest {
-            bucket = bucketName
-            key = backupKey
-            // We only know where to fetch until if we've also fetched EOCD64 (which isn't always the case).
-            // So just over-fetch a little bit, these headers aren't that big anyway.
-            range = "bytes=${cenOffset}-"
-        }
-        val cen = s3.getObject(censReq) { resp ->
-            val bytes = resp.body?.toByteArray() ?: throw IOException("S3 response is missing body")
-            var p = 0
-            while (p < bytes.size) {
-                try {
-                    val cen = CentralDirectoryFileHeader.fromByteArray(bytes, p)
-                    p += cen.size
-                    if (cen.fileName == fileName) return@getObject cen
-                } catch (_: InvalidSignatureException) {
-                    return@getObject null
+        val eocd = EndOfCentralDirectoryRecord.fromByteArray(eocdBytes, EndOfCentralDirectoryLocator.SIZE)
+        val eocd64 =
+            if (eocd.eocd64Required()) {
+                val locator = EndOfCentralDirectoryLocator.fromByteArray(eocdBytes, 0)
+                val eocd64Req =
+                    GetObjectRequest {
+                        bucket = bucketName
+                        key = backupKey
+                        range = "bytes=${locator.endOfCentralDirectory64Offset}-"
+                    }
+                s3.getObject(eocd64Req) { resp ->
+                    val bytes = resp.body?.toByteArray() ?: throw IOException("S3 response is missing body")
+                    EndOfCentralDirectoryRecord64.fromByteArray(bytes, 0)
                }
+            } else {
+                null
            }
-            null
-        } ?: throw FileNotFoundException("File '${fileName}' not found in backup")
+        val cenOffset =
+            if (eocd.centralDirectoryOffset == 0xffffffffU && eocd64 != null) {
+                eocd64.centralDirectoryOffset
+            } else {
+                eocd.centralDirectoryOffset.toULong()
+            }
+        val censReq =
+            GetObjectRequest {
+                bucket = bucketName
+                key = backupKey
+                // We only know where to fetch until if we've also fetched EOCD64 (which isn't always the case).
+                // So just over-fetch a little bit, these headers aren't that big anyway.
+                range = "bytes=$cenOffset-"
+            }
+        val cen =
+            s3.getObject(censReq) { resp ->
+                val bytes = resp.body?.toByteArray() ?: throw IOException("S3 response is missing body")
+                var p = 0
+                while (p < bytes.size) {
+                    try {
+                        val cen = CentralDirectoryFileHeader.fromByteArray(bytes, p)
+                        p += cen.size
+                        if (cen.fileName == fileName) return@getObject cen
+                    } catch (_: InvalidDataException) {
+                        return@getObject null
+                    }
+                }
+                null
+            } ?: throw FileNotFoundException("File '$fileName' not found in backup")

-        val localHeaderOffset = cen.extraFieldRecords.firstNotNullOfOrNull {
-            if (it is Zip64ExtraFieldRecord && it.localHeaderOffset != null) it else null
-        }?.localHeaderOffset ?: cen.localHeaderOffset.toULong()
-        val compressedSize = cen.extraFieldRecords.firstNotNullOfOrNull {
-            if (it is Zip64ExtraFieldRecord && it.compressedSize != null) it else null
-        }?.compressedSize ?: cen.compressedSize.toULong()
-        val req = GetObjectRequest {
-            bucket = bucketName
-            key = backupKey
-            range = "bytes=${localHeaderOffset}-${
-                // Add CEN min size (46 bytes) so that the next CEN / LOC header is seen by the ZipInputStream
-                // and so it can see the current entry has stopped.
-                // Note: yes ZipInputStream should know the exact content length from the LOC, but it was still sending
-                // EOF errors. Perhaps due to fetching multiples of a power of two, or something else. But this helps.
-                localHeaderOffset + cen.size.toULong() + compressedSize + CentralDirectoryFileHeader.SIZE.toULong()
-            }"
-        }
+        val localHeaderOffset =
+            cen.extraFieldRecords.firstNotNullOfOrNull {
+                if (it is Zip64ExtraFieldRecord && it.localHeaderOffset != null) it else null
+            }?.localHeaderOffset ?: cen.localHeaderOffset.toULong()
+        val compressedSize =
+            cen.extraFieldRecords.firstNotNullOfOrNull {
+                if (it is Zip64ExtraFieldRecord && it.compressedSize != null) it else null
+            }?.compressedSize ?: cen.compressedSize.toULong()
+        val req =
+            GetObjectRequest {
+                bucket = bucketName
+                key = backupKey
+                range = "bytes=$localHeaderOffset-${
+                    // Over-fetch a bit so that ZipInputStream can see the next header (otherwise it EOFs, even though
+                    // it knows exactly how much data is should read, so not sure why it reads ahead).
+                    localHeaderOffset + cen.size.toULong() + compressedSize + CentralDirectoryFileHeader.SIZE.toULong()
+                }"
+            }
        s3.getObject(req) { resp ->
            ZipInputStream(
                resp.body?.toInputStream()
-                    ?: throw IOException("S3 response is missing body")
+                    ?: throw IOException("S3 response is missing body"),
            ).use { zipStream ->
-                zipStream.decompress { name -> destination.resolve(name.takeLastWhile { it != '/' }) }
+                zipStream.decompress(limit = 1) { name -> destination.resolve(name.takeLastWhile { it != '/' }) }
            }
        }
    }
 }

-private fun UploadPartResponse.asCompletedPart(number: Int): CompletedPart {
+/**
+ * Convert an UploadPartResponse to a CompletedPart.
+ * @param number The part number that was used for this part upload.
+ * @return The CompletedPart object.
+ */
+internal fun UploadPartResponse.toCompletedPart(number: Int): CompletedPart {
    val part = this
    return CompletedPart {
        partNumber = number
@@ -199,38 +247,58 @@ private fun UploadPartResponse.asCompletedPart(number: Int): CompletedPart {
    }
 }

-private fun ByteArray.take(n: Int) =
-    if (n == size) this // No copy
-    else asList().subList(0, n).toByteArray() // TODO: One copy (toByteArray()), not sure how to do 0 copies here
+/**
+ * Take first `n` items from the beginning of a ByteArray.
+ * @param n The number of items to take.
+ * @return A ByteArray of the first `n` items.
+ */
+internal fun ByteArray.take(n: Int) =
+    if (n == size) {
+        this // No copy
+    } else {
+        asList().subList(0, n).toByteArray() // TODO: One copy (toByteArray()), not sure how to do 0 copies here
+    }

-private fun File.compressToZip(outputStream: OutputStream) = ZipOutputStream(outputStream).use { zipStream ->
-    val parentDir = this.absoluteFile.parent + "/"
-    val fileQueue = ArrayDeque<File>()
-    fileQueue.add(this)
-    fileQueue.forEach { subFile ->
-        val path = subFile.absolutePath.removePrefix(parentDir)
-        val subFiles = subFile.listFiles()
-        if (subFiles != null) { // Is a directory
-            val entry = ZipEntry("$path/")
-            setZipAttributes(entry, subFile.toPath())
-            zipStream.putNextEntry(entry)
-            fileQueue.addAll(subFiles)
-        } else { // Otherwise, treat it as a file
-            BufferedInputStream(subFile.inputStream()).use { origin ->
-                val entry = ZipEntry(path)
+/**
+ * Compress a file or directory as a ZIP file to an `OutputStream`.
+ * @param outputStream The `OutputStream` to write the ZIP file contents to.
+ */
+internal fun File.compressToZip(outputStream: OutputStream) =
+    ZipOutputStream(outputStream).use { zipStream ->
+        val parentDir = this.canonicalFile.parent + "/"
+        val fileQueue = ArrayDeque<File>()
+        fileQueue.add(this)
+        fileQueue.forEach { subFile ->
+            val path = subFile.canonicalPath.removePrefix(parentDir)
+            val subFiles = subFile.listFiles()
+            if (subFiles != null) { // Is a directory
+                val entry = ZipEntry("$path/")
                setZipAttributes(entry, subFile.toPath())
                zipStream.putNextEntry(entry)
-                origin.copyTo(zipStream)
+                fileQueue.addAll(subFiles)
+            } else { // Otherwise, treat it as a file
+                BufferedInputStream(subFile.inputStream()).use { origin ->
+                    val entry = ZipEntry(path)
+                    setZipAttributes(entry, subFile.toPath())
+                    zipStream.putNextEntry(entry)
+                    origin.copyTo(zipStream)
+                }
            }
        }
    }
-}

-private fun ZipInputStream.decompress(
+/**
+ * Decompress `ZipInputStream` contents to specified destination paths.
+ * @param bufSize The buffer size to use for writing the decompressed files.
+ * @param entryNameToPath A function to convert ZIP entry names to destination `Path`s.
+ */
+internal fun ZipInputStream.decompress(
    bufSize: Int = 1024 * 1024,
-    entryNameToPath: (String) -> Path
+    limit: Int? = null,
+    entryNameToPath: (String) -> Path,
 ) {
    var entry = this.nextEntry
+    var count = 1
    while (entry != null) {
        val path = entryNameToPath(entry.name)
        if (entry.isDirectory) {
@@ -246,11 +314,22 @@ private fun ZipInputStream.decompress(
            }
        }
        applyZipAttributes(entry, path)
+        // This is here, not in while loop, since we do not want to read more from the input stream.
+        // But this.nextEntry will read from the input stream.
+        if (limit != null && count++ >= limit) return
        entry = this.nextEntry
    }
 }

-private fun setZipAttributes(entry: ZipEntry, path: Path) {
+/**
+ * Set a `ZipEntry`'s attributes given a file's path.
+ * @param entry The `ZipEntry` to set attributes of.
+ * @param path The `Path` of the file to get the attributes from.
+ */
+internal fun setZipAttributes(
+    entry: ZipEntry,
+    path: Path,
+) {
    try {
        val attrs = Files.getFileAttributeView(path, BasicFileAttributeView::class.java).readAttributes()
        entry.setCreationTime(attrs.creationTime())
@@ -260,7 +339,15 @@ private fun setZipAttributes(entry: ZipEntry, path: Path) {
    }
 }

-private fun applyZipAttributes(entry: ZipEntry, path: Path) {
+/**
+ * Set a file's attributes given a `ZipEntry`.
+ * @param entry The `ZipEntry` to get the attributes from.
+ * @param path The `Path` of the file to set the attributes of.
+ */
+internal fun applyZipAttributes(
+    entry: ZipEntry,
+    path: Path,
+) {
    try {
        val attrs = Files.getFileAttributeView(path, BasicFileAttributeView::class.java)
        attrs.setTimes(entry.lastModifiedTime, entry.lastAccessTime, entry.creationTime)
--- a/src/main/kotlin/backup/main.kt
+++ b/src/main/kotlin/backup/main.kt
@@ -1,13 +0,0 @@
-package backup
-
-import aws.sdk.kotlin.services.s3.S3Client
-import kotlinx.coroutines.Dispatchers
-import kotlinx.coroutines.runBlocking
-import java.io.File
-import kotlin.io.path.Path
-
-fun main() = runBlocking {
-    S3Client.fromEnvironment().use { s3 ->
-        val backupClient = BackupClient(s3, "teamcity-executors-test-task", 1024 * 1024 * 10)
-    }
-}
--- a/src/main/kotlin/ziputils/CentralDirectoryFileHeader.kt
+++ b/src/main/kotlin/ziputils/CentralDirectoryFileHeader.kt
@@ -3,6 +3,9 @@ package ziputils
 import java.nio.ByteBuffer
 import java.nio.ByteOrder

+/**
+ * Represents a partial ZIP central directory file header.
+ */
 internal class CentralDirectoryFileHeader(
    val compressedSize: UInt,
    val uncompressedSize: UInt,
@@ -12,7 +15,7 @@ internal class CentralDirectoryFileHeader(
    val disk: UShort,
    val localHeaderOffset: UInt,
    val fileName: String,
-    val extraFieldRecords: List<ExtraFieldRecord>
+    val extraFieldRecords: List<ExtraFieldRecord>,
 ) {
    val size: Int
        get() = SIZE + nameLength.toInt() + extraFieldLength.toInt() + commentLength.toInt()
@@ -22,69 +25,92 @@ internal class CentralDirectoryFileHeader(
        const val SIZE = 46

        /**
-         * Create CentralDirectoryFileHeader from raw byte data.
-         * @throws InvalidDataException provided ByteArray is not a supported CEN.
+         * Create `CentralDirectoryFileHeader` from raw byte data.
+         * @throws InvalidDataException provided `ByteArray` is not a supported CEN.
+         * @param data Raw byte data.
+         * @param offset Skip first <offset> bytes in data array.
+         * @return A `CentralDirectoryFileHeader`.
         */
        @Throws(InvalidDataException::class)
-        fun fromByteArray(data: ByteArray, offset: Int): CentralDirectoryFileHeader {
+        fun fromByteArray(
+            data: ByteArray,
+            offset: Int,
+        ): CentralDirectoryFileHeader {
            if (data.size - offset < SIZE) {
                throw InvalidDataException("CEN must be at least 46 bytes")
            }
            val buf = ByteBuffer.wrap(data, offset, 46).order(ByteOrder.LITTLE_ENDIAN)
            if (buf.getInt().toUInt() != SIGNATURE) {
-                throw InvalidSignatureException("Invalid signature")
+                throw InvalidDataException("Invalid signature")
            }

            val extraFieldRecords = mutableListOf<ExtraFieldRecord>()
            val nameLength = buf.getShort(offset + 28).toUShort()
            buf.position(offset + 20)
-            val cen = CentralDirectoryFileHeader(
-                compressedSize = buf.getInt().toUInt(),
-                uncompressedSize = buf.getInt().toUInt(),
-                nameLength = nameLength
-                    .also { buf.position(offset + 30) },
-                extraFieldLength = buf.getShort().toUShort(),
-                commentLength = buf.getShort().toUShort(),
-                disk = buf.getShort().toUShort()
-                    .also { buf.position(offset + 42) },
-                localHeaderOffset = buf.getInt().toUInt(),
-                fileName = String(data.sliceArray(offset + SIZE..<offset + SIZE + nameLength.toInt())),
-                extraFieldRecords = extraFieldRecords
-            )
+            val cen =
+                CentralDirectoryFileHeader(
+                    compressedSize = buf.getInt().toUInt(),
+                    uncompressedSize = buf.getInt().toUInt(),
+                    nameLength =
+                        nameLength
+                            .also { buf.position(offset + 30) },
+                    extraFieldLength = buf.getShort().toUShort(),
+                    commentLength = buf.getShort().toUShort(),
+                    disk =
+                        buf.getShort().toUShort()
+                            .also { buf.position(offset + 42) },
+                    localHeaderOffset = buf.getInt().toUInt(),
+                    fileName = String(data.sliceArray(offset + SIZE..<offset + SIZE + nameLength.toInt())),
+                    extraFieldRecords = extraFieldRecords,
+                )
            if (data.size - offset < cen.size) {
                throw InvalidDataException("CEN is too short")
            }

            // Parse extra field records
-            val extraFieldsBuf = ByteBuffer.wrap(
-                data, offset + SIZE + cen.nameLength.toInt(), cen.extraFieldLength.toInt()
-            ).order(ByteOrder.LITTLE_ENDIAN)
+            val extraFieldsBuf =
+                ByteBuffer.wrap(
+                    data,
+                    offset + SIZE + cen.nameLength.toInt(),
+                    cen.extraFieldLength.toInt(),
+                ).order(ByteOrder.LITTLE_ENDIAN)
            while (extraFieldsBuf.remaining() > 0) {
                val id = extraFieldsBuf.getShort().toUShort()
                val size = extraFieldsBuf.getShort().toUShort()
-                extraFieldRecords.add(when (id) {
-                    Zip64ExtraFieldRecord.ID -> {
-                        Zip64ExtraFieldRecord(
-                            size,
-                            if (cen.uncompressedSize == 0xffffffffU) {
-                                extraFieldsBuf.getLong().toULong()
-                            } else null,
-                            if (cen.compressedSize == 0xffffffffU) {
-                                extraFieldsBuf.getLong().toULong()
-                            } else null,
-                            if (cen.localHeaderOffset == 0xffffffffU) {
-                                extraFieldsBuf.getLong().toULong()
-                            } else null,
-                            if (cen.disk == 0xffffU.toUShort()) {
-                                extraFieldsBuf.getInt().toUInt()
-                            } else null
-                        )
-                    }
-                    else -> {
-                        extraFieldsBuf.position(extraFieldsBuf.position() + size.toInt())
-                        ExtraFieldRecord(id, size)
-                    }
-                })
+                extraFieldRecords.add(
+                    when (id) {
+                        Zip64ExtraFieldRecord.ID -> {
+                            Zip64ExtraFieldRecord(
+                                size,
+                                if (cen.uncompressedSize == 0xffffffffU) {
+                                    extraFieldsBuf.getLong().toULong()
+                                } else {
+                                    null
+                                },
+                                if (cen.compressedSize == 0xffffffffU) {
+                                    extraFieldsBuf.getLong().toULong()
+                                } else {
+                                    null
+                                },
+                                if (cen.localHeaderOffset == 0xffffffffU) {
+                                    extraFieldsBuf.getLong().toULong()
+                                } else {
+                                    null
+                                },
+                                if (cen.disk == 0xffffU.toUShort()) {
+                                    extraFieldsBuf.getInt().toUInt()
+                                } else {
+                                    null
+                                },
+                            )
+                        }
+
+                        else -> {
+                            extraFieldsBuf.position(extraFieldsBuf.position() + size.toInt())
+                            ExtraFieldRecord(id, size)
+                        }
+                    },
+                )
            }

            return cen
--- a/src/main/kotlin/ziputils/EndOfCentralDirectoryLocator.kt
+++ b/src/main/kotlin/ziputils/EndOfCentralDirectoryLocator.kt
@@ -3,20 +3,34 @@ package ziputils
 import java.nio.ByteBuffer
 import java.nio.ByteOrder

+/**
+ * Represents a partial ZIP64 end of central directory locator.
+ */
 internal class EndOfCentralDirectoryLocator(
-    val endOfCentralDirectory64Offset: ULong
+    val endOfCentralDirectory64Offset: ULong,
 ) {
    companion object {
        const val SIGNATURE = 0x07064b50U
        const val SIZE = 20
+
+        /**
+         * Create `EndOfCentralDirectoryLocator` from raw byte data.
+         * @throws InvalidDataException Provided `ByteArray` is not a supported EOCD locator.
+         * @param data Raw byte data.
+         * @param offset Skip first <offset> bytes in data array.
+         * @return A `EndOfCentralDirectoryLocator`.
+         */
        @Throws(InvalidDataException::class)
-        fun fromByteArray(data: ByteArray, offset: Int): EndOfCentralDirectoryLocator {
+        fun fromByteArray(
+            data: ByteArray,
+            offset: Int,
+        ): EndOfCentralDirectoryLocator {
            if (data.size - offset < SIZE) {
                throw InvalidDataException("EOCD64 locator must be at least 20 bytes")
            }
            val buf = ByteBuffer.wrap(data, offset, SIZE).order(ByteOrder.LITTLE_ENDIAN)
            if (buf.getInt().toUInt() != SIGNATURE) {
-                throw InvalidSignatureException("Invalid signature")
+                throw InvalidDataException("Invalid signature")
            }
            buf.position(offset + 8)
            return EndOfCentralDirectoryLocator(buf.getLong().toULong())
--- a/src/main/kotlin/ziputils/EndOfCentralDirectoryRecord.kt
+++ b/src/main/kotlin/ziputils/EndOfCentralDirectoryRecord.kt
@@ -4,34 +4,39 @@ import java.nio.ByteBuffer
 import java.nio.ByteOrder

 /**
- * Partial End of Central Directory record class.
- * Only supports data required by the backup tool.
+ * Represents a partial ZIP end of central directory record.
 */
 internal class EndOfCentralDirectoryRecord(
-    val centralDirectoryOffset: UInt
+    val centralDirectoryOffset: UInt,
 ) {
-    fun eocd64Required(): Boolean =
-        centralDirectoryOffset == 0xffffffffU
+    fun eocd64Required(): Boolean = centralDirectoryOffset == 0xffffffffU

    companion object {
        const val SIGNATURE = 0x06054b50U
        const val SIZE = 22
+
        /**
-         * Create EndOfCentralDirectoryRecord from raw byte data.
-         * @throws InvalidDataException provided ByteArray is not a supported EOCD64.
+         * Create `EndOfCentralDirectoryRecord` from raw byte data.
+         * @throws InvalidDataException Provided `ByteArray` is not a supported EOCD64.
+         * @param data Raw byte data.
+         * @param offset Skip first <offset> bytes in data array.
+         * @return A `EndOfCentralDirectoryRecord`.
         */
        @Throws(InvalidDataException::class)
-        fun fromByteArray(data: ByteArray, offset: Int): EndOfCentralDirectoryRecord {
+        fun fromByteArray(
+            data: ByteArray,
+            offset: Int,
+        ): EndOfCentralDirectoryRecord {
            if (data.size - offset < SIZE) {
                throw InvalidDataException("EOCD must be at least 22 bytes")
            }
            val buf = ByteBuffer.wrap(data, offset, SIZE).order(ByteOrder.LITTLE_ENDIAN)
            if (buf.getInt().toUInt() != SIGNATURE) {
-                throw InvalidSignatureException("Invalid signature")
+                throw InvalidDataException("Invalid signature")
            }
            buf.position(offset + 16)
            return EndOfCentralDirectoryRecord(
-                centralDirectoryOffset = buf.getInt().toUInt()
+                centralDirectoryOffset = buf.getInt().toUInt(),
            )
        }
    }
--- a/src/main/kotlin/ziputils/EndOfCentralDirectoryRecord64.kt
+++ b/src/main/kotlin/ziputils/EndOfCentralDirectoryRecord64.kt
@@ -4,31 +4,37 @@ import java.nio.ByteBuffer
 import java.nio.ByteOrder

 /**
- * Partial End of Central Directory record (ZIP64) class.
- * Only supports data required by the backup tool.
+ * Represents a partial ZIP64 end of central directory record.
 */
 internal class EndOfCentralDirectoryRecord64(
-    val centralDirectoryOffset: ULong
+    val centralDirectoryOffset: ULong,
 ) {
    companion object {
        const val SIGNATURE = 0x06064b50U
        const val SIZE = 56
+
        /**
-         * Create EndOfCentralDirectoryRecord64 from raw byte data.
-         * @throws InvalidDataException provided ByteArray is not a supported EOCD.
+         * Create `EndOfCentralDirectoryRecord64` from raw byte data.
+         * @throws InvalidDataException Provided `ByteArray` is not a supported EOCD.
+         * @param data Raw byte data.
+         * @param offset Skip first <offset> bytes in data array.
+         * @return A `EndOfCentralDirectoryRecord64`.
         */
        @Throws(InvalidDataException::class)
-        fun fromByteArray(data: ByteArray, offset: Int): EndOfCentralDirectoryRecord64 {
+        fun fromByteArray(
+            data: ByteArray,
+            offset: Int,
+        ): EndOfCentralDirectoryRecord64 {
            if (data.size - offset < SIZE) {
                throw InvalidDataException("EOCD64 must be at least 56 bytes")
            }
            val buf = ByteBuffer.wrap(data, offset, SIZE).order(ByteOrder.LITTLE_ENDIAN)
            if (buf.getInt().toUInt() != SIGNATURE) {
-                throw InvalidSignatureException("Invalid signature")
+                throw InvalidDataException("Invalid signature")
            }
            buf.position(offset + 48)
            return EndOfCentralDirectoryRecord64(
-                centralDirectoryOffset = buf.getLong().toULong()
+                centralDirectoryOffset = buf.getLong().toULong(),
            )
        }
    }
--- a/src/main/kotlin/ziputils/Exceptions.kt
+++ b/src/main/kotlin/ziputils/Exceptions.kt
@@ -1,4 +0,0 @@
-package ziputils
-
-class InvalidDataException(message: String): Exception(message)
-class InvalidSignatureException(message: String): Exception(message)
--- a/src/main/kotlin/ziputils/ExtraFieldRecord.kt
+++ b/src/main/kotlin/ziputils/ExtraFieldRecord.kt
@@ -1,6 +1,9 @@
 package ziputils

+/**
+ * Represents a partial ZIP extra field record.
+ */
 internal open class ExtraFieldRecord(
    val id: UShort,
-    val size: UShort
+    val size: UShort,
 )
--- a/src/main/kotlin/ziputils/InvalidDataException.kt
+++ b/src/main/kotlin/ziputils/InvalidDataException.kt
@@ -0,0 +1,6 @@
+package ziputils
+
+/**
+ * Represents an invalid raw byte data exception.
+ */
+class InvalidDataException(message: String) : Exception(message)
--- a/src/main/kotlin/ziputils/Zip64ExtraFieldRecord.kt
+++ b/src/main/kotlin/ziputils/Zip64ExtraFieldRecord.kt
@@ -1,12 +1,15 @@
 package ziputils

+/**
+ * Represents a ZIP ZIP64 extra field record (ID 0x0001).
+ */
 internal class Zip64ExtraFieldRecord(
    size: UShort,
    val uncompressedSize: ULong?,
    val compressedSize: ULong?,
    val localHeaderOffset: ULong?,
-    val disk: UInt?
-): ExtraFieldRecord(ID, size) {
+    val disk: UInt?,
+) : ExtraFieldRecord(ID, size) {
    companion object {
        const val ID: UShort = 0x0001U
    }
--- a/src/test/kotlin/backup/BackupClientTest.kt
+++ b/src/test/kotlin/backup/BackupClientTest.kt
@@ -0,0 +1,165 @@
+package backup
+
+import aws.sdk.kotlin.services.s3.*
+import kotlinx.coroutines.runBlocking
+import org.junit.jupiter.api.*
+import org.junit.jupiter.api.Assertions.*
+import java.io.File
+import kotlin.io.path.*
+import kotlin.random.Random
+
+val bucketName = System.getenv("BACKUP_BUCKET") ?: "teamcity-executors-test-task"
+
+class BackupClientTest {
+    lateinit var s3: S3Client
+    lateinit var backupClient: BackupClient
+
+    @BeforeEach
+    fun `before each`() =
+        runBlocking {
+            s3 = S3Client.fromEnvironment {}
+            backupClient = BackupClient(s3, bucketName, 1024 * 1024 * 10)
+        }
+
+    @AfterEach
+    fun `after each`() {
+        s3.close()
+    }
+
+    @TestFactory
+    fun `round-trip tests`() =
+        listOf(
+            "empty directory" to {
+                listOf(
+                    Path("_test").createDirectory(),
+                )
+            },
+            "single file" to {
+                listOf(
+                    Path("_test.txt").apply { writeText("Hello World!") },
+                )
+            },
+            "directory structure" to {
+                listOf(
+                    Path("_test").createDirectory(),
+                    Path("_test/a.txt").apply { writeText("This is file A!\nAnother line here.") },
+                    Path("_test/folder").createDirectory(),
+                    Path("_test/another-folder").createDirectory(),
+                    Path("_test/another-folder/b").apply { writeText("This is file B\n") },
+                    Path("_test/another-folder/c.txt").createFile(),
+                    Path("_test/README.md").apply { writeText("# This is a test directory structure.") },
+                )
+            },
+            "single large file" to {
+                val bytes = ByteArray(1024 * 1024 * 32)
+                Random.nextBytes(bytes)
+                listOf(
+                    Path("_test.txt").apply { writeBytes(bytes) },
+                )
+            },
+            "large directory structure" to {
+                val bytes1 = ByteArray(1024 * 1024 * 32)
+                val bytes2 = ByteArray(1024 * 1024 * 48)
+                listOf(
+                    Path("_test").createDirectory(),
+                    Path("_test/a.txt").apply { writeBytes(bytes1) },
+                    Path("_test/folder").createDirectory(),
+                    Path("_test/another-folder").createDirectory(),
+                    Path("_test/another-folder/b").apply { writeText("This is file B\n") },
+                    Path("_test/another-folder/c.txt").createFile(),
+                    Path("_test/README.md").apply { writeBytes(bytes2) },
+                )
+            },
+        ).map { (name, pathsGen) ->
+            DynamicTest.dynamicTest(name) {
+                val paths = pathsGen()
+                val backupKey =
+                    assertDoesNotThrow("should upload files") {
+                        runBlocking {
+                            backupClient.upload(paths.first().toFile())
+                        }
+                    }
+                val restoreDir = Path("_test_restore").createDirectory()
+                assertDoesNotThrow("should recover files") {
+                    runBlocking {
+                        backupClient.restore(restoreDir, backupKey)
+                    }
+                }
+                assertEquals(
+                    paths.size,
+                    restoreDir.toFile().countEntries() - 1,
+                    "number of files in backup restore should be equal to original",
+                )
+                val individualRestoreDir = Path("_test_restore_individual").createDirectory()
+                paths.forEach { path ->
+                    if (path.isDirectory()) {
+                        individualRestoreDir.resolve(path).createDirectory()
+                    } else {
+                        assertDoesNotThrow("should recover file '$path'") {
+                            runBlocking {
+                                backupClient.restoreFile(
+                                    individualRestoreDir.resolve(path).parent,
+                                    backupKey,
+                                    path.toString(),
+                                )
+                            }
+                        }
+                    }
+                }
+                assertEquals(
+                    paths.size,
+                    individualRestoreDir.toFile().countEntries() - 1,
+                    "number of files in individual backup restore should be equal to original",
+                )
+                paths.asReversed().forEach { path ->
+                    val restorePath = restoreDir.resolve(path)
+                    val individualRestorePath = individualRestoreDir.resolve(path)
+                    if (path.isDirectory()) {
+                        assertTrue(restorePath.exists(), "'$path' should exist in backup")
+                        assertTrue(restorePath.isDirectory(), "'$path' should be a directory in backup")
+                        assertTrue(individualRestorePath.exists(), "'$path' should exist in backup (individual)")
+                        assertTrue(
+                            individualRestorePath.isDirectory(),
+                            "'$path' should be a directory in backup (individual)",
+                        )
+                    } else {
+                        val originalBytes = path.toFile().readBytes().asList()
+                        assertEquals(
+                            originalBytes,
+                            restorePath.toFile().readBytes().asList(),
+                            "File contents of '$path' should equal",
+                        )
+                        assertEquals(
+                            originalBytes,
+                            individualRestorePath.toFile().readBytes().asList(),
+                            "File contents of '$path' should equal (individual)",
+                        )
+                    }
+                    // cleanup
+                    path.deleteExisting()
+                    restorePath.deleteExisting()
+                    individualRestorePath.deleteExisting()
+                }
+                // cleanup
+                restoreDir.deleteExisting()
+                individualRestoreDir.deleteExisting()
+                runBlocking {
+                    s3.deleteObject {
+                        bucket = bucketName
+                        key = backupKey
+                    }
+                }
+            }
+        }
+}
+
+internal fun File.countEntries(): Int {
+    val queue = ArrayDeque<File>()
+    queue.add(this)
+    return queue.count { file ->
+        if (file.isDirectory) {
+            queue.addAll(file.listFiles()!!) // shouldn't ever be null, since we know it's a directory
+        }
+        true
+    }
+}
Author	SHA1	Message	Date
Gleb Koval	ba050cbc79	Implement multipart tests (#4 ) All checks were successful Test Workflow / Lint and test library (push) Successful in 1m59s Details Currently, tests do not test multipart uploads (since they use very small files). Implement some large (randomly generated) files as well. Reviewed-on: #4	2023-12-31 09:02:36 +00:00
Gleb Koval	ddcbe28eca	Revert "CI broken notice" All checks were successful Test Workflow / Lint and test library (push) Successful in 1m43s Details This reverts commit `53e6d30e53`.	2023-12-31 13:21:27 +06:00
Gleb Koval	53e6d30e53	CI broken notice All checks were successful Test Workflow / Lint and test library (push) Successful in 17m36s Details	2023-12-30 23:29:41 +06:00
Gleb Koval	97f9cc4b2b	Clarify testing only BackupClient in README All checks were successful Test Workflow / Lint and test library (push) Successful in 5m54s Details	2023-12-30 22:56:44 +06:00
Gleb Koval	e09bf9e9f8	Round-trip integration tests (#3 ) All checks were successful Test Workflow / Lint and test library (push) Successful in 1m43s Details Publish Workflow / Publish library (push) Successful in 2m7s Details - Implement round-trip tests for some cases. - Run tests in CI. Reviewed-on: #3	2023-12-30 16:42:51 +00:00
Gleb Koval	694b8d4d28	CLI implementation & linting (#2 ) All checks were successful Test Workflow / Lint and test library (push) Successful in 17m11s Details Publish Workflow / Publish library (push) Successful in 8m49s Details Implement a basic CLI interface & add linting to the project. Reviewed-on: #2	2023-12-30 06:36:49 +00:00
Gleb Koval	36728fa1e9	Setup CI testing and building (#1 ) All checks were successful Test Workflow / Lint and test library (push) Successful in 17m20s Details We want CI to be able to build the ShadowJAR for the CLI tool. Reviewed-on: #1	2023-12-29 15:43:53 +00:00