> ## Documentation Index
> Fetch the complete documentation index at: https://docs.visual-layer.com/llms.txt
> Use this file to discover all available pages before exploring further.

# S3 Configuration

> This tutorial explains key configuration flags used in the Visual Layer platform to allow indexing and serving images from S3.

The following configuration should be added to the config.json file inside the .vl/ folder.

## 🔧 Configuration Flags

```json theme={"theme":"monokai"}
{
  "SERVE_ORIGINAL_IMAGE": true,
  "CDN_FULLPATH": "/image",
  "SAVE_THUMBNAILS_METHOD": "crops",
  "PREPROCESS_IMAGE_TRANSFER": "keep_original_filename",
  "PREPROCESS_NORMALIZE_OUTPUT": false,
  "REMOTE_SAVE": "s3://visual-layer/tmp/remove-me-test-folder",
  "SOURCE_USE_FULLPATH": true,
  "USE_S5CMD_WHEN_AVAILABLE": true,
  "SKIP_IMAGE_THUMBNAIL_GENERATION": true,
  "EXPORT_ENTITIES_RESULT_BUCKET": "mybucket",
  "OBJECT_STORE_URL": "http://minio:9000"
}
```

### SERVE\_ORIGINAL\_IMAGE: true

* **Purpose**: Forces the UX to serve the original image directly from the remote storage (e.g., S3).
* **Use case**: Ideal for high-speed networks or when thumbnails are not needed.
* **Note**: This must be combined with `PREPROCESS_IMAGE_TRANSFER: "keep_original_filename"` to ensure proper file linking.

### PREPROCESS\_IMAGE\_TRANSFER: "keep\_original\_filename"

* **Purpose**: Ensures the original image filenames are preserved during transfer and storage.
* **Impact**: Required when using `SERVE_ORIGINAL_IMAGE` to allow consistent referencing of images in the frontend.

### CDN\_FULLPATH: "/image"

* **Purpose**: Defines the root path where images are served through a CDN or proxy.
* **Required**: Yes – this path is mandatory for the UX to resolve asset URLs correctly.

### SOURCE\_USE\_FULLPATH: true

* **Purpose**: Maintains the original folder structure (relative paths) of source files during preprocessing and storage.
* **Use case**: Helps track image provenance and supports hierarchical datasets.

### REMOTE\_SAVE: "s3://mybucket/visual-layer-intermediate/"

* **Purpose**: Specifies the destination path (S3 or other remote store) for intermediate processed assets.
* **Required**: Yes – used by the UX to access thumbnails, metadata, or originals.

### USE\_S5CMD\_WHEN\_AVAILABLE: true

* **Purpose**: Uses s5cmd for file transfers instead of AWS CLI or boto3.
* **Benefit**: Can achieve up to 10× faster S3 copy operations.
* **Optional**: Yes – recommended for large datasets or high-throughput pipelines.

### SKIP\_IMAGE\_THUMBNAIL\_GENERATION: true

* **Purpose**: Disables thumbnail generation during preprocessing.
* **Use case**: For fast uploads when bandwidth is not a concern and `SERVE_ORIGINAL_IMAGE` is enabled.
* **Optional**: Yes – reduces storage and processing cost in some environments.

### SAVE\_THUMBNAILS\_METHOD: "crops"

* **Purpose**: Defines the method used to save thumbnails. "crops" stores only relevant cropped regions (e.g., detections or ROIs).
* **Effect**: Reduces thumbnail size and focuses on semantically important regions.
* **Optional**: No

### EXPORT\_ENTITIES\_RESULT\_BUCKET: "mybucket"

* **Purpose**: Specifies an optional S3 bucket name where export files will be saved.
* **Format**: Should contain just the bucket name without the `s3://` prefix (e.g., `"mybucket"` instead of `"s3://mybucket/"`).
* **Use case**: When exporting entities or results, files will be stored in this bucket instead of the default location.
* **Optional**: Yes – if not specified, exports will use the default export location.

### OBJECT\_STORE\_URL: "[http://minio:9000](http://minio:9000)"

* **Purpose**: Configures a custom S3-compatible object storage endpoint URL for the system.
* **Use case**: Allows using alternative object storage services (like MinIO or on-premise S3-compatible storage) instead of AWS S3. Enables on-premise or custom object storage deployments.
* **Behavior**: When set, it overrides the default AWS S3 endpoint for all storage operations including downloads and preprocessing artifact storage.
* **Technical details**: When URL starts with `http://`, SSL verification is disabled for local deployments. Used for both data downloads and preprocessing operations.
* **Optional**: Yes – if not specified, the system will use the default AWS S3 endpoint.

## 🔐 AWS Credentials Setup

To enable remote saving and loading of assets from S3, you must provide AWS credentials. Visual Layer looks for credentials in the file:

```
.vl/aws_credentials
```

### 📄 File Format

The file must follow the standard AWS credentials format used by the AWS CLI and boto3:

```ini theme={"theme":"monokai"}
[default]
aws_access_key_id = YOUR_ACCESS_KEY_ID
aws_secret_access_key = YOUR_SECRET_ACCESS_KEY
region = YOUR_AWS_REGION #optional
```

**Example:**

```ini theme={"theme":"monokai"}
[default]
aws_access_key_id = AKIAIOSFODNN7EXAMPLE
aws_secret_access_key = wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY
region = us-west-2
```

### 📁 File Location

Make sure the file is located at:

```
<your_project_root>/.vl/aws_credentials
```

✅ **Tip**: .vl is a hidden folder used by Visual Layer to store configuration and credentials.

### 🔐 Security Best Practices

Use temporary credentials (e.g., via IAM roles or STS) in production if possible.

## 🔒 Custom SSL Certificate (Optional)

If your S3 endpoint requires a custom SSL certificate (e.g., when using a private MinIO or S3-compatible object store behind a proxy), you can set the `AWS_CA_BUNDLE` environment variable.

### 🧪 What it does:

This tells boto3, s5cmd, and other AWS SDK tools to use a specific .pem certificate file when making HTTPS requests.

### ✅ Usage

Copy your certificate .pem file into the local .vl folder. Set the environment variable before launching your Visual Layer job:

```bash theme={"theme":"monokai"}
export AWS_CA_BUNDLE=/.vl/custom-ca.pem
```

## 📂 Folder Structure Summary

```
project-root/
├── .vl/
│   ├── aws_credentials
│   └── aws_ca_bundle.pem   # Optional
├── config.json             # Your Visual Layer configuration
└── run_profiler.sh         # Run a pipeline
```

## 📞 Need Help?

For support, reach out to your Visual Layer technical contact or email: [support@visual-layer.com](mailto:support@visual-layer.com).