The following configuration should be added to the config.json file inside the .vl/ folder.

πŸ”§ Configuration Flags

{
  "SERVE_ORIGINAL_IMAGE": true,
  "CDN_FULLPATH": "/image",
  "SAVE_THUMBNAILS_METHOD": "crops",
  "PREPROCESS_IMAGE_TRANSFER": "keep_original_filename",
  "PREPROCESS_NORMALIZE_OUTPUT": false,
  "REMOTE_SAVE": "s3://visual-layer/tmp/remove-me-test-folder",
  "SOURCE_USE_FULLPATH": true,
  "USE_S5CMD_WHEN_AVAILABLE": true,
  "SKIP_IMAGE_THUMBNAIL_GENERATION": true,
  "EXPORT_ENTITIES_RESULT_BUCKET": "mybucket",
  "OBJECT_STORE_URL": "http://minio:9000"
}

SERVE_ORIGINAL_IMAGE: true

  • Purpose: Forces the UX to serve the original image directly from the remote storage (e.g., S3).
  • Use case: Ideal for high-speed networks or when thumbnails are not needed.
  • Note: This must be combined with PREPROCESS_IMAGE_TRANSFER: "keep_original_filename" to ensure proper file linking.

PREPROCESS_IMAGE_TRANSFER: β€œkeep_original_filename”

  • Purpose: Ensures the original image filenames are preserved during transfer and storage.
  • Impact: Required when using SERVE_ORIGINAL_IMAGE to allow consistent referencing of images in the frontend.

CDN_FULLPATH: β€œ/image”

  • Purpose: Defines the root path where images are served through a CDN or proxy.
  • Required: Yes – this path is mandatory for the UX to resolve asset URLs correctly.

SOURCE_USE_FULLPATH: true

  • Purpose: Maintains the original folder structure (relative paths) of source files during preprocessing and storage.
  • Use case: Helps track image provenance and supports hierarchical datasets.

REMOTE_SAVE: β€œs3://mybucket/visual-layer-intermediate/”

  • Purpose: Specifies the destination path (S3 or other remote store) for intermediate processed assets.
  • Required: Yes – used by the UX to access thumbnails, metadata, or originals.

USE_S5CMD_WHEN_AVAILABLE: true

  • Purpose: Uses s5cmd for file transfers instead of AWS CLI or boto3.
  • Benefit: Can achieve up to 10Γ— faster S3 copy operations.
  • Optional: Yes – recommended for large datasets or high-throughput pipelines.

SKIP_IMAGE_THUMBNAIL_GENERATION: true

  • Purpose: Disables thumbnail generation during preprocessing.
  • Use case: For fast uploads when bandwidth is not a concern and SERVE_ORIGINAL_IMAGE is enabled.
  • Optional: Yes – reduces storage and processing cost in some environments.

SAVE_THUMBNAILS_METHOD: β€œcrops”

  • Purpose: Defines the method used to save thumbnails. β€œcrops” stores only relevant cropped regions (e.g., detections or ROIs).
  • Effect: Reduces thumbnail size and focuses on semantically important regions.
  • Optional: No

EXPORT_ENTITIES_RESULT_BUCKET: β€œmybucket”

  • Purpose: Specifies an optional S3 bucket name where export files will be saved.
  • Format: Should contain just the bucket name without the s3:// prefix (e.g., "mybucket" instead of "s3://mybucket/").
  • Use case: When exporting entities or results, files will be stored in this bucket instead of the default location.
  • Optional: Yes – if not specified, exports will use the default export location.

OBJECT_STORE_URL: β€œhttp://minio:9000”

  • Purpose: Configures a custom S3-compatible object storage endpoint URL for the system.
  • Use case: Allows using alternative object storage services (like MinIO or on-premise S3-compatible storage) instead of AWS S3. Enables on-premise or custom object storage deployments.
  • Behavior: When set, it overrides the default AWS S3 endpoint for all storage operations including downloads and preprocessing artifact storage.
  • Technical details: When URL starts with http://, SSL verification is disabled for local deployments. Used for both data downloads and preprocessing operations.
  • Optional: Yes – if not specified, the system will use the default AWS S3 endpoint.

πŸ” AWS Credentials Setup

To enable remote saving and loading of assets from S3, you must provide AWS credentials. Visual Layer looks for credentials in the file:
.vl/aws_credentials

πŸ“„ File Format

The file must follow the standard AWS credentials format used by the AWS CLI and boto3:
[default]
aws_access_key_id = YOUR_ACCESS_KEY_ID
aws_secret_access_key = YOUR_SECRET_ACCESS_KEY
region = YOUR_AWS_REGION #optional
Example:
[default]
aws_access_key_id = AKIAIOSFODNN7EXAMPLE
aws_secret_access_key = wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY
region = us-west-2

πŸ“ File Location

Make sure the file is located at:
<your_project_root>/.vl/aws_credentials
βœ… Tip: .vl is a hidden folder used by Visual Layer to store configuration and credentials.

πŸ” Security Best Practices

Use temporary credentials (e.g., via IAM roles or STS) in production if possible.

πŸ”’ Custom SSL Certificate (Optional)

If your S3 endpoint requires a custom SSL certificate (e.g., when using a private MinIO or S3-compatible object store behind a proxy), you can set the AWS_CA_BUNDLE environment variable.

πŸ§ͺ What it does:

This tells boto3, s5cmd, and other AWS SDK tools to use a specific .pem certificate file when making HTTPS requests.

βœ… Usage

Copy your certificate .pem file into the local .vl folder. Set the environment variable before launching your Visual Layer job:
export AWS_CA_BUNDLE=/.vl/custom-ca.pem

πŸ“‚ Folder Structure Summary

project-root/
β”œβ”€β”€ .vl/
β”‚   β”œβ”€β”€ aws_credentials
β”‚   └── aws_ca_bundle.pem   # Optional
β”œβ”€β”€ config.json             # Your Visual Layer configuration
└── run_profiler.sh         # Run a pipeline

πŸ“ž Need Help?

For support, reach out to your Visual Layer technical contact or email: support@visual-layer.com.