Guides

Installation

Suggest Edits

Running the Application

Start the System

To start the application, run the following command:

docker compose -f compose.yaml up -d

Note: Ensure this command completes successfully before proceeding.

Connect to the Application

Once the system is running, connect to the application using your browser:

  • HTTP: http://<server name>:2080
  • HTTPS: https://<server name>:2443

Create a Dataset

Follow the instructions below to create a dataset (see relevant sections for details).

AWS / MinIO Credentials (DataIn)

Enabling Data-In with Docker Compose Profiles

To enable Data-In, a Docker Compose profile must be set. This is automatically configured during installation by adding the following to the .env file:

COMPOSE_PROFILES=cpu
  • Replace cpu with gpu if running on a GPU machine.
  • Required for running enrichment tasks.

MinIO Setup

  • MinIO is included in the docker-compose setup and starts by default (must be pulled first).
  • It is mounted by default to: .vl/minio_storage

Credentials File

The installer sets credentials in .vl/aws_credentials with the following content (modifiable as needed):

[default]
credential_source=Ec2InstanceMetadata

[ingestion]
aws_access_key_id = minioadmin
aws_secret_access_key = minioadmin
endpoint_url = http://minio:9000
  • The default profile is used to connect to object storage (AWS or MinIO) and process data.
  • The ingestion profile is used internally for processing data via the UI.

Configuration Options

The following environment variables control the ingestion pipeline:

PREFECT_PIPELINE_ENABLED=true
S3_VL_INGESTION_SERVICE_STORAGE=local-ingestion-service-storage
S3_VL_INGESTION_SERVICE_PROFILE=ingestion

Operations

Common Operations

Start the System

docker compose -f compose.yaml --profile pipeline up -d

Stop the System

docker compose -f compose.yaml down

Creating Datasets

Without Enrichment

./run_profiler.sh -n "<desired dataset name>" -p "<dataset source dir>" -e compose -w cpu
  • This process can take several hours, depending on data size.
  • If the dataset name includes spaces, use quotation marks (e.g., "My Dataset").

With Enrichment (Objects & Image Descriptions)

  1. Modify the configuration file:
    • Remove (or rename) the PREPROCESS_STEPS parameter in .vl/config.json.
  2. Run the dataset creation command: 
    ./run_profiler.sh -n "desired dataset name" -p "dataset source dir" -e compose -w gpu
    Note: This may take several hours, depending on dataset size.

Upgrading the System

Applying Updates via Command-Line

Step 1: Stop the System

docker compose -f compose.yaml down

Step 2: Back Up the Database

sudo cp -rp .vl/pgdata .vl/pgdata_backup

Step 3: Download and Copy Migration Content

docker create --name vl_installer visuallayer/onprem-installer:<VL_VERSION>
docker cp vl_installer:/app/migration.sql .
docker cp vl_installer:/app/config.json .vl/default.config.json
if [ ! -f .vl/default_aws_credentials ]; then docker cp vl_installer:/app/default_aws_credentials .vl/; fi
mkdir -p .vl/minio_storage/local-ingestion-service-storage

Step 4: Update Version in .env File

Modify the .env file and set the VL_VERSION to the target version:

VL_VERSION=2.27.0

Step 5: Migrate PostgreSQL Database

Replace <password> with the actual PostgreSQL password (found in compose.yaml):

docker compose up -d pg
psql postgresql://postgres:<password>@localhost:5666/postgres < migration.sql

Step 6: Restart the System

docker compose -f compose.yaml --profile pipeline up -d

Step 7: Validate the Upgrade

To confirm the upgrade was successful:

  • Run docker ps and verify 5 images are in the "UP" state (prefect-server, fe, be, cd, pg).
  • Connect to the application at http://<server name>:2080 and check for errors.

Troubleshooting

Resolving Common Errors

Problem: Image Not Found

docker: Error response from daemon: pull access denied for visuallayer/onprem-installer, repository does not exist or may require 'docker login'.

Solution: Ensure you are logged into Docker.

Problem: Docker Daemon Not Running

docker: Cannot connect to the Docker daemon at unix:///var/run/docker.sock.

Solution: Start the Docker daemon.

Problem: Image Pulling Fails

manifest for visuallayer/onprem-cd:2.25.6-rc6 not found: manifest unknown

Solution: Verify the software version and that you are logged into Docker with the correct user.

Problem: Platform Mismatch Warning

WARNING: The requested image's platform (linux/arm64) does not match the detected host platform (linux/amd64/v3).

Solution: This is a warning only—can be ignored.

Problem: docker compose Not Found

./run_profiler.sh: line 159: docker compose: command not found

Solution: Install or reinstall Docker following the official installation instructions.

Problem: PostgreSQL Operational Error

sqlalchemy.exc.OperationalError: (psycopg.OperationalError) [Errno -3] Temporary failure in name resolution

Solution: Ensure the system was brought up using the docker up command.

Accessing Logs & Support

Debugging Run

Use the following environment variables to enable debugging:

LOG_LEVEL=10  # Set log level to debug
REPORTING_LOG_PATH="/path/to/log.txt"  # Write pipeline log to file

Updated 2 days ago