> ## Documentation Index
> Fetch the complete documentation index at: https://docs.visual-layer.com/llms.txt
> Use this file to discover all available pages before exploring further.

# The VL Profiler Script: run_profiler.sh

> Enable processing and profiling of image datasets for the Visual Layer platform.

The `run_profiler.sh` script enables processing and profiling of image datasets for the Visual Layer platform. It supports multiple data sources, including local directories, S3 buckets, HTTP/HTTPS URLs, and file lists in `.txt`, `.csv`, or `.parquet` formats. With flexible execution modes and broad data compatibility, the script adapts to a wide range of environments and workflows.

The rest of this article outlines script usage syntax and provides detailed explanations for each command-line parameter.

## Usage

Following is the command-line syntax for running the script:

```bash theme={"theme":"monokai"}
./run_profiler.sh [-h] [-p <path> -n <dataset_name>] [-e compose|local] [-r]
```

## Parameters

Following are the command-line parameters supported by the script:

***

### Dataset Path (`-p`, *required*)

Specifies the source location of your dataset. Multiple formats are supported:

<ParamField path="-p" type="string" required>
  | | |
  | - | - |
  | Local directory | `~/data/images` |
  | S3 bucket | `s3://mybucket/images` |
  | HTTP/HTTPS URL | `https://example.com/dataset` |
  | File list | Path to a `.txt`, `.csv`, or `.parquet` file containing image paths |
</ParamField>

***

### Dataset Name (`-n`, *required*)

A human-readable name for your dataset. This name will be used for identification in logs and results.

<ParamField path="-n" type="string" required />

***

### Execution Mode (`-e`)

Determines how the processing pipeline is executed.

**default**: `compose`

<ParamField path="-e" default="compose" type="string">
  | | |
  | - | - |
  | `compose` | Uses Docker Compose and the Visual Layer API (recommended for most users, default mode) |
  | `local` | Runs directly on your local machine using a Python virtual environment (advanced) |
</ParamField>

***

### Reduce Disk Space (`-r`)

Enables reduced disk space consumption mode. Useful for large datasets or environments with limited storage.

**default**: `false`

When enabled, activates `serve_mode=reduce_disk_space` for optimized storage usage.

<ParamField path="-r" default="false" type="boolean" />

***

### Help (`-h`)

Displays usage information and example commands.

<ParamField path="-h" type="boolean" />

## Examples

<CodeGroup>
  ```bash Local Directory theme={"theme":"monokai"}
  ./run_profiler.sh -p ~/data/ds1 -n 'dataset1'
  ```

  ```bash S3 Directory theme={"theme":"monokai"}
  ./run_profiler.sh -p s3://mybucket/images -n 'dataset1'
  ```

  ```bash File List (Local) theme={"theme":"monokai"}
  ./run_profiler.sh -p ~/filelist.txt -n 'dataset1'
  ```

  ```bash File List (S3) theme={"theme":"monokai"}
  ./run_profiler.sh -p s3://mybucket/images/filelist.parquet -n 'dataset1'
  ```

  ```bash Reduced Disk Space theme={"theme":"monokai"}
  ./run_profiler.sh -p ~/data/ds1 -n 'dataset1' -r
  ```
</CodeGroup>

## Execution Modes

The script supports two execution modes for processing datasets:

### Compose Mode (Default)

<Info>
  **Recommended for most users**. Uses Docker Compose to run the processing pipeline with the Visual Layer API.
</Info>

| Feature | Description |
| - | - |
| Processing Method | Processes datasets through HTTP API endpoint |
| Supported Data Sources | Local, S3, HTTP/HTTPS |
| Path Handling | Automatically handles path encoding for remote sources |
| Integration | Integrates with the full Visual Layer service stack |

**API Endpoint**: `POST http://localhost:2080/api/v1/process`

**Parameters sent to API**:

| Parameter | Description |
| - | - |
| `path` | Dataset source path (URL-encoded for remote sources) |
| `name` | Dataset name |
| `serve_mode` | Set to `reduce_disk_space` when `-r` flag is used |

### Local Mode

<Warning>
  **Advanced users only**. Runs the pipeline directly using Python virtual environment.
</Warning>

* Executes `pipeline.controller.controller` module directly
* Requires local Python virtual environment setup
* Uses manual flow configuration (`MANUAL_FLOW=yes`)
* Configures device settings based on hardware type (CPU/GPU)

**Environment Variables Set**:

* `MANUAL_FLOW=yes`
* `FASTDUP_PRODUCTION=1`
* `PREFECT_LOGGING_SETTINGS_PATH=./.vl/prefect-logging.yaml`
* Device-specific settings for CPU mode

## Data Source Support

### Local Paths

* Converts relative paths to absolute paths using `realpath`
* For file inputs (lists), copies to `.vl/` directory for container access
* Validates path existence before processing

### Remote Paths

| Source Type | Example/Details |
| - | - |
| **S3** | `s3://bucket/path` |
| **HTTP/HTTPS** | `http://` or `https://` URLs |
| **File Detection** | Automatically detects file extensions (`.txt`, `.csv`, `.parquet`) |
| **URL Encoding** | Applies proper encoding for API transmission |

### Supported File Formats

* **Image Directories**: Any directory containing image files
* **File Lists**:
  * `.txt`: Plain text file with one file path per line
  * `.csv`: CSV file with file paths
  * `.parquet`: Parquet file containing file path data

## Error Handling

The script includes comprehensive error handling:

| Error Type | Description |
| - | - |
| **Missing Arguments** | Displays usage information and exits |
| **Invalid Paths** | Validates local path existence |
| **Invalid Execution Mode** | Ensures mode is either `compose` or `local` |
| **API Failures** | Captures and displays API error responses |
| **Pipeline Failures** | Handles local pipeline execution errors |

## Dependencies

| Dependency | Purpose/Usage |
| - | - |
| **Bash** | Shell environment |
| **curl** | API communication (compose mode) |
| **Python 3** | Local execution and URL encoding |
| **Docker Compose** | Compose mode execution |
| **Virtual Environment** (`./venv_local/`) | Local mode Python environment |

## Output

### Compose Mode

* Success: Displays dataset processing confirmation with response
* Failure: Shows API error message in red text

### Local Mode

* Runs pipeline with full logging output
* Returns to original shell environment on completion

## Performance Considerations

### Reduced Disk Space Mode (`-r`)

* Activates `serve_mode=reduce_disk_space` parameter
* Optimizes storage usage during processing
* Recommended for large datasets or limited storage environments

### Hardware Configuration

* **CPU Mode**: Automatically configures all processing devices to use CPU
* **GPU Mode**: Uses default GPU acceleration when available

## Integration

This script integrates with the core components of the **Visual Layer** platform:

| Component | Description |
| - | - |
| Pipeline Controller | Orchestrates the complete processing workflow |
| Database | Stores processed dataset metadata and results |
| API Service | Provides RESTful interface for dataset operations |
| Storage Systems | Supports local filesystem, S3, and HTTP sources |

## Troubleshooting

<Accordion title="Common Issues">
  **Path not found errors**

  * Verify local paths exist and are accessible
  * Check S3 credentials and permissions for S3 paths
  * Ensure HTTP/HTTPS URLs are accessible
</Accordion>


This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.