Skip to main content

What This Helps With

Automate custom metadata uploads from CSV files to Visual Layer datasets with comprehensive validation, multiple field type support, and detailed error reporting.
This page provides a production-ready Python script that demonstrates how to upload custom metadata from CSV files to Visual Layer. The script validates data before upload, supports all six field types, and provides clear error messages for troubleshooting.
This script demonstrates the complete Visual Layer custom metadata API workflow. You can modify the script to suit your specific needs and data formats.

Working with DICOM Medical Imaging Data?

For medical imaging workflows, check out our specialized DICOM converter and upload scripts that handle DICOM-specific field types, date/time formats, and medical metadata standards.

View Complete Script Code

Access the full custom metadata upload Python script with complete implementation, ready to copy and use.

​
Features

The script provides a comprehensive solution for CSV-based metadata uploads:
  • Multiple Field Types - Supports all six Visual Layer field types: string, float, datetime, enum, multi-enum, and link
  • Comprehensive Validation - Validates all data before upload with detailed error messages showing row and column information
  • Enum Limit Checking - Pre-upload validation ensures enum fields don’t exceed the 600 unique values limit
  • Flexible Datetime Formats - Accepts multiple datetime formats and auto-converts to ISO 8601 UTC
  • Clear Progress Tracking - Detailed status reporting throughout the upload workflow
  • Auto-Detection - Automatically discovers enum options from your CSV data

​
How the Script Works

The script follows the complete Visual Layer custom metadata API workflow:
  1. Export dataset - Calls Visual Layer API to get filename-to-media_id mapping
  2. Read and validate CSV - Validates all values match field type requirements before upload
  3. Check enum constraints - Verifies enum fields don’t exceed 600 unique values limit
  4. Create custom metadata fields - Creates each field in Visual Layer via API calls
  5. Upload metadata values - Uploads validated data for each field using proper JSON format
  6. Monitor upload progress - Tracks completion status and reports results

​
Prerequisites

Before using the script, ensure you have:
  1. Visual Layer installation (cloud or on-premises)
    • Cloud: Requires API key and secret (see Authentication Guide)
    • On-Premises: No authentication needed
  2. Dataset in READY state in Visual Layer with images already uploaded
  3. Python environment with required packages
  4. CSV file formatted according to requirements below

​
Install Dependencies

pip install requests pandas pyjwt
PyJWT is required for cloud installations that use JWT authentication. On-premises installations without authentication can omit this package.

​
CSV Requirements

The CSV file must follow this structure to work with the script.

​
Required Column

Your CSV must include a file_fullpath column that matches the format from Visual Layer’s export: 
file_fullpath,patient_id,score,created_at,status,tags,pdf_url
images/img1.jpg,P001,95.5,2024-01-15,active,"tag1, tag2",https://example.com/doc.pdf
images/img2.jpg,P002,87.2,2024-01-16,pending,"tag3, tag4",https://example.com/doc2.pdf
The file_fullpath values are used to map your metadata to Visual Layer media_ids.

​
Optional Metadata Columns

You can include any number of metadata columns. Column names become field names in Visual Layer. Specify the type for each column using command-line arguments.

​
How to Get file_fullpath Values

To get the correct file_fullpath values for your dataset:
  1. Export your dataset from Visual Layer using the media_id export endpoint
  2. The export returns a CSV with filename and media_id columns
  3. Use the filename values as your file_fullpath values in your metadata CSV

​
Supported Field Types

The script supports all six Visual Layer custom metadata field types.

​
String (--string-fields)

Any text value. 
file_fullpath,patient_id,notes
images/img1.jpg,P001,High quality sample
Example: patient_id, notes, description

​
Float (--float-fields)

Decimal numbers. 
file_fullpath,score,confidence
images/img1.jpg,95.5,0.98
Format: 95.5, 0.98, 123.456 Example: score, confidence, temperature

​
Datetime (--datetime-fields)

Date and time values in UTC. The script accepts multiple formats and converts to ISO 8601 UTC: Accepted formats:
  • 2024-01-15T10:30:00Z (ISO 8601 UTC - preferred)
  • 2024-01-15 (date only - converts to midnight UTC)
  • 2024-01-15 10:30:00 (auto-converts to UTC)
Uploaded as: YYYY-MM-DDTHH:MM:SSZ 
file_fullpath,created_at,scan_date
images/img1.jpg,2024-01-15T10:30:00Z,2024-01-15
Example: created_at, updated_at, scan_date

​
Enum (--enum-fields)

Single-select categorical data with a maximum of 600 unique values per field. 
file_fullpath,status,category
images/img1.jpg,active,approved
images/img2.jpg,pending,review
The script automatically collects unique values from your CSV and validates the count before upload. Example: status, category, priority

​
Multi-Enum (--multi-enum-fields)

Multi-select categorical data with a maximum of 600 unique values total across all lists. Accepts two formats:
  • Comma-separated: "tag1, tag2, tag3"
  • JSON array: ["tag1", "tag2", "tag3"]
file_fullpath,tags
images/img1.jpg,"tag1, tag2, tag3"
images/img2.jpg,"[""tag4"", ""tag5""]"
The script validates the total unique count before upload. Example: tags, labels, keywords URL values that must include a protocol. 
file_fullpath,pdf_url,documentation_link
images/img1.jpg,https://example.com/doc.pdf,https://docs.example.com
Must include protocol: http:// or https:// Example: pdf_url, documentation_link, image_url

​
Usage

Save the script as custom_metadata_upload_script.py and run with field type specifications.

​
Basic Example

# Cloud installation (requires authentication)
python custom_metadata_upload_script.py \
  --csv metadata.csv \
  --dataset-id YOUR-DATASET-ID \
  --base-url https://app.visual-layer.com \
  --api-key YOUR-API-KEY \
  --api-secret YOUR-API-SECRET \
  --string-fields patient_id notes \
  --float-fields score confidence \
  --datetime-fields created_at updated_at \
  --enum-fields status category \
  --multi-enum-fields tags labels \
  --link-fields pdf_url

# On-premises installation (no authentication needed)
python custom_metadata_upload_script.py \
  --csv metadata.csv \
  --dataset-id YOUR-DATASET-ID \
  --base-url http://localhost:8080 \
  --string-fields patient_id notes \
  --float-fields score confidence \
  --datetime-fields created_at updated_at \
  --enum-fields status category \
  --multi-enum-fields tags labels \
  --link-fields pdf_url

​
Command-Line Parameters

ParameterRequiredDescriptionDefault
--csv✅Path to CSV file with metadata-
--dataset-id✅Visual Layer dataset identifier-
--base-url⭕Visual Layer installation URL (use https://app.visual-layer.com for cloud)http://localhost:8080
--api-key⭕API key for Visual Layer Cloud authentication-
--api-secret⭕API secret for Visual Layer Cloud authentication-
--string-fields⭕Space-separated list of string field names-
--float-fields⭕Space-separated list of float field names-
--datetime-fields⭕Space-separated list of datetime field names-
--enum-fields⭕Space-separated list of enum field names-
--multi-enum-fields⭕Space-separated list of multi-enum field names-
--link-fields⭕Space-separated list of link field names-
--batch-size⭕Records per upload batch50000
Cloud vs On-Premises:
  • Cloud (https://app.visual-layer.com): Requires --api-key and --api-secret for JWT authentication
  • On-Premises: No authentication needed, omit --api-key and --api-secret parameters
At least one field type must be specified. The script will return an error if no field types are provided.

​
Example Use Cases

The following examples demonstrate common metadata upload scenarios.

​
Medical Imaging

# Cloud example
python custom_metadata_upload_script.py \
  --csv medical_metadata.csv \
  --dataset-id abc-123 \
  --base-url https://app.visual-layer.com \
  --api-key YOUR-API-KEY \
  --api-secret YOUR-API-SECRET \
  --string-fields patient_id doctor_notes diagnosis \
  --float-fields confidence_score tumor_size_mm \
  --datetime-fields scan_date report_date \
  --enum-fields scan_type organ priority \
  --multi-enum-fields findings abnormalities \
  --link-fields dicom_url report_pdf
For on-premises installations, omit --api-key and --api-secret and change --base-url to your local URL (e.g., http://localhost:8080).
CSV format: 
file_fullpath,patient_id,scan_date,scan_type,confidence_score,findings,report_pdf
scans/chest/scan001.dcm,P12345,2024-01-15,CT,0.95,"nodule, calcification",https://reports.com/P12345.pdf

​
E-commerce Products

# Cloud example
python custom_metadata_upload_script.py \
  --csv product_metadata.csv \
  --dataset-id xyz-789 \
  --base-url https://app.visual-layer.com \
  --api-key YOUR-API-KEY \
  --api-secret YOUR-API-SECRET \
  --string-fields product_id product_name sku \
  --float-fields price rating \
  --datetime-fields created_at last_updated \
  --enum-fields category status availability \
  --multi-enum-fields tags colors sizes \
  --link-fields product_url manual_pdf

​
Document Management

# Cloud example
python custom_metadata_upload_script.py \
  --csv document_metadata.csv \
  --dataset-id doc-456 \
  --base-url https://app.visual-layer.com \
  --api-key YOUR-API-KEY \
  --api-secret YOUR-API-SECRET \
  --string-fields document_id title author \
  --datetime-fields created_at modified_at \
  --enum-fields document_type status department \
  --multi-enum-fields keywords topics \
  --link-fields pdf_url source_url

​
Authentication (Cloud Only)

Visual Layer Cloud requires JWT authentication for all API requests. On-premises installations can skip this section.

On-Premises Installations

If you’re using an on-premises deployment, you can skip authentication parameters. The script works without --api-key and --api-secret arguments.

​
Get Your API Credentials

To use the script with Visual Layer Cloud:
  1. Obtain API Key and Secret - Follow the instructions in the Authentication Guide to retrieve your API credentials from Visual Layer Cloud
  2. Pass credentials to script - Use the --api-key and --api-secret parameters when running the script
The script automatically generates JWT tokens and includes them in API requests when credentials are provided.
JWT tokens are short-lived (10 minutes). The script generates a fresh token at the start of execution, which is sufficient for most uploads. For very large datasets that take longer than 10 minutes, the upload may fail with authentication errors. Contact Visual Layer support for guidance on long-running uploads.

​
Authentication Example

# Cloud installation with authentication
python custom_metadata_upload_script.py \
  --csv metadata.csv \
  --dataset-id YOUR-DATASET-ID \
  --base-url https://app.visual-layer.com \
  --api-key YOUR-API-KEY \
  --api-secret YOUR-API-SECRET \
  --string-fields patient_id notes \
  --float-fields score confidence
For complete details on obtaining and testing JWT tokens, see the Authentication Guide.

​
Validation and Error Handling

The script validates all data before uploading to catch issues early.

​
Pre-Upload Validation Checks

The script performs these validations before any upload:
  • Float values are valid numbers
  • Datetime values match accepted formats
  • Enum fields don’t exceed 600 unique values
  • Multi-enum fields don’t exceed 600 unique values total
  • Multi-enum values are properly formatted (comma-separated or JSON array)
  • Links include protocol (http:// or https://)
  • File paths exist in Visual Layer dataset

​
Error Messages

The script provides clear error messages showing exactly what’s wrong: 
❌ Validation Errors (3 total):
   • Row 5, field 'score': Invalid float value 'abc'. Must be a valid number.
   • Row 8, field 'created_at': Invalid datetime format '01-15-2024'. Expected ISO 8601 format: YYYY-MM-DDTHH:MM:SSZ or YYYY-MM-DD

❌ Enum Constraint Errors (1 total):
   • Field 'status' (enum): Has 750 unique values, exceeds limit of 600
Each error includes the row number and field name for easy troubleshooting in your CSV.

​
Script Output

The script provides detailed progress information throughout the workflow.

​
Successful Upload Example

======================================================================
🚀 Generic CSV Metadata Upload to Visual Layer
======================================================================
📁 CSV File: metadata.csv
🎯 Dataset ID: abc-123
🌐 Base URL: http://localhost:8080
======================================================================

📤 Exporting dataset to get media_id mappings...
   ✅ Exported 150 media items

📖 Reading and validating CSV file: metadata.csv
   ✅ Loaded 150 rows
   🔍 Checking enum field constraints...
      ✅ status (enum): 3 unique values
      ✅ tags (multi-enum): 12 unique values
   🔍 Validating 8 metadata fields...

   📋 Field Configuration:
      • annotated_at: datetime
      • category: enum
      • confidence: float
      • notes: string
      • patient_id: string
      • pdf_url: link
      • status: enum
      • tags: multi-enum

   ✅ Validated 150 rows successfully

🔧 Creating and uploading 8 custom fields...

📝 Processing: annotated_at (datetime)
   🔧 Creating field: annotated_at (datetime)
      ✅ Created (task_id: task-123)
      📤 Uploading 150 values
      ✅ Upload completed
      ✅ Completed (150 rows)

📝 Processing: confidence (float)
   🔧 Creating field: confidence (float)
      ✅ Created (task_id: task-124)
      📤 Uploading 150 values
      ✅ Upload completed
      ✅ Completed (150 rows)

...

======================================================================
🎉 Workflow Completed!
======================================================================
✅ Successfully uploaded: 8/8 fields
📊 Total rows processed: 150
======================================================================

​
Tips and Best Practices

Follow these best practices for successful metadata uploads.

​
Test with Sample Data First

Start with a small CSV (10-20 rows) to verify:
  • Field types are correct
  • Data formats are valid
  • Mappings work as expected

​
Use Datetime Flexibility

The script accepts multiple datetime formats: 
created_at
2024-01-15                    # ✅ Converts to 2024-01-15T00:00:00Z
2024-01-15T10:30:00Z         # ✅ Already in correct format
2024-01-15 10:30:00          # ✅ Converts to UTC

​
Multi-Enum Format Options

Both formats work equally well: 
tags
"tag1, tag2, tag3"           # ✅ Comma-separated
["tag1", "tag2", "tag3"]     # ✅ JSON array

​
Handle Empty Values

Empty values are automatically skipped. There is no need to remove rows with missing data.

​
Use Descriptive Field Names

Column names become field names in Visual Layer. Make them descriptive and consistent:
  • ✅ Good: patient_id, scan_date, confidence_score
  • ❌ Avoid: field1, data, x

​
Troubleshooting

Common issues and solutions for metadata upload problems.

​
”No field types specified”

Problem: Script returns error about missing field types. Solution: Add at least one --*-fields argument to specify which columns to upload and their types.

​
”Failed to export dataset”

Problem: Script cannot export dataset to get media_id mappings. Solution:
  • Check that the dataset ID is correct
  • Verify the base URL is accessible
  • Ensure Visual Layer is running
  • Confirm the dataset is in READY state

​
”Validation failed with N errors”

Problem: CSV data doesn’t match specified field types. Solution:
  • Read error messages carefully (shows row and column)
  • Fix data format issues in CSV
  • Verify field type assignments match your data
  • Check that datetime values use accepted formats

​
”Enum field exceeds 600 unique values limit”

Problem: Enum or multi-enum field has too many unique values. Solution:
  • Use string field type instead of enum or multi-enum
  • Reduce the number of unique values in your data
  • Check unique count with: cut -d',' -f2 file.csv | sort | uniq | wc -l

​
”Field already exists”

Problem: Script reports that a field was created previously. Solution: This is normal behavior, not an error. The script will skip creation and proceed with upload. This occurs when re-running the script or adding data to existing fields.
Adding vs. Updating Metadata:
  • Adding new fields: The script will create new fields if they don’t exist
  • Updating existing fields: Currently updating existing metadata values is not supported. Contact Visual Layer for assistance with updating existing metadata values

​
Getting Help

For advanced use cases, custom metadata deletion, or technical support:

Contact Visual Layer Support

Reach out to Visual Layer’s support team for assistance with complex metadata workflows, custom field requirements, or troubleshooting upload issues.

​
Expected Result in Visual Layer

After running the script successfully, your custom metadata fields will be visible in Visual Layer’s interface for each image.
Custom metadata displayed in Visual Layer showing fields like name, float value, list of tags, condition, and date
These fields become searchable and filterable, enabling you to query and analyze your dataset based on your custom metadata.