> ## Documentation Index
> Fetch the complete documentation index at: https://docs.visual-layer.com/llms.txt
> Use this file to discover all available pages before exploring further.
# Google Cloud Storage Integration
> Access GCP buckets from on-premise machines using command-line tools or filesystem mounting.
This guide explains how to access Google Cloud Storage (GCS) buckets from on-premise machines using filesystem mounting or command-line access.
Use **Method 1 (Filesystem Mounting)** for seamless integration with Visual Layer. This approach allows you to work with your data as if it were stored locally.
## Prerequisites
Before you begin, ensure you have:
* Linux-based system (Ubuntu/Debian recommended).
* Internet connectivity.
* Google Cloud account with access to the target bucket.
* Appropriate permissions on the GCS bucket.
## Method 1: Filesystem Mounting with GCS FUSE
This method provides direct access to your GCS bucket without copying files locally. GCS FUSE mounts a bucket as a local filesystem, making it appear as a regular folder on your machine.
### Install Google Cloud SDK
Update your system and install required dependencies:
```bash theme={"theme":"monokai"}
sudo apt-get update && sudo apt-get install -y curl unzip python3
```
Download and install the Google Cloud CLI:
```bash theme={"theme":"monokai"}
curl -O https://dl.google.com/dl/cloudsdk/channels/rapid/downloads/google-cloud-cli-465.0.0-linux-x86_64.tar.gz
tar -xf google-cloud-cli-465.0.0-linux-x86_64.tar.gz
./google-cloud-sdk/install.sh
```
Restart your shell to apply the changes:
```bash theme={"theme":"monokai"}
exec -l $SHELL
```
### Authenticate with Google Cloud
Choose one of the following authentication methods based on your use case.
#### User Authentication
Authenticate using your Google account:
```bash theme={"theme":"monokai"}
gcloud auth login user@example.com
```
This command opens a browser window (or provides a link to copy) where you sign in with your Google account and generate an authentication code to paste back into the terminal.
#### Service Account Authentication
For automated processes or production environments, use a service account.
Create and download a service account key file from the Google Cloud Console, then set the environment variable:
```bash theme={"theme":"monokai"}
export GOOGLE_APPLICATION_CREDENTIALS="/path/to/your/service-account-key.json"
```
Authenticate using the service account:
```bash theme={"theme":"monokai"}
gcloud auth activate-service-account --key-file="/path/to/your/service-account-key.json"
```
### Install GCS FUSE
Add the Google Cloud packages repository:
```bash theme={"theme":"monokai"}
curl -fsSL https://packages.cloud.google.com/apt/doc/apt-key.gpg | gpg --dearmor | sudo tee /usr/share/keyrings/cloud.google.gpg > /dev/null
echo "deb [signed-by=/usr/share/keyrings/cloud.google.gpg] http://packages.cloud.google.com/apt gcsfuse-jammy main" | sudo tee /etc/apt/sources.list.d/gcsfuse.list
```
Update package list and install GCS FUSE:
```bash theme={"theme":"monokai"}
sudo apt-get update
sudo apt-get install -y gcsfuse
```
### Create Mount Point
Create a directory where you want to mount the bucket:
```bash theme={"theme":"monokai"}
mkdir -p ~/gcs_mount
```
### Authenticate for GCS FUSE
Choose the authentication method that matches what you used when authenticating with Google Cloud.
**For User Authentication:**
```bash theme={"theme":"monokai"}
gcloud auth application-default login
```
**For Service Account Authentication:**
If you're using a service account, the `GOOGLE_APPLICATION_CREDENTIALS` environment variable will be automatically used by GCS FUSE.
### Mount the Bucket
Mount your GCS bucket to the local filesystem:
```bash theme={"theme":"monokai"}
gcsfuse your-bucket-name ~/gcs_mount
```
**Example:**
```bash theme={"theme":"monokai"}
gcsfuse my-example-bucket ~/gcs_mount
```
### Verify the Mount
Verify that the bucket is successfully mounted:
```bash theme={"theme":"monokai"}
ls ~/gcs_mount
```
You should see the contents of your bucket.
## Method 2: Command-Line Access
Use this method when you need to copy specific files locally before running Visual Layer profiler, or for one-time data transfers.
### Install Google Cloud SDK
If you haven't already installed it for Method 1, update your system and install required dependencies:
```bash theme={"theme":"monokai"}
sudo apt-get update && sudo apt-get install -y curl unzip python3
```
Download and install the Google Cloud CLI:
```bash theme={"theme":"monokai"}
curl -O https://dl.google.com/dl/cloudsdk/channels/rapid/downloads/google-cloud-cli-465.0.0-linux-x86_64.tar.gz
tar -xf google-cloud-cli-465.0.0-linux-x86_64.tar.gz
./google-cloud-sdk/install.sh
```
Restart your shell to apply the changes:
```bash theme={"theme":"monokai"}
exec -l $SHELL
```
### Authenticate with Google Cloud
Choose one of the following authentication methods based on your use case.
#### User Authentication
Authenticate using your Google account:
```bash theme={"theme":"monokai"}
gcloud auth login user@example.com
```
#### Service Account Authentication
For automated processes or production environments, use a service account.
Create and download a service account key file from the Google Cloud Console, then set the environment variable:
```bash theme={"theme":"monokai"}
export GOOGLE_APPLICATION_CREDENTIALS="/path/to/your/service-account-key.json"
```
Authenticate using the service account:
```bash theme={"theme":"monokai"}
gcloud auth activate-service-account --key-file="/path/to/your/service-account-key.json"
```
### Access Bucket Contents
Once authenticated, you can list and access bucket contents using `gsutil`:
```bash theme={"theme":"monokai"}
gsutil ls gs://your-bucket-name/
```
**Example operations:**
```bash theme={"theme":"monokai"}
# List all files in a bucket
gsutil ls gs://my-example-bucket/
# Copy files from bucket to local machine
gsutil cp gs://my-example-bucket/data.zip ./
# Copy files from local machine to bucket
gsutil cp ./local-file.txt gs://my-example-bucket/
# Sync directories
gsutil rsync -r ./local-folder gs://my-example-bucket/remote-folder
```
## Integration with Visual Layer
After setting up access to your GCS bucket, you can run Visual Layer's profiler using either method.
### Using Mounted Bucket
With the bucket mounted as a local filesystem, run the profiler directly on the mounted directory:
```bash theme={"theme":"monokai"}
# Ensure bucket is mounted
gcsfuse my-example-bucket ~/gcs_mount
# Run Visual Layer profiler on the mounted folder
./run_profiler.sh --input ~/gcs_mount/dataset-folder/
```
### Using Command-Line Copied Data
First copy the data locally using gsutil, then run the profiler:
```bash theme={"theme":"monokai"}
# Copy data from bucket to local directory
gsutil -m cp -r gs://my-example-bucket/dataset-folder/ ~/local-dataset/
# Run Visual Layer profiler on the local folder
./run_profiler.sh --input ~/local-dataset/
```
### Configuration Options
When using GCS buckets with Visual Layer on-premise installations, you can:
* **Use mounted buckets** for seamless filesystem access (recommended).
* **Copy data locally** using gsutil commands for faster processing.
* **Configure bucket URLs** directly in Visual Layer dataset creation.
For Visual Layer specific configuration, refer to the [S3 Configuration](/docs/self-hosting/s3-configuration) documentation for similar patterns that can be adapted for GCS.
## Related Resources
Configure S3 bucket access for Visual Layer
On-premise deployment guides and scripts
## Need Help?
For additional support:
* Contact Visual Layer support: [support@visual-layer.com](mailto:support@visual-layer.com)
* Refer to [Google Cloud Storage documentation](https://cloud.google.com/storage/docs)
* Check [GCS FUSE documentation](https://cloud.google.com/storage/docs/gcs-fuse) for advanced configuration options