PreviewLab4.5 hours

Vector Buckets Lab

Create vector buckets and indexes, write and query embeddings, and configure IAM and encryption.

Prerequisites

0 of 6 checked

Vector Bucket + Index: Create, Inspect, Understand the Config Contract

Goal

Create a vector bucket and two indexes with different configurations. Understand which parameters are immutable at create time and why. Inspect the ARN structure. Try changing immutable fields and observe the hard failure.

Estimated time: 1 hour

Create a vector bucket

What's happening here

A vector bucket is a first-class resource with its own CLI namespace (s3vectors), its own IAM namespace (s3vectors:*), and its own ARN pattern (arn:aws:s3vectors:...). It is invisible to s3api: you cannot head-bucket, list-objects-v2, or put-object on a vector bucket. Bucket names follow the same naming rules as General Purpose buckets (3–63 chars, lowercase, hyphens only), but the name is scoped to your account and region, not globally unique like GP bucket names. Block Public Access is always on for vector buckets and cannot be disabled.

bash

mkdir svec-lab && cd svec-lab

export VEC_BUCKET="${LAB_PREFIX}-vecs"

aws s3vectors create-vector-bucket \
  --vector-bucket-name $VEC_BUCKET \
  --region $AWS_REGION

echo "Vector bucket: $VEC_BUCKET"

# Describe the bucket
aws s3vectors get-vector-bucket \
  --vector-bucket-name $VEC_BUCKET \
  --region $AWS_REGION | jq '.vectorBucket | {name, arn, createdAt, encryptionConfiguration}'

# List all vector buckets in the region
aws s3vectors list-vector-buckets --region $AWS_REGION | jq '.vectorBuckets[] | {name, arn}'

the ARN format is arn:aws:s3vectors:{region}:{account}:bucket/{bucket-name}: note the s3vectors service name, not s3. Default encryption is SSE-S3 (same as GP buckets post-2023).

Checkpoint: bucket created, ARN visible in get-vector-bucket output.

Create indexes: cosine vs euclidean, filterable vs non-filterable metadata

What's happening here

A vector index is the unit of storage and query. You define three things at create time that cannot be changed afterward:

dimension: the number of floats per vector (must match your embedding model output exactly)
distance-metric: cosine or euclidean
metadata-configuration.nonFilterableMetadataKeys: which metadata keys are excluded from filter indexing

These are immutable because changing them would require rebuilding the ANN index structure from scratch. If you get any wrong, you must delete the index and recreate it. You do not need to pre-declare all metadata keys, only the ones you want to mark as non-filterable.

Cosine vs Euclidean:

Cosine measures the angle between vectors (direction similarity); euclidean measures straight-line distance (magnitude + direction). Most text embedding models recommend cosine. Image embedding models often use euclidean. Use the metric your embedding model's documentation specifies; mixing them gives wrong results.

Filterable vs non-filterable metadata:

By default all metadata keys are filterable and count toward the 2 KB per-vector filterable limit. Large text payloads (e.g. the original document chunk) must be declared non-filterable to use the 40 KB per-vector total limit without burning filterable capacity.

bash

# Index 1: 8-dimensional, cosine distance (simulates text embeddings)
# In production: 1536 dims for Amazon Titan, 1024 for Cohere, 768 for many others
export IDX_TEXT="text-embeddings"

aws s3vectors create-index \
  --vector-bucket-name $VEC_BUCKET \
  --index-name $IDX_TEXT \
  --data-type float32 \
  --dimension 8 \
  --distance-metric cosine \
  --region $AWS_REGION

# Index 2: 4-dimensional, euclidean distance, with non-filterable key for large payloads
export IDX_IMG="image-embeddings"

aws s3vectors create-index \
  --vector-bucket-name $VEC_BUCKET \
  --index-name $IDX_IMG \
  --data-type float32 \
  --dimension 4 \
  --distance-metric euclidean \
  --metadata-configuration '{"nonFilterableMetadataKeys": ["raw_text", "document_url"]}' \
  --region $AWS_REGION

# Inspect both indexes
aws s3vectors get-index \
  --vector-bucket-name $VEC_BUCKET \
  --index-name $IDX_TEXT \
  --region $AWS_REGION | jq '.index | {name, arn, dimension, distanceMetric, dataType, createdAt}'

aws s3vectors get-index \
  --vector-bucket-name $VEC_BUCKET \
  --index-name $IDX_IMG \
  --region $AWS_REGION | jq '.index | {name, dimension, distanceMetric, metadataConfiguration}'

# List all indexes in the bucket
aws s3vectors list-indexes \
  --vector-bucket-name $VEC_BUCKET \
  --region $AWS_REGION | jq '.indexes[] | {name, dimension, distanceMetric}'

index ARN format is arn:aws:s3vectors:{region}:{account}:bucket/{bucket}/index/{index}. The metadataConfiguration on image-embeddings shows nonFilterableMetadataKeys: ["raw_text", "document_url"].

Checkpoint: two indexes created with different distance metrics.

The immutability contract

What's happening here

The three immutable fields (dimension, distance metric, non-filterable keys) are locked at index creation because they define the ANN (approximate nearest neighbor) index structure. Changing the dimension would make all stored vectors the wrong shape. Changing the distance metric would make all similarity scores meaningless. Changing non-filterable keys would require re-indexing all metadata. There is no update API for these fields. The update-index operation (if it existed) would only cover mutable properties like tags. The only way to change them is delete + recreate.

bash

# Try to create an index with the same name, conflicts
aws s3vectors create-index \
  --vector-bucket-name $VEC_BUCKET \
  --index-name $IDX_TEXT \
  --data-type float32 \
  --dimension 16 \
  --distance-metric cosine \
  --region $AWS_REGION 2>&1 | head -5

ConflictException: index name already exists. There is no "update dimension" path. If you need dimension 16, delete the index and recreate it, then re-insert all vectors.

bash

# Zero vector constraint for cosine (zero vectors have no direction)
aws s3vectors put-vectors \
  --vector-bucket-name $VEC_BUCKET \
  --index-name $IDX_TEXT \
  --vectors '[{"key": "zero-vec", "data": {"float32": [0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0]}}]' \
  --region $AWS_REGION 2>&1 | head -5

ValidationException: zero vectors are explicitly rejected for cosine distance because cosine similarity is undefined for the zero vector (you'd be dividing by zero when normalizing). Euclidean allows zero vectors.

Writing Vectors: Put, Get, List, Delete

Goal

Insert vectors with metadata, retrieve them by key, list with parallel segmented scanning, and delete. Understand the difference between filterable and non-filterable metadata in practice. Observe strong consistency.