PreviewLab4.5 hours

Bucket Types Lab

Compare S3 bucket types, general purpose, directory buckets (Express One Zone), and table buckets.

Prerequisites

0 of 6 checked

General Purpose Buckets: Recap + Advanced Patterns

Goal

Lock in what makes a General Purpose bucket the baseline. Explore cross-bucket replication, object tagging for cost allocation, and batch operations, features that only exist on General Purpose. Contrast the API explicitly against what Directory and Table buckets can and can't do.

Estimated time: 1–2 hours

Create a General Purpose bucket and verify defaults

What's happening here

A General Purpose bucket is the original S3 bucket type. Names are globally unique across all AWS accounts and all bucket types, no two buckets anywhere can share a name. Data is stored across ≥3 AZs in the region, giving 99.999999999% (11 nines) durability. There's no per-AZ placement control. AWS decides. This is the right default for almost everything. The other bucket types exist for specific performance or query-model needs, not as general improvements.

bash

mkdir bktypes-lab && cd bktypes-lab

export GP_BUCKET="${LAB_PREFIX}-gp"

aws s3api create-bucket \
  --bucket $GP_BUCKET \
  --region $AWS_REGION

# Confirm bucket type (general purpose has no LocationType field)
aws s3api get-bucket-location --bucket $GP_BUCKET --region $AWS_REGION
aws s3api head-bucket --bucket $GP_BUCKET --region $AWS_REGION

echo "General Purpose bucket: $GP_BUCKET"

Check what IAM actions are available on a GP bucket vs the others:

bash

# These API calls only work on General Purpose buckets
aws s3api get-bucket-versioning --bucket $GP_BUCKET --region $AWS_REGION
aws s3api get-bucket-replication --bucket $GP_BUCKET --region $AWS_REGION 2>&1 || echo "No replication configured yet"
aws s3api get-bucket-lifecycle-configuration --bucket $GP_BUCKET --region $AWS_REGION 2>&1 || echo "No lifecycle configured"
aws s3api get-bucket-notification-configuration --bucket $GP_BUCKET --region $AWS_REGION

all these config APIs exist and are queryable. None of them exist on Directory or Table buckets, they have a fundamentally different management surface.

Checkpoint: GP bucket exists, all config APIs return without auth errors.

Object tagging for cost allocation

What's happening here

Object tags are key-value metadata separate from user-defined metadata. They serve two purposes: cost allocation (tags flow into Cost Explorer and Billing Reports so you can break down S3 spend by team, env, or service) and lifecycle rule filtering (you can target lifecycle rules at objects with a specific tag rather than a key prefix). Tags are mutable after upload, you can tag an object without re-uploading it. Max 10 tags per object. Directory and Table buckets do not support per-object tagging.

bash

# Upload a few objects with tags
echo '{"id": "inv-001", "amount": 9900}' > invoice.json
echo '{"id": "rpt-2024-q1", "rows": 50000}' > report.json
echo 'raw log data' > access.log

aws s3api put-object \
  --bucket $GP_BUCKET \
  --key invoices/inv-001.json \
  --body invoice.json \
  --tagging 'team=billing&env=prod&cost-center=fin-001' \
  --region $AWS_REGION

aws s3api put-object \
  --bucket $GP_BUCKET \
  --key reports/rpt-2024-q1.json \
  --body report.json \
  --tagging 'team=analytics&env=prod&cost-center=data-002' \
  --region $AWS_REGION

aws s3api put-object \
  --bucket $GP_BUCKET \
  --key logs/access.log \
  --body access.log \
  --tagging 'team=platform&env=prod&cost-center=infra-003' \
  --region $AWS_REGION

# Read tags on an object
aws s3api get-object-tagging \
  --bucket $GP_BUCKET \
  --key invoices/inv-001.json \
  --region $AWS_REGION | jq '.TagSet[]'

# Modify tags without re-uploading
aws s3api put-object-tagging \
  --bucket $GP_BUCKET \
  --key invoices/inv-001.json \
  --tagging '{"TagSet": [{"Key": "team", "Value": "billing"}, {"Key": "env", "Value": "prod"}, {"Key": "status", "Value": "archived"}]}' \
  --region $AWS_REGION

aws s3api get-object-tagging \
  --bucket $GP_BUCKET \
  --key invoices/inv-001.json \
  --region $AWS_REGION | jq '.TagSet[]'

the status=archived tag was added without touching the object body. Tags are a side-channel on the object, independent of content.

Tag-based lifecycle rule

What's happening here

Lifecycle rules on General Purpose buckets can filter by both prefix and tag. A tag-based rule lets you use the same key prefix (invoices/) for live and archived objects, differentiating them by the status tag rather than moving them to a separate prefix. This avoids polluting your key namespace with lifecycle-motivated path segments. The rule here transitions objects tagged status=archived to GLACIER_IR after 1 day. In production you'd set a longer duration, 1 day is lab-speed.

bash

cat > tag-lifecycle.json << 'EOF'
{
  "Rules": [
    {
      "ID": "archive-status-tagged",
      "Status": "Enabled",
      "Filter": {
        "And": {
          "Prefix": "invoices/",
          "Tags": [{"Key": "status", "Value": "archived"}]
        }
      },
      "Transitions": [
        {
          "Days": 1,
          "StorageClass": "GLACIER_IR"
        }
      ]
    }
  ]
}
EOF

aws s3api put-bucket-lifecycle-configuration \
  --bucket $GP_BUCKET \
  --lifecycle-configuration file://tag-lifecycle.json \
  --region $AWS_REGION

aws s3api get-bucket-lifecycle-configuration \
  --bucket $GP_BUCKET \
  --region $AWS_REGION | jq '.Rules[] | {ID, Status, Filter}'

Checkpoint: lifecycle rule set with tag filter status=archived.

Cross-region replication (GP-only feature)

What's happening here

Cross-region replication (CRR) asynchronously copies objects from a source GP bucket to a destination GP bucket in a different region. Key design facts: (1) versioning must be enabled on both source and destination, (2) replication only copies objects written after the rule is enabled, existing objects are not replicated automatically (use S3 Batch Operations for backfill), (3) delete markers are not replicated by default (configurable), (4) there's no guarantee on replication lag, it's typically seconds to minutes. Directory and Table buckets do not support replication.

bash

# Create destination bucket in a different region
export DEST_REGION="us-west-2"
export DEST_BUCKET="${LAB_PREFIX}-gp-replica"

aws s3api create-bucket \
  --bucket $DEST_BUCKET \
  --create-bucket-configuration LocationConstraint=$DEST_REGION \
  --region $DEST_REGION

# Enable versioning on both (required for replication)
aws s3api put-bucket-versioning \
  --bucket $GP_BUCKET \
  --versioning-configuration Status=Enabled \
  --region $AWS_REGION

aws s3api put-bucket-versioning \
  --bucket $DEST_BUCKET \
  --versioning-configuration Status=Enabled \
  --region $DEST_REGION

# Create IAM role for S3 replication
cat > replication-trust.json << 'EOF'
{
  "Version": "2012-10-17",
  "Statement": [{
    "Effect": "Allow",
    "Principal": {"Service": "s3.amazonaws.com"},
    "Action": "sts:AssumeRole"
  }]
}
EOF

export REPL_ROLE_ARN=$(aws iam create-role \
  --role-name "${LAB_PREFIX}-s3-replication" \
  --assume-role-policy-document file://replication-trust.json \
  --query 'Role.Arn' --output text)

echo "Replication role ARN: $REPL_ROLE_ARN"

bash

# Attach replication permissions to the role
cat > replication-policy.json << EOF
{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": ["s3:GetReplicationConfiguration", "s3:ListBucket"],
      "Resource": "arn:aws:s3:::$GP_BUCKET"
    },
    {
      "Effect": "Allow",
      "Action": ["s3:GetObjectVersionForReplication", "s3:GetObjectVersionAcl", "s3:GetObjectVersionTagging"],
      "Resource": "arn:aws:s3:::$GP_BUCKET/*"
    },
    {
      "Effect": "Allow",
      "Action": ["s3:ReplicateObject", "s3:ReplicateDelete", "s3:ReplicateTags"],
      "Resource": "arn:aws:s3:::$DEST_BUCKET/*"
    }
  ]
}
EOF

aws iam put-role-policy \
  --role-name "${LAB_PREFIX}-s3-replication" \
  --policy-name replication-perms \
  --policy-document file://replication-policy.json

# Configure replication rule
cat > replication-config.json << EOF
{
  "Role": "$REPL_ROLE_ARN",
  "Rules": [{
    "ID": "replicate-all",
    "Status": "Enabled",
    "Filter": {"Prefix": ""},
    "Destination": {
      "Bucket": "arn:aws:s3:::$DEST_BUCKET",
      "StorageClass": "STANDARD"
    },
    "DeleteMarkerReplication": {"Status": "Disabled"}
  }]
}
EOF

aws s3api put-bucket-replication \
  --bucket $GP_BUCKET \
  --replication-configuration file://replication-config.json \
  --region $AWS_REGION

aws s3api get-bucket-replication --bucket $GP_BUCKET --region $AWS_REGION | jq '.ReplicationConfiguration.Rules[] | {ID, Status, Destination: .Destination.Bucket}'

bash

# Upload a new object AFTER replication is configured
echo '{"id": "inv-002", "amount": 15000}' > invoice2.json
aws s3api put-object \
  --bucket $GP_BUCKET \
  --key invoices/inv-002.json \
  --body invoice2.json \
  --region $AWS_REGION

echo "Object uploaded. Replication is async, checking destination in 10s..."
sleep 10

# Check if it appeared in the destination
aws s3api head-object \
  --bucket $DEST_BUCKET \
  --key invoices/inv-002.json \
  --region $DEST_REGION 2>&1 | jq '{ContentLength, ReplicationStatus: .ReplicationStatus}' 2>/dev/null || echo "Not yet replicated, try again in a few seconds"

# Check replication status on source object
aws s3api head-object \
  --bucket $GP_BUCKET \
  --key invoices/inv-002.json \
  --region $AWS_REGION | jq '{ReplicationStatus}'

source object shows ReplicationStatus: COMPLETED. Destination object has a matching copy. Objects uploaded *before* the replication rule show no ReplicationStatus: they were never replicated.

Checkpoint: replicated object appears in destination bucket.

Break It: Features That Don't Exist on the Other Types

bash

# Note these commands for comparison, we'll try their equivalents on Directory
# and Table buckets in Sessions 2 and 3 and observe the failures.

# Tag a GP object (works)
aws s3api get-object-tagging \
  --bucket $GP_BUCKET \
  --key invoices/inv-001.json \
  --region $AWS_REGION | jq '{tagsAvailable: true, count: (.TagSet | length)}'

# Versioning on GP (works)
aws s3api get-bucket-versioning --bucket $GP_BUCKET --region $AWS_REGION | jq '{versioningAvailable: true}'

# Replication on GP (works)
aws s3api get-bucket-replication --bucket $GP_BUCKET --region $AWS_REGION | jq '{replicationAvailable: true}'

echo "Save these as the GP baseline. Sessions 2 and 3 will call the equivalent APIs on Directory/Table buckets and show what fails."

Directory Buckets (S3 Express One Zone)

Goal

Create a Directory bucket in a specific AZ. Understand the performance contract and the trade-offs. Use the Express-specific auth flow (create-session). Observe which GP features are absent and why.