Well-known URIs: Standardizing Web Metadata Discovery

Every web developer has run into this: the metadata a site exposes lives in a different place on every site. Where do you find a site’s security contact? How do you discover its OAuth endpoints? What about the password-change URL a password manager needs? The web is decentralized by design, and for most of its history that meant each site published this kind of service metadata however it happened to feel like.

RFC 8615, published by the Internet Engineering Task Force (IETF), fixes that. It reserves a single path prefix, /.well-known/, as the place a site publishes machine-readable information about its services and policies. If you know the suffix, you know the URL.

Understanding Well-known URIs

Well-known URIs are defined in RFC 8615. They carve out a namespace under the /.well-known/ path prefix where a site can expose machine-readable information about its services, policies, and capabilities.

The problem behind the spec is a real one. More and more protocols need a piece of information to sit at a predictable location, no matter how a given host organizes the rest of its URLs. Pin that location down once and automated discovery stops being a per-site integration project.

The Technical Foundation

Well-known URIs follow a simple pattern:

https://example.com/.well-known/{service-name}

This structure provides several key advantages:

• Predictability: Clients know exactly where to look for specific metadata
• Namespace Isolation: The .well-known prefix prevents conflicts with existing site structure
• Extensibility: New services can be added without affecting existing implementations
• Cross-Origin Compatibility: Standard HTTP mechanisms apply for access control

The Problem Well-known URIs Solve

Before standardization, discovering service metadata required ad-hoc approaches that varied significantly across implementations. Consider these common scenarios:

Security Contact Discovery

Before Well-known URIs:

# Multiple possible locations, no standard format
https://example.com/security
https://example.com/contact/security
https://example.com/about/security-team
https://example.com/responsible-disclosure

With Well-known URIs:

# Single, predictable location with standardized format
https://example.com/.well-known/security.txt

OAuth/OpenID Connect Discovery

Before Well-known URIs:

# Provider-specific discovery mechanisms
https://accounts.google.com/.well-known/openid_configuration
https://login.microsoftonline.com/common/.well-known/openid_configuration
# But many providers used different paths entirely

With Well-known URIs:

# Standardized discovery endpoint
https://any-provider.com/.well-known/openid-configuration

That predictability cuts integration work and lets tooling behave the same way against any provider.

Implementation Architecture

Server Configuration

Implementing well-known URIs requires configuring your web server to serve content from the /.well-known/ directory. Here are examples for common server configurations:

Apache Configuration

# Enable .well-known directory
<Directory "/var/www/html/.well-known">
    Options -Indexes
    AllowOverride None
    Require all granted

    # Set appropriate content types
    <Files "security.txt">
        Header set Content-Type "text/plain; charset=utf-8"
    </Files>

    <Files "openid-configuration">
        Header set Content-Type "application/json; charset=utf-8"
    </Files>
</Directory>

Nginx Configuration

location /.well-known/ {
    root /var/www/html;

    # Security headers
    add_header X-Content-Type-Options nosniff;
    add_header Cache-Control "public, max-age=3600";

    # Content type mapping
    location ~ \.txt$ {
        add_header Content-Type "text/plain; charset=utf-8";
    }

    location ~ /openid-configuration$ {
        add_header Content-Type "application/json; charset=utf-8";
    }
}

Dynamic Implementation

For applications requiring dynamic well-known URI generation:

// Express.js implementation
const express = require('express');
const app = express();

// Well-known URI middleware
app.use('/.well-known', (req, res, next) => {
    // Set security headers
    res.set({
        'X-Content-Type-Options': 'nosniff',
        'Cache-Control': 'public, max-age=3600'
    });
    next();
});

// Security.txt endpoint
app.get('/.well-known/security.txt', (req, res) => {
    res.type('text/plain');
    res.send(`Contact: security@example.com
Expires: 2025-12-31T23:59:59.000Z
Encryption: https://example.com/pgp-key.txt
Preferred-Languages: en
Canonical: https://example.com/.well-known/security.txt`);
});

// OpenID Connect discovery
app.get('/.well-known/openid-configuration', (req, res) => {
    res.json({
        issuer: 'https://example.com',
        authorization_endpoint: 'https://example.com/auth',
        token_endpoint: 'https://example.com/token',
        userinfo_endpoint: 'https://example.com/userinfo',
        jwks_uri: 'https://example.com/.well-known/jwks.json',
        response_types_supported: ['code', 'token', 'id_token'],
        subject_types_supported: ['public'],
        id_token_signing_alg_values_supported: ['RS256']
    });
});

Essential Well-known URIs

IANA keeps the registry of standardized well-known URIs. A few that come up most often:

Security and Policy

security.txt - Security contact information and vulnerability disclosure policies

Contact: security@example.com
Expires: 2025-12-31T23:59:59.000Z
Encryption: https://example.com/pgp-key.txt
Policy: https://example.com/security-policy

change-password - Direct link to password change functionality for password managers

https://example.com/.well-known/change-password
# Redirects to: https://example.com/account/password

Authentication and Identity

openid-configuration - OAuth 2.0/OpenID Connect provider metadata webfinger - Identity discovery for federated protocols host-meta - General host metadata in XML format

Application Integration

apple-app-site-association - iOS Universal Links configuration assetlinks.json - Android App Links verification matrix - Matrix protocol server discovery

Development and Automation

robots.txt equivalent URIs for specialized crawlers nodeinfo - Federated social network metadata timezone - Time zone data distribution

Security Considerations

Well-known URIs come with security benefits and a few risks worth thinking about:

Security Benefits

1. Standardized Security Contact: The security.txt standard provides a reliable way for security researchers to report vulnerabilities
2. Reduced Attack Surface: Centralized metadata reduces the need for custom discovery mechanisms
3. Improved Transparency: Standardized policy disclosure enhances security posture visibility

Potential Risks

1. Information Disclosure: Well-known URIs may reveal sensitive information about system architecture
2. Attack Vector Expansion: Improperly configured endpoints could expose internal services
3. Cache Poisoning: Incorrect caching headers could lead to stale or malicious metadata

Best Practices

# Security-focused well-known configuration
location /.well-known/ {
    # Prevent directory traversal
    location ~ \.\. {
        deny all;
    }

    # Rate limiting
    limit_req zone=wellknown burst=10 nodelay;

    # Security headers
    add_header X-Content-Type-Options nosniff;
    add_header X-Frame-Options DENY;
    add_header Referrer-Policy strict-origin-when-cross-origin;

    # Appropriate caching
    expires 1h;
    add_header Cache-Control "public, immutable";
}

Practical Implementation Guide

Step 1: Create the Well-known Directory Structure

# Create the directory structure
mkdir -p /var/www/html/.well-known

# Set appropriate permissions
chmod 755 /var/www/html/.well-known

Step 2: Implement Security.txt

# Create security.txt file
cat > /var/www/html/.well-known/security.txt << EOF
Contact: mailto:security@example.com
Contact: https://example.com/security-contact
Expires: 2025-12-31T23:59:59.000Z
Encryption: https://example.com/pgp-key.txt
Acknowledgments: https://example.com/security-acknowledgments
Preferred-Languages: en, es
Canonical: https://example.com/.well-known/security.txt
Policy: https://example.com/vulnerability-disclosure-policy
EOF

Step 3: Validation and Testing

#!/usr/bin/env python3
"""
Well-known URI validator script
"""
import requests
import json
from urllib.parse import urljoin

def validate_security_txt(base_url):
    """Validate security.txt implementation"""
    url = urljoin(base_url, '/.well-known/security.txt')

    try:
        response = requests.get(url, timeout=10)
        response.raise_for_status()

        # Check content type
        content_type = response.headers.get('content-type', '')
        if 'text/plain' not in content_type:
            print(f"Warning: Unexpected content-type: {content_type}")

        # Parse and validate required fields
        content = response.text
        required_fields = ['Contact', 'Expires']

        for field in required_fields:
            if field not in content:
                print(f"Error: Missing required field: {field}")
                return False

        print("✓ security.txt validation passed")
        return True

    except requests.RequestException as e:
        print(f"Error accessing security.txt: {e}")
        return False

def validate_openid_configuration(base_url):
    """Validate OpenID Connect configuration"""
    url = urljoin(base_url, '/.well-known/openid-configuration')

    try:
        response = requests.get(url, timeout=10)
        response.raise_for_status()

        # Check content type
        content_type = response.headers.get('content-type', '')
        if 'application/json' not in content_type:
            print(f"Warning: Unexpected content-type: {content_type}")

        # Parse JSON and validate required fields
        config = response.json()
        required_fields = [
            'issuer', 'authorization_endpoint',
            'token_endpoint', 'jwks_uri'
        ]

        for field in required_fields:
            if field not in config:
                print(f"Error: Missing required field: {field}")
                return False

        print("✓ OpenID Connect configuration validation passed")
        return True

    except (requests.RequestException, json.JSONDecodeError) as e:
        print(f"Error accessing OpenID configuration: {e}")
        return False

if __name__ == "__main__":
    base_url = "https://example.com"
    validate_security_txt(base_url)
    validate_openid_configuration(base_url)

Advanced Use Cases

Content Delivery Network Integration

// Cloudflare Workers implementation
addEventListener('fetch', event => {
    event.respondWith(handleRequest(event.request))
})

async function handleRequest(request) {
    const url = new URL(request.url)

    // Handle well-known URIs
    if (url.pathname.startsWith('/.well-known/')) {
        return handleWellKnownRequest(url.pathname)
    }

    // Forward other requests
    return fetch(request)
}

async function handleWellKnownRequest(pathname) {
    const wellKnownRoutes = {
        '/.well-known/security.txt': () => new Response(
            generateSecurityTxt(),
            {
                headers: {
                    'Content-Type': 'text/plain; charset=utf-8',
                    'Cache-Control': 'public, max-age=3600'
                }
            }
        ),

        '/.well-known/openid-configuration': () => new Response(
            JSON.stringify(generateOpenIDConfig()),
            {
                headers: {
                    'Content-Type': 'application/json; charset=utf-8',
                    'Cache-Control': 'public, max-age=3600'
                }
            }
        )
    }

    const handler = wellKnownRoutes[pathname]
    if (handler) {
        return handler()
    }

    return new Response('Not Found', { status: 404 })
}

function generateSecurityTxt() {
    return `Contact: security@example.com
Expires: ${new Date(Date.now() + 365 * 24 * 60 * 60 * 1000).toISOString()}
Encryption: https://example.com/pgp-key.txt
Canonical: https://example.com/.well-known/security.txt`
}

function generateOpenIDConfig() {
    return {
        issuer: 'https://example.com',
        authorization_endpoint: 'https://example.com/oauth/authorize',
        token_endpoint: 'https://example.com/oauth/token',
        userinfo_endpoint: 'https://example.com/oauth/userinfo',
        jwks_uri: 'https://example.com/.well-known/jwks.json',
        response_types_supported: ['code'],
        subject_types_supported: ['public'],
        id_token_signing_alg_values_supported: ['RS256']
    }
}

Monitoring and Analytics

# Well-known URI monitoring script
import requests
import time
import logging
from datetime import datetime, timedelta

class WellKnownMonitor:
    def __init__(self, base_url):
        self.base_url = base_url
        self.logger = logging.getLogger(__name__)

    def check_endpoint(self, path, expected_content_type):
        """Monitor a specific well-known endpoint"""
        url = f"{self.base_url}/.well-known/{path}"

        try:
            start_time = time.time()
            response = requests.get(url, timeout=10)
            response_time = time.time() - start_time

            # Log metrics
            self.logger.info(f"Endpoint: {path}")
            self.logger.info(f"Status: {response.status_code}")
            self.logger.info(f"Response Time: {response_time:.3f}s")
            self.logger.info(f"Content-Type: {response.headers.get('content-type')}")

            # Validate content type
            if expected_content_type not in response.headers.get('content-type', ''):
                self.logger.warning(f"Unexpected content-type for {path}")

            # Check for security headers
            security_headers = [
                'X-Content-Type-Options',
                'Cache-Control'
            ]

            for header in security_headers:
                if header not in response.headers:
                    self.logger.warning(f"Missing security header: {header}")

            return response.status_code == 200

        except requests.RequestException as e:
            self.logger.error(f"Error checking {path}: {e}")
            return False

    def run_checks(self):
        """Run all well-known URI checks"""
        endpoints = [
            ('security.txt', 'text/plain'),
            ('openid-configuration', 'application/json'),
            ('change-password', 'text/html')
        ]

        results = {}
        for path, content_type in endpoints:
            results[path] = self.check_endpoint(path, content_type)

        return results

# Usage
if __name__ == "__main__":
    logging.basicConfig(level=logging.INFO)
    monitor = WellKnownMonitor("https://example.com")
    results = monitor.run_checks()
    print(f"Check results: {results}")

Registry and Standardization Process

The Internet Assigned Numbers Authority (IANA) maintains the official Well-Known URIs registry, which serves as the authoritative source for standardized well-known URI suffixes. This registry ensures global coordination and prevents conflicts between different specifications.

Proposing New Well-known URIs

To propose a new well-known URI, you must follow the IETF specification process:

1. Draft Specification: Create an Internet-Draft describing the proposed URI and its purpose
2. Community Review: Submit to the wellknown-uri-review@ietf.org mailing list
3. IANA Registration: Complete the registration template with required fields
4. Expert Review: IANA designated experts review the proposal
5. Publication: Upon approval, the URI is added to the official registry

Registration Template

URI suffix: example-service
Change controller: IETF
Specification document: RFC 8615, Section 3.1
Status: permanent
Related information: Optional additional context

Current Adoption and Future Trends

Adoption is broad at this point:

Industry Adoption

• Major Platforms: Google, Microsoft, Apple, and other tech giants extensively use well-known URIs
• Security Tools: Security scanners and vulnerability management platforms rely on security.txt
• Identity Providers: OAuth and OpenID Connect providers universally implement discovery endpoints
• Password Managers: Modern password managers use change-password to send people straight to the right page

Emerging Trends

1. Federated Protocols: Matrix, Mastodon, and other federated platforms use well-known URIs for server discovery
2. Privacy Standards: Global Privacy Control (GPC) and similar privacy frameworks adopt well-known URIs
3. AI and Automation: Machine learning platforms use well-known URIs for model and API discovery
4. IoT Integration: Internet of Things devices increasingly expose metadata via well-known URIs

Performance and Caching Considerations

Caching strategy matters here:

# Optimal caching headers for well-known URIs
Cache-Control: public, max-age=3600, immutable
ETag: "v1.2.3-20250114"
Last-Modified: Tue, 14 Jan 2025 10:00:00 GMT
Vary: Accept-Encoding

CDN Configuration

# CloudFront distribution configuration
wellknown_cache_behavior:
  path_pattern: "/.well-known/*"
  target_origin_id: "primary-origin"
  viewer_protocol_policy: "redirect-to-https"
  cache_policy:
    default_ttl: 3600
    max_ttl: 86400
    min_ttl: 0
  compress: true
  headers:
    - "Content-Type"
    - "Cache-Control"

Troubleshooting Common Issues

CORS Configuration

// Express.js CORS configuration for well-known URIs
app.use('/.well-known', cors({
    origin: true,
    methods: ['GET', 'HEAD'],
    allowedHeaders: ['Content-Type'],
    maxAge: 3600
}));

Content-Type Issues

# Apache MIME type configuration
<Files "security.txt">
    ForceType text/plain
</Files>

<Files "openid-configuration">
    ForceType application/json
</Files>

<Files "jwks.json">
    ForceType application/json
</Files>

SSL/TLS Considerations

# Nginx SSL configuration for well-known URIs
location /.well-known/ {
    # Allow HTTP for ACME challenges
    if ($request_uri ~ "^/.well-known/acme-challenge/") {
        # ACME challenge can use HTTP
    }

    # Force HTTPS for other well-known URIs
    if ($scheme = http) {
        return 301 https://$server_name$request_uri;
    }
}

Integration with Modern Development Workflows

Docker Implementation

# Dockerfile for well-known URI server
FROM nginx:alpine

# Copy well-known files
COPY .well-known/ /usr/share/nginx/html/.well-known/

# Copy nginx configuration
COPY nginx.conf /etc/nginx/nginx.conf

# Set appropriate permissions
RUN chmod -R 644 /usr/share/nginx/html/.well-known/

EXPOSE 80 443

CMD ["nginx", "-g", "daemon off;"]

Kubernetes Deployment

apiVersion: v1
kind: ConfigMap
metadata:
  name: wellknown-config
data:
  security.txt: |
    Contact: security@example.com
    Expires: 2025-12-31T23:59:59.000Z
    Canonical: https://example.com/.well-known/security.txt

  openid-configuration: |
    {
      "issuer": "https://example.com",
      "authorization_endpoint": "https://example.com/oauth/authorize",
      "token_endpoint": "https://example.com/oauth/token"
    }

---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: wellknown-server
spec:
  replicas: 2
  selector:
    matchLabels:
      app: wellknown-server
  template:
    metadata:
      labels:
        app: wellknown-server
    spec:
      containers:
      - name: nginx
        image: nginx:alpine
        ports:
        - containerPort: 80
        volumeMounts:
        - name: wellknown-volume
          mountPath: /usr/share/nginx/html/.well-known
      volumes:
      - name: wellknown-volume
        configMap:
          name: wellknown-config

Conclusion and Strategic Recommendations

The payoff of well-known URIs is boring in the best way: metadata discovery stops being guesswork. Less integration glue, a clearer security posture, tooling that behaves the same everywhere.

If you’re rolling these out, here’s what I’d do:

1. Start with Security: Implement security.txt as your first well-known URI to improve security posture
2. Plan for Scale: Design your implementation to handle high traffic and provide appropriate caching
3. Monitor Continuously: Implement monitoring to ensure well-known URIs remain accessible and current
4. Follow Standards: Adhere to IANA registry specifications and IETF best practices
5. Consider Privacy: Evaluate what information you expose through well-known URIs

The point isn’t the individual files. It’s that discovery stops depending on convention and starts depending on a spec. Implement these and you’re part of a web that machines can navigate without a scraper and a prayer.

As more traffic comes from automated clients rather than people, that predictability is worth a little more every year.

---

Want to explore more web standards and protocols? Check out my post on building MCP servers for more on connecting open protocols to modern AI.