# Troubleshooting
This article provides troubleshooting information for various tools and features in Pomerium.
# Pomerium Core
# HTTP Strict Transport Security (HSTS)
By default, Pomerium sends the Strict-Transport-Security
(opens new window) response header to the browser, which pins the certificate to our browser for one year. This is common best practice to help prevent man-in-the-middle attacks but can create issues while a new Pomerium configuration is in development.
When you visit an endpoint while Pomerium is using an untrusted certificate (like the self-signed bootstrap certificate or a Let's Encrypt staging certificate), that certificate may be pinned in your browser. Once Pomerium is switched to a trusted production certificate, the untrusted cert must reset in the browser.
While developing your Pomerium environment, consider adjusting the SET_RESPONSE_HEADERS
key to remove Strict-Transport-Security
or reduce the max-age
value until your production certificates are in place.
See this article (opens new window) for more information on clearing HSTS for specific endpoints across common browsers.
# JWT Authentication
When securing the Pomerium Authenticate service with a certificate signed by Let's Encrypt, your upstream applications may reject the certificate when attempting to access the JWT signing key. Here's an example log line from Grafana:
logger=context error=Get "https://authenticate.localhost.pomerium.io/.well-known/pomerium/jwks.json": x509: certificate signed by unknown authority
Copied!
This is often due to the recent expiration of the DST Root CA X3 (opens new window) certificate. Many default keystores used by docker images and less-frequently updated distributions still carry this expired certificate. Even though Let's Encrypt certs are cross-signed with the ISRG Root X1 (opens new window) CA certificate, some applications will still reject them.
To clarify; this does not mean that the upstream service is rejecting the JWT signing key. Rather, it doesn't trust the Let's Encrypt certificate used by the Authorize service for TLS, and so it will not read the JWKS file.
For upstream applications that can use a local signing key file, you can circumvent this issue using curl
or wget
to download the signing key locally (relative to the upstream service). Using Grafana again as an example:
Download the
jwks.json
file from the authenticate server:curlwgetcurl https://authenticate.localhost.pomerium.io/.well-known/pomerium/jwks.json > /etc/grafana/jwks.json
Copied!Edit the upstream service configuration to use the local key to verify tokens:
[auth.jwt] enabled = true header_name = X-Pomerium-Jwt-Assertion email_claim = email jwk_set_file = /etc/grafana/jwks.json cache_ttl = 60m
Copied!
# Kubernetes Ingress Controller
# View Event History
Pomerium Ingress Controller will add events to the Ingress objects as it processes them.
kubectl describe ingress/my-ingress
Copied!
Events: Type Reason Age From Message ---- ------ ---- ---- ------- Normal Updated 18s pomerium-ingress updated pomerium configuration
Copied!
If an error occurs, it may be reflected in the events:
Events: Type Reason Age From Message ---- ------ ---- ---- ------- Normal Updated 5m53s pomerium-ingress updated pomerium configuration Warning UpdateError 3s pomerium-ingress upsert routes: parsing ingress: annotations: applying policy annotations: parsing policy: invalid rules in policy: unsupported conditional "maybe", only and, or, not, nor and action are allowed
Copied!
# Redirect Loop with Redis Databroker
When using Redis, the shared secret is used to encrypt data in Redis itself. If you change the configured shared secret, data from Redis can no longer be decrypted. This results in errant behavior, including redirect loops when a user session cannot be retrieved from the databroker.
The resolution is to flush the Redis database with FLUSHDB
(opens new window) or FLUSHALL
(opens new window).
# RPC Errors
# certificate signed by unknown authority
When authenticating and authorizing a user for the first time, you may see the following in your Pomerium logs:
ERR http-error error="401 Unauthorized: ..... rpc error: code = DeadlineExceeded desc = latest connection error: connection error: desc = "transport: authentication handshake failed: x509: certificate signed by unknown authority...."
Copied!
Why
This error means that the proxy is rejecting the Authorize service's supplied certificate (used to establish a secure connection) because it doesn't know or trust the certificate authority that signed it.
Solution
Ensure that the Proxy service knows about and trusts the certificate authority that signed the Authorize service's certificate.
Add the certificate authority directly into Pomerium using the certificate authority config setting.
Add the certificate authority to the system's underlying trust store.
Replace your system / docker image certificate bundle.
For Docker:
COPY /etc/ssl/certs/your-cert-bundle.crt /etc/ssl/certs/ca-certificates.crt
Copied!Finally, ensure that you aren't being man-in-the-middle'd or that some eager router isn't injecting its own certificate along the way. Use openssl to verify that your Proxy service is getting the certificate you think its getting.
openssl s_client -servername pomerium.io -connect pomerium.io:443 </dev/null \ | sed -ne '/-BEGIN CERTIFICATE-/,/-END CERTIFICATE-/p'
Copied!
# rpc error: code = DeadlineExceeded
When authenticating and authorizing a user for the first time, you may get the following in your Pomerium logs.
{"level":"error",..."error":"rpc error: code = DeadlineExceeded desc = context deadline exceeded","http-message":"rpc error: code = DeadlineExceeded desc = context deadline exceeded","http-code":500,"message":"http-error"}
Copied!
Why
The Proxy service is not able to create a connection with the authorization service to validate a user.
Solution
Usually, this is the result of either a routing issue or a configuration error. Make sure that you are using the internally routable URL for the Authorize service. Many cloud loud balancers do not yet support gRPC transposing the ingress. So while your authenticate service url will probably look like https://authenticate.corp.example.com
, your authorizer service url will likely be more like https://pomerium-authorize-service.default.svc.cluster.local
or https://localhost:5443
.
# Pomerium Enterprise
# Generate Recovery Token
In the event that you lose access to the console via delegated access (the policy defined in Pomerium), there exists a fallback procedure to regain access to the console via a generated recovery token.
To generate a token, run the pomerium-console generate-recovery token
command with the following flags:
Flag | Description |
---|---|
--database-encryption-key | base64-encoded encryption key for encrypting sensitive data in the database. |
--database-url | The database to connect to (default "postgresql://pomerium:pomerium@localhost:5432/dashboard?sslmode=disable "). |
--namespace | The namespace to use (default "9d8dbd2c-8cce-4e66-9c1f-c490b4a07243 " for Global). |
--out | Where to save the JWT. If not specified, it will be printed to stdout. |
--ttl | The amount of time before the recovery token expires. Requires a unit (example: 30s , 5m ). |
TIP
You can run the pomerium-console
binary from any device with access to the database.
# Miscellaneous
# Invalid Certificates from Command Line Tools
When using Let's Encrypt certificates, you must use the fullchain.pem
file, not cert.pem
in order to include intermediate certs. Browsers like Chrome will store intermediate certs for LE but other tools (like curl
) don't, which is why your route might look fine in a web browser, but not when curl'd or used for TCP tunneling.
← Background Upgrading →