At Verkada, a leader in cloud-managed physical security, we manage over 1 million devices for over 28,000 customers worldwide. Unlike legacy systems that are limited to local network access, Verkada’s devices communicate with our backend over the internet, enabling advanced features like remote monitoring and over-the-air updates.
However, this connectivity with our backend brings unique challenges: we must reliably distinguish legitimate device requests from malicious traffic – a task our authentication service handles as part of our microservices backend architecture. However, a concern of having a centralized authentication service is that if it ever goes down, we face a difficult choice: block incoming requests until it recovers - disrupting functionality - or allow unverified traffic, compromising security. Neither option is acceptable.
System availability is critical for all Verkada products, and especially for Verkada Alarms. Designed to detect, verify, and respond to threats in real-time, our alarm systems depend on uninterrupted uptime to effectively safeguard our customers and their facilities. To address the challenges introduced by the authentication service acting as a centralized dependency, the Alarms team explored an alternative device authentication flow. Could we decentralize device authentication to maintain availability even when the authentication service faces disruptions?
The team found a solution: Cryptographically signed auth tokens - implemented via JWTs (JSON Web Tokens) - which allow us to decouple device authentication from verifying individual backend requests. Let’s dive deeper!
Backend Architecture & the Authentication Bottleneck
In our previous authentication flow, devices initially fetch a long-lived authentication token, which they then include with every request to the backend. For instance, when an alarm panel needs to notify the backend of a potential threat, it attaches this token to its backend request. The alarms service, upon receiving the request, calls the authentication service to verify the token. If verified successfully, the alarms service proceeds to handle the request. However, if the authentication service becomes temporarily unavailable, device requests can no longer be processed.
Previous authentication flow: device requests require real-time verification from the authentication service, creating a single point of failure
A New Authentication Flow using Signed Auth Tokens
To address the limitations of our previous authentication flow, we needed a way for devices to authenticate themselves without relying on real-time verification from the authentication service. That way, devices could authenticate themselves when the authentication service is healthy and use this “proof” later, even if the service becomes unavailable.
Our solution: cryptographically signed authentication tokens. With signed tokens, we can decentralize request authentication across our backend, enabling product services to authenticate device requests independently.
Here’s how the new flow works:
Device Requests Auth Token: The device periodically contacts the authentication service to request an auth token, proving its identity using a unique secret embedded in the device during manufacturing.
Authentication Service Mints Auth Token: The authentication service mints a new token and signs it using its private key. This private key is stored in AWS KMS (Key Management Service) and is only accessible to the authentication service.
Device Sends Backend Request: After obtaining the signed auth token, the device can send requests to the alarms service, including the token with each request.
Backend Verifies Token Signature: The alarms service can verify the token’s signature using the corresponding public key - without having to contact the authentication service. The public key is retrieved from AWS KMS (99.99% availability guarantee!), and is cached locally for future use.
Backend Handles Request: If the token’s signature is valid, the Alarms service proceeds with handling the request.
New authentication flow: decentralized authentication of device requests using signed auth token
Industry Standard for Signed Auth Tokens: JWT
As part of the implementation of this new authentication flow, we adopted JSON Web Tokens (JWTs), a widely used industry standard for signed authentication tokens (RFC 7519). JWTs allow us to securely represent and verify device identity in a compact, portable format.
A JWT, represented as an ASCII string, is made up of three Base64-encoded sections:
Header: Specifies the token type and the signing algorithm used. For our tokens, we use ES256 (Elliptic Curve Digital Signature Algorithm with the P-256 curve and the SHA-256 hash function), an efficient and secure signing algorithm.
Payload: Contains the claims about the device's identity, such as:
subject: The unique device ID, tying the token specifically to that device.
expiration: A timestamp that limits the token’s validity period. This ensures that even if a token is ever compromised, it can only be used for a short amount of time.
Signature: A cryptographic signature generated using the authentication service’s private key stored securely in AWS KMS. The signature covers both the header and payload, making the token tamper-proof – any modification will invalidate the signature.
Anatomy of JWTs used in the new authentication flow
Impact of Signed Auth Tokens
By adopting signed auth tokens, the Alarms team achieved two major improvements to their device authentication flow:
Enhanced Availability and Reliability: By enabling the alarms service to authenticate incoming device requests independently, devices can continue interacting with our backend even during authentication service outages.
Reduced Load on the Authentication Service: Since product services can verify auth tokens without contacting the authentication service for each request, system traffic and processing overhead decreased significantly.
With JWT-based authentication, we’ve observed a significant improvement in system resilience and reliability. The chart below illustrates this impact: devices using the old authentication flow would experience an increased number of failures if the authentication infrastructure encountered issues, while those utilizing JWTs would operate with minimal disruption. This contrast demonstrates the effectiveness of JWTs in enhancing the authentication process.
Sampled traces of requests to a specific Alarms service endpoint without and with JWT-based authentication
Future-Ready Device Authentication Using JWTs
The shift to JWT-based device authentication has strengthened our system, improving availability, reducing dependency on real-time verification, and laying a foundation for future scalability. With this new flow, devices can remain operational and secure even during temporary authentication service outages, making our Alarms product more resilient and dependable.
At Verkada, we are committed to setting new standards in secure, reliable device management. As our portfolio grows, we will continue to innovate to improve our systems’ robustness and responsiveness, giving our customers the confidence that their security solutions will perform reliably when it matters most.