Rolling out access hours at Temporal

A few months ago we made a change to how engineers at Temporal access our infrastructure. However, it took some time before we realized just how much of an improvement the change to using Common Fate for Just in Time (JIT) had made to our security posture. JIT access means users have access only to what they request, when they request it, for only as long as they need it. It was in the process of writing up this blog post I realized there was a seemingly too-good-to-be-true metric: hours of access. Even without reducing the permissions a user has, there are still impressive security benefits to be had by limiting the amount of time a user has access to those permissions.

23.8%

If an employee works the standard eight-hour workday, that employee does not need any form of access for the other 16 hours of the day. (In some countries like Germany, to promote better work/life balance, employers even need to prevent email delivery outside of working hours.) Additionally, there are weekends to consider. Out of the 168 total hours in a week, an employee needs access for just 40 of them. If you do the math (really, if you get a calculator to do it for you), 40 hours out of 168 total hours in a week means only 23.8% of the time an employee needs access. I can say with fairly high confidence that in organizations I worked at before Temporal, I could leverage my AWS access 100% of the time. Nights, weekends, whenever. At Temporal, even if I requested all the access I could on every workday, my access would expose me and Temporal less than one-quarter of the time as compared to not using JIT access provisioning. Furthermore, the tokens used to authenticate to AWS will be different each day—if my system is compromised and today's access exfiltrated, an attacker must be capable of acting on that access within hours to pivot deeper.

Access is rare

Starting with the preceding figure, we know 23.8% of hours in the week are actually business hours when we might expect users to need access. Can we drive that down further? Let's set aside for the moment on-call rotations, incidents, and other abnormal situations. (If that sounds normal for your organization, consider this a reminder that no, actually, incidents should be abnormal.) Using some simple math, we can calculate a rough estimate of how many hours an employee actually needs access for, regardless of the level of that access, as an approximate upper bound.

Temporal currently has over 150 AWS accounts, a number which is increasing constantly to ensure strong protections against compromise, vulnerability, or accident. Although access to these accounts may be needed, those situations are relatively infrequent when viewed across all accounts. Let's say half of all engineers need access to half of all accounts for half of a day. Not even half of a working day (four hour), but for a full 12 hours: that gives 36,000 access hours. By implementing JIT, we are at just 12.5% of the number of access hours we were at previously. This can be reduced even further! If access is really only needed for half of the working day, that reduces the number of access hours to under 5% of the number of hours access used to be available for.

Why does it matter?

You can't give up something you don't have. Just like phishing-resistant forms of MFA, you can't be tricked into giving up access you never had. Even if a user is phished or has their device compromised, if they do not have access at the same time an attacker makes their way past the defenses in place, the attack won't be successful.

The concept of the attack chain gains an extra link, or if you're into dimensions, the attack chain expands into the temporal realm.

Plenty of tools exist to detect anomalous access, from unknown IP addresses, impossible travel, outside of normal hours, and a huge number of other signals. But these tools still have to extract a signal from background noise. If users have access 100% of the time, your detection and response capabilities must take into account the background noise which persistent access naturally generates.

When using JIT, that background noise is cut dramatically. Now, access to AWS accounts is only provisioned when a user needs it. Even this stage can be further de-risked by implementing approvals for levels of access. Perhaps an employee can leverage read-only access anytime, but if they wish to have access to modify infrastructure, a teammate or manager must approve. The signal-to-noise ratio is improved yet again, and the number of hours in which an attacker could modify your cloud environment is decreased further.

Recent Events

Our implementation of Just in Time access began before the LastPass breach was announced, where an employee's device was compromised outside of working hours. And to be clear, it is true that a persistent and dedicated attacker will not be stopped just by JIT access. This is not a magic cure for your security ills. But, reducing access to just what is needed when it is needed, will make an attacker's efforts harder and, hopefully, force them to be noisier— resulting in increased signal— due to the enforced deadline of credential expiration.

Need vs. Want

My advice to other organizations introducing JIT access is to begin slowly. There is already an ongoing push to eliminate IAM Users and move to IAM Roles. This is a strongly-recommended practice because it results in no static credentials to rotate, or accidentally check-in to a code repository. The next step is JIT access to those IAM Roles; in fact, if you are making the change from IAM Users to IAM Roles, you could skip straight to JIT access.

Regardless of where you are beginning, this can be a big shift in how users are accustom to working. While it may be tempting to jump straight into mandatory requests with approval, minimally scoped permissions, and maximum lengths measured in minutes, this will not go smoothly.

As a JIT roll-out begins, my suggestion is to give users what they want. If the default in your organization is Administrator Access, even if this is what access is requested for the full eight-hour workday, every day, without approval from a teammate or manager, there is still a significant change in security posture. The rough math above shows that users will still only be vulnerable for a fraction of the time they would otherwise have been at risk. If your organizational default is tighter in scope, you will see even more benefits from JIT access. For example, gaining insight into access patterns will provide further signals into what tasks should be automated, what permissions should be scoped tighter, or other controls to continuously improve your organization's security.