ssh access and key management for Nucleator Cages and Stacksets
About your Cage's bastion server and ssh communication
Benefits of the bastion communication architecture
In Nucleator Cages, all ssh communication to instances within the Cage must occur through the Cage's bastion instance. This provides a number of security benefits.
Benefits of Bastion Communications Architecture
- All ssh traffic can be consistently logged and forwarded as needed
- Provides a consistent deployment & data access method for audit and intrusion-detection workloads
- Makes both external and internal traffic available to above workloads
- Minimizes available external attack surface
Limits to ssh communication in Nucleator Cages
This architecture also limits the ways in which ssh communication can occur in Nucleator Cages:
- Security group setup enforces the constraint that all external ssh traffic must transit bastion. Except for bastion, it is not possible to ssh directly to instances within the Cage, not even to instances in the public subnet.
- Any required instance-to-instance ssh traffic must also transit bastion.
- Open inbound ports within both security groups and network ACLs are minimal. Because this is the case, by default, inbound ssh communications are only possible on port 22.
About your Cage's ssh keys
When and how are EC2 keypairs generated?
Nucleator creates an ec2 keypair for each AWS Account and Region in which a Nucleator Cage has been provisioned. It is important that keypairs are named consistently when generated, so that Nucleator can use the locate and use the correct keypair when it is interacting with instances in a specified Cage. Generated ec2 keypairs follow a uniform naming convention <customer>-<account>-<region>.pem
An ec2 keypair is generated at the time that a new Cage is provisioned, if no keypair exists with the above-described name. If an existing keypair already exists with that name, that keypair is used.
Where are ssh keys stored?
Whenever Nucleator creates a new EC2 keypair, the resulting private key is stored at ~/.nucleator/<keypair_name> on the instance where the nucleator command is running. This means that if you have not yet installed the builder Stackset in a Nucleator Cage and are running Nucleator from a local machine or from an EC2 instance in AWS outside of a Nucleator Cage, the resulting keys will be stored on your local machine or on that EC2 instance.
If you are using Nucleator through Nucleator's user interface, the resulting nucleator commands are running on the nucleator instance created by the builder stackset within your Customer's build Cage. ssh keys that are generated for new Nucleator Cages will be stored on the nucleator instance within your Customer's build Cage.
You should take care not to delete the generated ssh keys. If they are not available, you will not be able to access the instances that Nucleator has provisioned. Nucleator will also be unable to access the provisioned instances.
What if I need to use different keys for different Cages?
Nucleator currently uses the same ssh key to interact with all instances in the same AWS Region in the same AWS Account, no matter which Cages those instances are a part of. We are interested in feedback on this choice, it would be straightforward to support per-Cage ssh keys, but may also increase management complexity. We are also interested in feedback on ssh key rotation approaches. We are interested in additional, standard Cage operations that could be used to support seamless and safe specification and/or rotation of keys.
What if I need to add per-user ssh keys?
Nucleator does not currently help with this. But it could, we are interested in Community feedback regarding value and requirements.
Using your Cage's ssh-config to access instances via ssh
Customer- and Cage- specific ssh-configs
To make it easy for operators to access instances, even though all traffic must transit their containing Cage's bastion server, Nucleator generates an ssh config file that specifies the proper IdentityFile (ssh key) and configures an ssh ProxyCommand that allows you to use a single ssh command to tunnel through bastion to the desired instance.
Each time Nucleator provisions a Cage (including each time that any Stackset is applied within that Cage), Nucleator generates an ssh-config file that can be used to easily access any instance within the Cage. These ssh-config files are accessible at the following path on the instance that is running the nucleator command:
~/.nucleator/ssh-config/<customer>/<cage>
Instance naming conventions in ssh-config
The naming conventions for entries in the generated ssh-config makes it easy for you to use ssh to reach any of the instances in your Cage. You can use any of the following names to reach a desired instance:
| name type | syntax | examples |
|---|---|---|
| shorthand | <nucleator_role>-<cage> |
|
| private ip | <private_ip> | 172.32.4.83 |
| fqdn | <nucleator_role>-<cage>-<customer_domain> |
|
When an instance is part of an autoscaling group, it's name is not likely to be unique. In this case Nucleator adds the private ip to the fqdn notation to ensue uniqueness. You can still use the shorthand notation if you need to ssh to a representative instance within the autoscaling group, but don't particularly care which one.
Accessing instances within your Cage
You can use any of naming conventions described above for ssh config entries to easily ssh to any instance within your Cage:
ssh -F ~/.nucleator/ssh-config/<customer>/<cage> <instance>
Examples:
ssh -F ~/.nucleator/ssh-config/mycompany/build bastion-build ssh -F ~/.nucleator/ssh-config/mycompany/build 172.32.4.83 ssh -F ~/.nucelator/ssh-config/mycustomer/ci beanstalkworker.ci.department.mycustomer.com
Regenerating the ssh-config for your Cage
The ssh-config for a cage is regenerated automatically each time a cage configure command for that Cage is run, and any time that a stackset configure command is run for any Stackset within the Cage. This ensures the ability for all hosts within the Cage to be reached.
Copying generated ssh-configs to your local machine
You can use scp to copy the ssh-config to your local machine, if it was generated on a different host (e.g. on the nucleator instance within the builder stackset). You can use the ssh-config on your local host that was generated when configuring the builder Stackset to access the nucleator instance:
scp -F ~/.nucleator/ssh-config/mycustomer/build nucleator-build:~/.nucleator/ssh-config/mycustomer/build /tmp/build
Next: Changing Default Instance Types within Nucleator Cages and Stacksets