The main annoyance with agent-based configuration management is the management of the agents themselves. They usually don't require any handholding after the initial install, except when you need to migrate them to a new environment. For example, migrate from a Puppet 5-based server to Puppet 6.
The migration process is not overly difficult, but in order to scale to more than a handful of nodes you will need to orchestrate the process. Historically we would have used Ansible for this task, but nowadays we prefer Puppet Bolt, which is more powerful and flexible in our use-cases. The Bolt plans typically end up being ugly, but based on my experience with Ansible Playbooks that tends to be the case anyways. I guess this is the natural result of mixing state-based configuration management, ad hoc commands and orchestration together.
Anyhow, here is the Bolt plan we wrote to do migration of a large population of nodes from Puppet 5 server to a Puppet 6 server:
The README.md tells how to use it, but here's a recap. This is for the worst case where you need to reach out to the agent nodes via public IP without any DNS and you need to manage a /etc/hosts entry:
$ bolt plan run puppetmaster_common::setup_puppet6_agent \ old_master_orchestration_address=<public-ip-of-old-master> \ master_address=<private-ip-of-new-master> \ master_orchestration_address=<public-ip-of-new-master> \ puppet_environment=production \ master_hostname=puppet6.example.org \ manage_host_entry=true \ ensure_puppet=stopped \ -u <ssh-username> \ --sudo-password <sudo-password> \ --run-as root \ -t <public-ip-of-the-targets>
You can define a single node or a group of nodes as the target (-t).
It is recommended to leave the Puppet 6 Agent stopped after the migration, so that you can do a --noop run against your desired branch, e.g. "production", before enabling it again on the agent nodes.
If all looks good just start and enable the agent. When all targets have systemd you can just do this:
$ bolt command run "systemctl enable puppet; systemctl start puppet" --run-as root --sudo-password-prompt -t <targets>