This was a revelation that came to me when fixing Vagrant on my Fedora 34 laptop: Vagrant plugins seem to be just Ruby gems installed into an isolated runtime environment, with "vagrant plugin" ensuring that the gems are installed in the correct place.
The same gems that you can install with "vagrant plugiin install" can be found using vanilla gem command:
$ vagrant plugin list
rexml (3.2.5, global)
vagrant-hostmanager (1.8.9, global)
vagrant-vbguest (0.29.0, global)
winrm (2.3.6, global)
winrm-elevated (1.2.3, global)
winrm-fs (1.3.5, global)
$ gem info --remote rexml
*** REMOTE GEMS ***
rexml (3.2.5)
Author: Kouhei Sutou
Homepage: https://github.com/ruby/rexml
An XML toolkit for Ruby
$ gem info --remote vagrant-vbguest
*** REMOTE GEMS ***
vagrant-vbguest (0.29.0)
Author: Robert Schulze
Homepage: https://github.com/dotless-de/vagrant-vbguest
A Vagrant plugin to install the VirtualBoxAdditions into the guest
VM
$ gem list|grep vagrant
<no output>
On the remote side you can find all the familiar Vagrant plugins as gems, but vanilla gem won't find them installed locally, because they're in a different path.
This is essentially the same approach that Puppetlabs' official Puppet packages use: they create isolated Ruby runtimes for gems that Puppet and Puppetserver need. And it makes sense, because otherwise things would end up in a dependency hell.