When you deploy a Build agent in VSTS / TFS, in the administration page you have the ability to add custom Capabilities to the agent, as you can see in Figure 1.
Figure 1: Adding capabilities to a standard build agent.
With the new Release Management, you can install agents in machine that will be added to Deployment Groups. If you look at the UI, you can see that the capabilities tab is listing all the capabilities of the agent, but you have not the option to specify custom capabilities. If you need to add some capabilities, as an example you want to add the VSTest capability because Test Agent was installed manually, you can simply add an Environment Variable in the machine. The agent will translate all the environment variables in Agent Capabilities for you.
As an example here is the result of a release
Environment variables added to the Machine will be added to the capabilities of the agent installed on that machine.
In that specific scenario I do not have vstest capabilities, even if Test runner was installed by the build with WinRm, so I simply added the environment variable in the machine.
Figure 2: VSTest environment variable added to the machine
Then restart the service of the agent and the new capabilities is now showing up in the summary.
Figure 3: VSTest capabilities now shows up in the agent capabilities list.
This will allow you to specify any capabilities you want in agents installed in machines on Deployment Groups for VSTS Release management.