The key vault is a location within the Azure platform that needs to have the proper controls in place in order to be protected. Compromise of a key results in an attacker gaining access to any resource associated to the key. One of the controls that can filter out unwanted connections is through the use of a firewall. This is disabled by default and should be one of this first considerations when setting up a key vault. Instructions to set this up can be found here:
https://docs.microsoft.com/en-us/azure/key-vault/general/network-security