By default, Remote Display only works on localhost / 127.0.0.1 and cannot be accessed by ip address or hostname.

Check VRDE / Remote Display IP Address
You can check VRDE / Remote Display ip address using the following methods:

Open command prompt and run netstat -an |find /i “listening” or netstat -an |find /i “[PORT_NUMBER]” and you shall notice it is listening on 127.0.0.1:PORT.

TCP 127.0.0.1:5940 0.0.0.0:0 LISTENING

Open command prompt, cd C:\Program Files\Oracle\VirtualBox and run

VBoxManage showvminfo "[VM_NAME]" | find /i "VRDE" 

and observe configuration of TCP/Address = “127.0.0.1”

VRDE property : TCP/Ports = "5940"
VRDE property : TCP/Address = "127.0.0.1"

Open [VM_NAME].vbox file in a text editor and search for

<VRDEProperties>
<Property name="TCP/Address" value="127.0.0.1"/>
<Property name="TCP/Ports" value="5940"/>
</VRDEProperties>

Modify VRDE / Remote Display IP Address

To allow VirtualBox VRDE / Remote Display to listen on Local Network IP Address, you can either run 

VBoxManage modifyvm "[VM_NAME]" --vrdeaddress 0.0.0.0

or edit [VM_NAME].vbox file TCP/Address to 0.0.0.0

And start your VM with Normal Start or Headless Start.