How to Gain Access to Backdoored Server?

Hi every one,

Last night, I scanned my server with nmap and found it was backdoored . I 've removed the backdoor and hardened my server. And an idea opens in my mind :): Someone (mr A) hacked into my server and installed a backdoor. An other guy (mr B) scans my server and found the backdoor like "31337/tcp open shell bind shell (*BACKDOOR*)". Is there anyway mr B can gain access to the backdoor was installed by mr A?