Ensure that listener is removed only if it was added

Bug: 23016088
Change-Id: Ia6f17739ffc841ce8483e9c6c8132a94ecd62200
1 file changed