When we restart a container via podman restart or restart policy the
rootlessport process fails with `address already in use` because the
socketfile still exists.
This is a regression and was introduced in commit abdedc31a2.
Signed-off-by: Paul Holzinger <pholzing@redhat.com>