Better handle label references that aren't actually references

Fixes #1690

See merge request !705