If you use primitive types for your Id's (int, long etc) ... then these beans can be removed from the persistence context.
This issue is to do with autoboxing creating Object instances as keys - and these keys are not referenced by any other object - specifically the beans that they relate to.
The fix was to change the persistence context to use a WeakValueMap taking the reference to the id out of the issue.