They still shouldn't be touching the selector fork face with the gear or any part of it, indicating a misalignment in that fork. The gear can't do it as it runs perpendicular to it's own shaft.
The fork has to have angle on it other that 90 degree to the shaft it's sliding on to touch just at that point and give that wear mark, up near the pivot.
Something like a surface plate to hold the pads flat to, then check the fork shaft angle against vertical to see what you've got there.
It is quite normal to see forks with that wear mark if bent in most gearbox assembly.