Refactor to support the per-file-ignores

Most of the problems with our logic were due to not having the same
logic as our exclude parameter. This refactors that out into a separate
function so we can confidently achieve that.