I am wondering if this is just a Raven quirk, or is there something wrong with my files?
I've experienced the same mismatch in Raven with my files, so I'm fairly confident that the issue you're experiencing at least in part a Raven quirk, but potentially also related to your dataset.
I believe that this quirk arises from the way that Raven handles recording timing, and particularly the conflict between timing the number of samples since start vs. reading date and time from the timestamps embedded in the file name.
Raven's ability to interpret the timestamps in file names is a fairly recent feature for this software (compared to the ability to load multiple files anyway). The main way that Raven keeps track of time for sequences of digital files is by counting samples for the nominal sample rate (i.e. the sample rate that comes embedded in the file metadata). If some of the wav files are incomplete, missing, or duty cycled, then this can cause a discrepancy between the clock times from the file names and the duration of recorded audio since the start.
This is just speculation on my part, and I've never tried to confirm this is really what's going on. I suppose it could also just be a bug in Raven.
Another workaround in addition to the one you suggested would be to navigate by choosing the file name instead of time. You can do this by selecting the file name from the File Drop-down instead of choosing the time (see red circled & highlighted bit of screenshot).
Of course the feasibility of this workaround depends on how many files you have and how long they are. I have mostly continuous recording with hour-long files, so this workaround works well for me, but if you've got much longer or much shorter files it might be too awkward.