TIL That if you keep your Photos for Mac library on an external drive that was formatted for Time Machine you’ll have issues with your library.

Thanks Apple Support for helping me out.