The authority of St Peter is not about having the authority to have everybody to serve him. It is an authority to serve others. This is what the Church was established for and strives to do.
[youtube https://www.youtube.com/watch?v=WJXR-TuImMs?list=UUZV9Z_SkvSeYobOYMfSSd7A]