I heard the new speaker, ... (what/who/whose) was boring

Who

What refers to a thing or idea. Since you're actually trying to replace the name of a person you need the word that refers to a person. Who and Whose both refer to a person. However, whose asks for ownership. For example, whose shoes are in the hallway? In this sentence, you need the subject noun Who.