What kind of software would you use to view and troubleshoot protocol-specific problems such as TCP retries caused by missed acknowledgements, or duplicate acknowledgements that cause network slow-down?

The kind of software to use is a protocol analyser.

Explanation:

The Protocol analyser, which could either be a hardware or a software is a tool that is used for capturing and analysing signals and also interpreting network traffic over a communication channel or between two or more connected computer systems

Protocol analysers can be used to find out location of problems on Local Area Network by examining flow of traffic.