I think i know what is the couse of your problems. The potmeter should have the output signal of the DAC on the leftmost pin, output to AMP middle (slider), and GND on rightmost pin. That way you will generate a voltage divider over the whole range. https://www.st-andrews.ac.uk/~www_pa/Scots_Guide/i...
When putting the potentiometer in the output line of an amp, you can overload the potentiometer. On higher audio levels with low impedence speakers, dead spots in the volume range can appear. Also Magic black smoke will appear on serious overloads. Iam sure placing it after the amp will be just fine...