I switched over to a 3 pin pot, and switched the wiring around to match your advice. I'm still not getting audio out of the speaker though when the headphones are unplugged. I tried attaching the speaker to 1 pin and 2 pins. If you have any other suggestions I'd love to hear them. At this point I'm ...

Thanks Jake! Appreciate the detailed reply. I have a 3 pin potentiometer as well, should I use that instead?
Here's a new diagram (hopefully following your advice) let me know what you think. I'll try it out when I get home tonight.

This is my first build, and for some reason I decided to use the Adafruit MAX98357A combo DAC/AMP instead of the usual PAM8302 setup... I have it mostly working, but when I wire everything up through the potentiometer and headphone jack I can only get sound through the headphones. I've tried a bunch...