I drew a diagram to show how I wired things up, if needed I can post the prototype that I'm working on. Its quite messy though.
Here's the diagram:

I've confirmed that isn't because of the speaker, and neither the audio input. Also the power are functioning correctly as well.
I'm really confused now...
