A potentiometer is just a resister that goes from 0 ohms to whatever maximum value it's rated. So what you have to make sure is that your resistors that are connected to the LED legs are high enough that it will be safe when the potentiometer is set at 0 ohms. The more resistance the dimmer the LED will be. My RGB LEDs are rated to run safety at (your RGB LED might be rated the the same or slightly different):
Red: 2.0 volts (20ma)
Green: 3.0 volts (20ma)
Blue 3.2 volts (20ma)
Assuming your power supply is 5.0 volts we can use an LED calculator to find out what minimum resistor we need:
http://ledcalc.com/#calc
Red: 150 ohms
Green: 100 ohms
Blue: 90 ohms
Now these are just the minimum resistors that you have to use for the LEDs to be safe. You can use larger value resistors but it makes the light dimmer.
Now for the potentiometers. You'll have to experiment to find how strong of a potentiometer you need for each color. In my RGB LED the Green color was a lot brighter than my Red and Blue. To make them all very dim I used the following resistance:
Red: 10k ohms
Green: 47k ohms
Blue: 20k ohms
If you want the LEDs to turn completely off you'll have to get potentiometers that are rated higher than that. Again, you'll have to experiment on your own there. Keep in mind it takes a lot of resistance to turn off an LED. Even my 1m (million) ohm resistor couldn't completely turn off my LED. You'll need some potentiometers with a lot of ohms.