did you make sure to record the original sounds in a really quiet environment? with no echoes etc.?
because once you've recorded those, voice-dialing on a busy street is possible too. it's just important that the first recording is perfect.
also, make sure you keep your voice steady. the phone doesn't know what you are saying, it's just comparing the sound-curve.
sorry if you already knew this and i did not help you at all, it's just what I think could be the problem since you said it works "sometimes".
am i dumb because I want a smartphone ?