If you want the code you change to actually run on the teensy then yes you have to press the arrow every time, once programmed it will run the code every boot. Have you installed the teensy drivers properly and read how to actually use it? You have to press the button on the teensy (double tap? I can't remember, google it) but there might be a little pop up that you need to look at? I'm not very familiar with teensy other than that sorry..
Oh also you could email where you got it from and explain what you have done, if the bootloader was corrupt the pressing the button isn't going to bring it back.. so it might be that (guessing here).. so best to ask the people you got it from.