I don't think that we know the answer to this one yet.

Leaving aside the question of whether travel back in time violates any physical laws, there are some reasons to think that it is conceptually incoherent. For one thing, there is the well-known "Grandfather Paradox." Suppose you were to travel back in time and kill your own grandfather before the time your mother was conceived, thus preventing her conception and obviously your own as well. This leads to a logical puzzle, since you would cause something to happen that would make it the case that you had never existed.

It is very hard to tell a coherent time travel story. Science fiction representations of time travel often involve some kind of paradox. Very often, they fall victim to the "second time around" fallacy -- they depict events in some year, say 2005; our hero travels back in time and makes some crucial change; when he then returns to 2005 things are different. But this suggests that we have 2005-the-first-time-around and 2005-the-second-time-around, and that doesn't seem to make sense. How could 2005 occur twice? (Back to the Future depicts time travel this way.) One interesting recent depiction of time travel that avoids this mistake is the novel The Time Traveler's Wife.

There might be various ways to get around these puzzles. A good introduction to the philosophical puzzles of time travel is David Lewis's paper, "The paradoxes of time travel."

One last thought. A common objection to time travel that you often hear is that if time travel to the past were possible, then we should be surrounded now by time travelers. Some students at MIT recently tried to test this hypothesis by holding The Time Travelers Convention. There were not as many attendees as they had hoped.