From Thiruvannamalai its hardly 100-110 kms to Pondicherry i guess.
Route had few potholes which can be easily managed.
This is the shortest route.
I have done the trip upto Thiruvannamalai but not to Pondicherry.
On this route Krishnagiri deviation towards left just besides the flyover to left.
Followed by 1st right leads to Thiruvannamalai.
You get a Ashram(Ramana Maharshi) i guess but not sure.
You can visit all these places and head towards pondicherry.
You can have a look on this thread for route upto Thiruvannamalai:My Visit to Golden Temple (Vellore) - Thiruvannamalai!