The legend says that Lord Rama performed the Ashwagandha Yagya in this place. A new temple has been built at this site with idols of Rama, Lakshman, Sita, Bharat, Shatrughan etc.
The temple restored the original idols carved out of single black sandstone, which were wrecked over the time.
This temple stands at the place where Rama is said to have performed the Ashvamedha Yajnya. About 300 years ago, the Raja of Kulu built a new temple here, which was improved by Ahalyabai Hokar of Indore during 1784, at the same time the adjoining Ghats were also built. The initial idols in black sandstone were recovered from Saryu and placed in the new temple, famous as Kaleram-ka-Mandir.