Learning Loss from Heat