You could say either yes or no in that context, but it would be more clear to just answer affirmatively with “I’m afraid.”