1. Check whether you are using Custom field or standard Contact field. 

2. If you are using Standard Contact field(CallerId), check the following in Related Record Configuration on the Lightning Record Page.

Also, using the following SOQLs, check whether the Contact Id associated to the CallerId is correct.

a. Fetch the CallerId.
SELECT Id, CallerId, FromPhoneNumber
FROM VoiceCall

b. Fetch the ParticipantEntityId from the ConversationParticipant record. The ParticipantEntityId will be the Contact Id of the CallerId.
SELECT Id, ParticipantEntityId
FROM ConversationParticipant