I was able to reproduce the behavior with a similar app.
The issue appears to be with the following part of your script:
kintone.app.record.set(record);
Your button is being added on the Record Details page using the app.record.detail.show event. While you can retrieve the record data and calculate the time difference there, kintone.app.record.set() does not save those changes back to the record from the Record Details page.
If you want to calculate the time when the button is clicked and save the result to the table, you can use the Update Record REST API instead.
This calculates the time difference, updates the Number field in each applicable table row through the Update Record REST API, and reloads the page so the saved values are displayed.
Also, this sample assumes that the start and end times are on the same day. If a task can span midnight, additional logic would be needed to handle that case.