Technical Disclosure Commons Technical Disclosure Commons

Defensive Publications Series

19 Jan 2021

Automatic Timestamp Recalculation During Audio Transcript Automatic Timestamp Recalculation During Audio Transcript

Editing Editing

Hao Wu

Follow this and additional works at: https://www.tdcommons.org/dpubs_series

Recommended Citation Recommended Citation

Wu, Hao, "Automatic Timestamp Recalculation During Audio Transcript Editing", Technical Disclosure

Automatic Timestamp Recalculation During Audio Transcript Editing

ABSTRACT

Automatic speech-to-text software sometimes produces transcripts that include errors

which users correct manually, e.g., by inserting or deleting words. It is important that the edited

transcript bears word-level timestamps that are faithful to the raw (unedited) transcript. This

disclosure describes techniques that can be applied in real time to automatically and accurately

recalculate timestamps of words in a transcript even as a user makes edits to the transcript. With

immediate/ real-time timestamp recalculation as described herein, users can play a recording and

edit the corresponding transcript while simultaneously enjoying transcript-editing utilities like

karaoke-style highlighting; word, sentence, or paragraph insertion, replacement, or deletion;

● Speech-to-text

BACKGROUND

Automatic speech-to-text software, e.g., used for online or offline meetings or in other

contexts, sometimes produces transcripts that include errors. Users edit and correct transcripts for

various reasons, e.g., to have high-quality records; to reduce misunderstandings that may arise

from inaccuracy; to enable searches through transcripts; to restructure transcripts to make them

easy to read, to quote, or refer to; to enable accurate sharing of information; to generally parse

paragraph timestamps (104a).

In Fig. 1(b), as the audio plays, the user introduces an edit, e.g., the insertion of a

paragraph-break. The newly-formed paragraph is assigned its own paragraph-level timestamp

(104b) based on word-level timestamps. The word-level timestamps remain invisible to the user.

In Fig. 1(c), as the audio continues playing, the user makes another edit (106a), e.g., the

relabeling of the speaker of the second paragraph. Fig. 1(d) shows the paragraph with the re-

labeled speaker (106b). Even as the audio continues playing and the user simultaneously makes

edits, the currently played word is highlighted karaoke-style (108, pink).

As shown in Fig. 2, speech-to-text software generally annotates each word in a transcript

A transcript that is edited by the user may include insertions or deletions. For transcript-

editing features to continue to work, it is important that the edited transcript bears word-level

timestamps that are faithful to the raw (unedited) transcript. Specifically, the timestamps of the

new (inserted) words must lie on a continuum between original words at the ends of an inserted

string, the deletion of words must not introduce timestamp inaccuracy, and timestamps must

monotonically increase with the text in the edited transcript.

Although there are transcript-editors that enable users to modify timestamps, such editors

do not automatically recalculate word timestamps, or even display the timestamps on the UI. As

a result, the user has to resort to manually associating sentences or chunks of text to parts of the

recording by looking at and replaying chunks of the audio. It is important to reduce or eliminate

the burden of assigning timestamps to edited words that falls on the user who edits the transcript.

corresponding transcript while simultaneously enjoying transcript-editing utilities like karaoke-

style highlighting; word, sentence, or paragraph insertion, replacement, or deletion; paragraph

(or other) break insertion; paragraph (or other) merging; playing or sharing selected audio

segments (sentences, paragraphs, etc.) of the transcript; etc.

As illustrated in Fig. 3, raw and edited transcripts are compared to determine the longest

common subsequences (LCS) between them (302). LCS determination is cast as a dynamic

programming problem whose complexity is reduced from exponential to polynomial using

windowing, memoization (304), and pruning (306). Once LCS is determined between raw and

2

denoted by C(s

1

, s

2

). For example,

C(‘and yet this mottled dawn’, ‘mottled dawn is unlike our dream’) = 2,

since the longest common subsequence between the input strings is ‘to be not to’ (of length 4),

obtained from the first argument as ‘to be or not to’ and from the second argument as ‘to or be

not is to.’

Given an unedited string s

1

of length n + 1 words and an edited string s

words, the following recursion holds.

C( s

C( s

[0 … n−1], s

[0 … m−1] )

1

2

1

) ),

1

[n] ≠ s

[m]

The above recursion sets up the determination of the LCS between strings s

1

and s

2

as a dynamic

programming problem, which, in its raw form, however, is exponential in complexity.

To reduce the complexity of LCS computation, the sequences s

1

and s

2

dimensional matrix. In this context, state refers to C(s

1

n]. The row-header of the matrix is the unedited transcript, e.g., ABCDA, and the column-header

is the edited transcript, e.g., ACBDEA (where A, B, …, E represent words). An entry represents

an LCS solution for some truncated s

. The final LCS between s

1

and s

2

backtracking along the pink pathway: the LCS sequence is represented by the dashed red boxes.

For example, the longest common subsequences at various time-points in the matrix are

as follows.

Pruning

Pruning builds on the observation that at every step the recursion increases the length of

the longest common subsequence by at most one. Thus, the maximum value in row j can be at