Features for Apps

Unrupt

We encourage developers to evolve our Unrupt demo code to create Apps that can make calls so parties can speak their thoughts at the same time, a liberating experience. Existing Apps that already make voice or video calls such as Wire can add the unrupt features now and copy Tim's excellent logic from our open source prototype.

Sections of Tim's presentation

Adding unrupt to calls

Unrupting of calls can be added as an "alway-on" feature when debugged and stable in which case there is no extra user control needed. A status indicator to show that the other person is speaking while you are is useful.

In our Unrupt demo we have added a Pause button to the usual call controls and a Pause-while-speaking button which turns the unrupt feature of a call on or off. The latter can be added to a settings menu in an existing call service.

During a call in a production service we visualise a prominent Pause Button overlay. The manual use of the Pause button lets us focus on an interruption then catch up with what was said; on unpausing playback can be speeded up.

The Unrupt feature can be presented as automating a pause of incoming audio which could have been done with button presses to stop and start it instead of just speaking which makes the process so much more natural and usable.

Voicechat

Voice messaging provides the entry point for features we have pioneered in Vv Studio which extend voice messaging into "persistent intermitent searchable conversations" experienced like phone calls which we call "Voicechat".

Improvements we can show in Vv Studio let us annotate, tag and thereby search for voice messages in the chat list. We have tried sending voice messages to Internet services for transcription with variable results and would like to try that again now that the services are growing in accuracy. These improvements can be made in the message list and do not affect the recording which is controlled by the Mic button.

After prototyping voice messaging some years ago, a key improvement we discovered is in the use of the Mic Button and in how we create voice messages. After trying both Push-to-talk and Mic-on/off modes, we settled on the Mic-on/off toggle acting essentially as a Mute button. We added silence detection, really of a sound below a settable threshold for a settable duration, and created separate voice messages for each spoken utterance, omitting the silences.

As the Microphone is on until the Mic button is pressed and quickly released again to mute it, a series of voice messages can be recorded hands-free. Moreover, when voice messages are being played back, the voice detection can pause the playback while the listener is speaking. This enables a listener to speak replies as if on a live call, uncanny at first, but we soon get used to it as it feels so normal.

Mic Button

In Wire, touching the Mic symbol brings up a larger dialog box with at present just a Record button in it. Pressing the record button starts the recording and the button turns into a Stop button and pressing it again stops the recording and the dialog box dissapears. The recording is sent as a voice message and shown in the message list. Other services use a press to talk Mic button which is held while speaking.

To preserve existing Mic button behavior where voice messagng is already inplemented, there can be a settings toggle to the new Voicechat behaviour. Possible in an App such as Line for example; when the Mic button is pressed and held preserves the existing behavior of Line of recording a voice message, and if the Mic button is pressed and immediately released, the Voicechat behaviour is turned on and a message wll be recorded each time you speak until you turn it off again.

The Mic-on indicator is lit continualy and each utterance until a silence is detected is made into a buffered or recorded snippet which can be indicated visually by variations in the Record button. Each utterance is sent as a separate voice message as if the Mic button had been turned off. This means the user can now make several voice messages in a row hands-free.

Replies

As long as the device has echo cancelling or is using a headset, messages can be spoken in reply while audio is being played. Users can reply to voice messages as if they were on a call and playback is automatically paused during the reply. The precise moment at which playback triggered the reply is noted, so playback resumes there when the reply is finished.

This enables threaded speach messaging like a text message board but with both the added clarity and fullness of speach. Playback is like being in a live conversation that one can add to so it keeps growing and is always navigable to make sense of by playing back how it happened. Playback omits all periods of silence so that an intermittent conversation that may have accumulated over days while playing back feels like it is happening now.

When first entering a chat the user can immediatley click the Mick button to start speaking and it will be apparent from the chat list that he has not yet played any accumulated messages. If there are accumulated in-buffered messages, either they can be played individually as now, or there can be a Pause Button like the one recommended to add to live calls which is lit up with a Play invitation image.

When pressed this plays the accumulated messages in sequence; at any time the user may speak a reply and this is tagged as a reply to that mesage at that point so that playback can be in threaded mode.

Utterances can be:

  • New in the sequence of this conversation
  • Continuation in sequence by same speaker
  • Reply in context starting a new thread
  • Aside to the conversation starting or continuing another topic

Simple and obvious tags and visual confirmation of these types of utterance in a voice message and the links between utterances and their replies are prototyped in the threaded list display of Vv Studio and can be adapted.

If messages are streamed live as they are recorded, the voicechat experience when people are using voice messaging together, is like that of a live call. For discource such as think tank, voicechatting together works fine to capture all ideas and where they lead anyone.

For repartee back and forth such as wit and winning argument, people who find they are together in a voicechat can upgrade to a regular call. As we get used to replying when a thought is triggered in an intermittent voicechat conversation, we will expect it in live converations which the Unrupt feature then delivers.

I recommend a user choice by which the unrupt buffers can in a live call be saved to provide the mechanism for generating recordings of each speaker's utterances as they are spoken. Both voicechat and live calls can generate voice messages into the same chat list which then doubles as the recorded converation's playlist.

Asides

If the Mic is pressed and held during a live voicechat conversation [whether or not Mic already on] [or during a call if made available] and the existing individual voice message behavor is retained, this message can be tagged as an aside to the conversation, separate from the main flow.

Persistent Conversations

When speaking we generate a lot of utterances so the simple chat message list needs better navigation. Here is the timeline in an extended voicechat conversation using Verbol voice Studio. Each colored vertical bar corresponds to an utterance either created as a voice message or generated on a call by speach delimited by a short silence. Each participant has a color ringing their icon above the cursor, which displays the message number currently playing from the list, and the same color is used for that speaker's utterances in the timeline.