Freud-Bot Breakdown

Freud on Zoom [image]

Check here for the making of from the Realtime Character “Freud on Zoom)

Introduction

What it is?

Steirischer Herbst

Exhibition

Content

Chatbot  & QnA Maker > Answer / Multiple

Entity / Intent (Screenshot)

Material Collection: All Freud Books / Publications / Letters in German & as much as we could in english.

Jokes (about Psychology)

Standard casual answers

Freud specific & Art festival specific content

Content Crawl: I wrote a few small scripts that would extract the most used words in German Freud texts and removed the 10`000 most used German words I found online from this list to get a list of at least somewhat relevant Freud-specific words. From these words we selected roughly 500 good words and used those to go through the collected texts again and extract sentences where those words were present and that were in a specific length.

Prototype Bot Question Collection: Very early on we launched a webpage with a text-input version of the chatbot and asked colleagues, friends and family to start chatting with him. We used those questions to get a sense for what people might ask Freud and created a lot of relevant keyword/answers based on it. (Thanks to everyone who participated!)

GPT2: Early on we started experimenting with automatic text generation with a deeplearning network based on GPT2. The network would basically go through the collected texts and look for meaningful connection between words. We found out pretty quick that training a model from scratch and teach it German as well as all Freud specific content was a bigger task that originally anticipated. One model that was easy to retrain and was unfortunately only available in englisch and although we only had limited englisch Freud-specific content we used it to generate lots of sentences and translated it all back to german with machinelearning supported translation service Deepl.

voice

The Chatbot automatically synthesizes speech with a range of pretrained voices available in a wide range of languages and dialects. To have a voice of the chatbot that resembles the one from the actor (and the realtime character) we tried to train a neural network ourselves, made the actor read a bunch (1 hour worth) of letters from Freud, cut it down to small chunks with matching annotations to realize that we needed more (a lot more!) material to make it sound remotely natural. So we ditched this idea and resorted back to a pretrained voice with a slight austrian accent that we further modified with a software called MorphVox – more about this a bit later.

3D Character

Blendshapes

Groom

Body

Subtitles

hard to understand sometimes

OSC, no special characters possible, convert it

Split into parts

Overlay in Unreal, determine duration by character length

Animation

Basic Head and Eye Animation prerecorded

Lip movement automatically generated with Oculus Viseme Plugin

System diagramm

Statistics

Wake events during exhibition

Number of Questions

Duration of exhibition

Current location (Haus der Architektur, Graz/Austria until the end of 2020)

images

diverse Bilder und Screengrabs.

kurzes Video in Aktion

credits

Tobias Stärk – Lead Developer
Marc Pantenburg – 3D Artist
Simon Schmalfeld – Chatbot Integration
Tobias Wüstefeld – Creative Consultant