Freud on Zoom [image]
Check here for the making of from the Realtime Character “Freud on Zoom)
What it is?
Chatbot & QnA Maker > Answer / Multiple
Entity / Intent (Screenshot)
Material Collection: All Freud Books / Publications / Letters in German & as much as we could in english.
Jokes (about Psychology)
Standard casual answers
Freud specific & Art festival specific content
Content Crawl: I wrote a few small scripts that would extract the most used words in German Freud texts and removed the 10`000 most used German words I found online from this list to get a list of at least somewhat relevant Freud-specific words. From these words we selected roughly 500 good words and used those to go through the collected texts again and extract sentences where those words were present and that were in a specific length.
Prototype Bot Question Collection: Very early on we launched a webpage with a text-input version of the chatbot and asked colleagues, friends and family to start chatting with him. We used those questions to get a sense for what people might ask Freud and created a lot of relevant keyword/answers based on it. (Thanks to everyone who participated!)
GPT2: Early on we started experimenting with automatic text generation with a deeplearning network based on GPT2. The network would basically go through the collected texts and look for meaningful connection between words. We found out pretty quick that training a model from scratch and teach it German as well as all Freud specific content was a bigger task that originally anticipated. One model that was easy to retrain and was unfortunately only available in englisch and although we only had limited englisch Freud-specific content we used it to generate lots of sentences and translated it all back to german with machinelearning supported translation service Deepl.
The Chatbot automatically synthesizes speech with a range of pretrained voices available in a wide range of languages and dialects. To have a voice of the chatbot that resembles the one from the actor (and the realtime character) we tried to train a neural network ourselves, made the actor read a bunch (1 hour worth) of letters from Freud, cut it down to small chunks with matching annotations to realize that we needed more (a lot more!) material to make it sound remotely natural. So we ditched this idea and resorted back to a pretrained voice with a slight austrian accent that we further modified with a software called MorphVox – more about this a bit later.
hard to understand sometimes
OSC, no special characters possible, convert it
Split into parts
Overlay in Unreal, determine duration by character length
Basic Head and Eye Animation prerecorded
Lip movement automatically generated with Oculus Viseme Plugin
Wake events during exhibition
Number of Questions
Duration of exhibition
Current location (Haus der Architektur, Graz/Austria until the end of 2020)
diverse Bilder und Screengrabs.
kurzes Video in Aktion
Tobias Stärk – Lead Developer
Marc Pantenburg – 3D Artist
Simon Schmalfeld – Chatbot Integration
Tobias Wüstefeld – Creative Consultant