Abstract
Advancements in machine learning have recently enabled the hyper-realistic synthesis of prose, images, audio and video data, in what is referred to as artificial intelligence (AI)-generated media. These techniques offer novel opportunities for creating interactions with digital portrayals of individuals that can inspire and intrigue us. AI-generated portrayals of characters can feature synthesized faces, bodies and voices of anyone, from a fictional character to a historical figure, or even a deceased family member. Although negative use cases of this technology have dominated the conversation so far, in this Perspective we highlight emerging positive use cases of AI-generated characters, specifically in supporting learning and well-being. We demonstrate an easy-to-use AI character generation pipeline to enable such outcomes and discuss ethical implications as well as the need for including traceability to help maintain trust in the generated media. As we look towards the future, we foresee generative media as a crucial part of the ever growing landscape of human–AI interaction.
This is a preview of subscription content, access via your institution
Access options
Access Nature and 54 other Nature Portfolio journals
Get Nature+, our best-value online-access subscription
$29.99 / 30 days
cancel any time
Subscribe to this journal
Receive 12 digital issues and online access to articles
$119.00 per year
only $9.92 per issue
Buy this article
- Purchase on SpringerLink
- Instant access to full article PDF
Prices may be subject to local taxes which are calculated during checkout
Similar content being viewed by others
References
Boden, M. A. & Edmonds, E. A. What is generative art? Digital Creativity 20, 21–46 (2009).
Goodfellow, I. et al. Generative adversarial nets. In Advances in Neural Information Processing Systems 2672–2680 (NIPS, 2014).
Mirsky, Y. & Lee, W. The creation and detection of deepfakes: a survey. ACM Comput. Surveys 54, 1–41 (2021).
Karras, T. et al. Analyzing and improving the image quality of StyleGAN. In Proc. IEEE/CVF Conference on Computer Vision and Pattern Recognition 8110–8119 (IEEE, 2020).
Zhang, Y. et al. Learning to speak fluently in a foreign language: multilingual speech synthesis and cross-language voice cloning. Preprint at https://arxiv.org/abs/1907.04448 (2019).
Isola, P., Zhu, J.-Y., Zhou, T. & Efros, A. A. Image-to-image translation with conditional adversarial networks. In Proc. IEEE Conference on Computer Vision and Pattern Recognition 1125–1134 (IEEE, 2017).
Zhu, J., Shen, Y., Zhao, D. & Zhou, B. In-domain GAN inversion for real image editing. Preprint at https://arxiv.org/abs/2004.00049 (2020).
Ramesh, A. Zero-shot text-to-image generation. In Proc. 38th International Conference on Machine Learning Vol. 139, 8821–8831 (PMLR, 2021).
Takahashi, D. Deepdub uses AI to localize voice dubbing for foreign language films. Venture Beat (16 December 2020); https://venturebeat.com/2020/12/16/deepdub-uses-ai-to-localize-dubbing-for-foreign-language-films/
Dalí lives (via artificial intelligence) (Salvador Dali Museum, 2020); https://thedali.org/exhibit/dali-lives/
Westerlund, M. The emergence of deepfake technology: a review. Technol. Innov. Manag. Rev. 9, 40–53 (2019).
McCammon, M. N. in The Handbook of Communication Rights, Law and Ethics Ch. 24 (Wiley, 2021); https://doi.org/10.1002/9781119719564.ch24
ReFace. Swap. Share. Hype. https://reface.app/ (accessed 10 July 2020).
Pinscreen. The most advanced AI-driven virtual avatars. https://www.pinscreen.com/ (accessed 8 October 2020).
Emilia, P. Who is Lil Miquela, the digital avatar instagram influencer? https://www.thecut.com/2018/05/lil-miquela-digital-avatar-instagram-influencer.html (accessed 24 December 2020).
Prajwal, K. R., Mukhopadhyay, R., Namboodiri, V. P. & Jawahar, C. A lip sync expert is all you need for speech to lip generation in the wild. In Proc. 28th ACM International Conference on Multimedia 484-492 (ACM, 2020); https://doi.org/10.1145/3394171.3413532
Osimo, S. A., Pizarro, R., Spanlang, B. & Slater, M. Conversations between self and self as Sigmund Freud—a virtual body ownership paradigm for self counselling. Sci. Rep. 5, 13899 (2015).
Slater, M. et al. Virtually being Lenin enhances presence and engagement in a scene from the Russian revolution. Front. Robot. AI 5, 91 (2018).
Peck, T. C., Seinfeld, S., Aglioti, S. M. & Slater, M. Putting yourself in the skin of a black avatar reduces implicit racial bias. Conscious. Cogn. 22, 779–787 (2013).
Pataranutaporn, P., Vega Gálvez, T., Yoo, L., Chhetri, A. & Maes, P. Wearable wisdom: an intelligent audio-based system for mediating wisdom and advice. In Extended Abstracts of the 2020 CHI Conference on Human Factors in Computing Systems 1–8 (ACM, 2020); https://doi.org/10.1145/3334480.3383092
Soulmachines Inc. https://www.soulmachines.com/2018/08/meet-will-vectors-new-renewable-energy-educator-in-schools/ (accessed 25 April 2021).
Deci, E. L., Vallerand, R. J., Pelletier, L. G. & Ryan, R. M. Motivation and education: the self-determination perspective. Educational Psychol. 26, 325–346 (1991).
Skinner, E. A. & Belmont, M. J. Motivation in the classroom: reciprocal effects of teacher behavior and student engagement across the school year. J. Educ. Psychol. 85, 571–581 (1993).
Alavi, M. & Leidner, D. E. Research commentary: technology-mediated learning—a call for greater depth and breadth of research. Inf. Syst. Res. 12, 1–10 (2001).
Hudson, I. & Hurter, J. Avatar types matter: review of avatar literature for performance purposes. In Proc. International Conference on Virtual, Augmented and Mixed Reality 14–21 (Springer, 2016).
Kosmyna, N., Gross, A. & Maes, P. ‘The thinking cap 2.0’ preliminary study on fostering growth mindset of children by means of electroencephalography and perceived magic using artifacts from fictional sci-fi universes. In Proc. Interaction Design and Children Conference 458–469 (ACM, 2020).
Edwards, C. Male professor turns himself into anime schoolgirl to teach students remotely during coronavirus lockdown. The U.S. Sun (18 March 2020); https://www.the-sun.com/lifestyle/tech/556889/male-professor-turns-himself-into-anime-schoolgirl-to-teach-students-remotely-during-coronavirus-lockdown/
Kilteni, K., Bergstrom, I. & Slater, M. Drumming in immersive virtual reality: the body shapes the way we play. IEEE Trans. Vis. Comput. Graph. 19, 597–605 (2013).
Peck, T. C., Good, J. J. & Bourne, K. A. Inducing and mitigating stereotype threat through gendered virtual body-swap illusions. In Proc. 2020 CHI Conference on Human Factors in Computing Systems 1–13 (ACM, 2020).
Guegan, J., Buisine, S., Mantelet, F., Maranzana, N. & Segonds, F. Avatar-mediated creativity: when embodying inventors makes engineers more creative. Comput. Human Behav. 61, 165–175 (2016).
Banakou, D., Kishore, S. & Slater, M. Virtually being Einstein results in an improvement in cognitive task performance and a decrease in age bias. Front. Psychol. 9, 917 (2018).
Leong, J. et al. Exploring the use of real-time camera filters on embodiment and creativity. In Extended Abstracts of the 2021 CHI Conference on Human Factors in Computing Systems 1–7 (ACM, 2021).
Ames, C. Motivation: what teachers need to know. Teach. Coll. Rec. 91, 409–421 (1990).
Henderlong, J. & Lepper, M. R. The effects of praise on children’s intrinsic motivation: a review and synthesis. Psychol. Bull. 128, 774–795 (2002).
Bandura, A. Self-efficacy: toward a unifying theory of behavioral change. Psychol. Rev. 84, 191–215 (1977).
Shiomi, M., Okumura, S., Kimoto, M., Iio, T. & Shimohara, K. Two is better than one: social rewards from two agents enhance offline improvements in motor skills more than single agent. PLoS ONE 15, e0240622 (2020).
Kory-Westlund, J. M. & Breazeal, C. A long-term study of young children’s rapport, social emulation and language learning with a peer-like robot playmate in preschool. Front. Robot. AI 6, 81 (2019).
Musek, J. & Polic, M. in Encyclopedia of Quality of Life and Well-Being Research (ed. Michalos, A. C.) 4752–4755 (Springer, 2014).
National Institute of Mental Health. Mental illness; https://www.nimh.nih.gov/health/statistics/ (accessed 8 October 2021).
Torous, J. & Weiss Roberts, L. Needed innovation in digital health and smartphone applications for mental health transparency and trusts. JAMA Psychiatry. 74, 437–438 (2017).
Zhou, L., Gao, J., Li, D. & Shum, H.-Y. The design and implementation of Xiaoice, an empathetic social chatbot. Comput. Linguistics 46, 53–93 (2020).
Laranjo, L. et al. Conversational agents in healthcare: a systematic review. J. Am. Med. Inform. Assoc. 25, 1248–1258 (2018).
Gaffney, H., Mansell, W. & Tai, S. Conversational agents in the treatment of mental health problems: mixed-method systematic review. JMIR Mental Health 6, e14166 (2019).
Loveys, K., Fricchione, G., Kolappa, K., Sagar, M. & Broadbent, E. Reducing patient loneliness with artificial agents: design insights from evolutionary neuropsychiatry. J. Med. Internet Res. 21, e13664 (2019).
Johnson, L. A. & Caldwell, B. E. Race, gender and therapist confidence: effects on satisfaction with the therapeutic relationship in MFT. Am. J. Family Therapy 39, 307–324 (2011).
Banerjee, A. & Sanyal, D. Dynamics of doctor-patient relationship: a cross-sectional study on concordance, trust and patient enablement. J. Family Community Med. 19, 12–19 (2012).
Matsangidou, M. et al. ‘Now i can see me’ designing a multi-user virtual reality remote psychotherapy for body weight and shape concerns. Hum. Comput. Interact. https://doi.org/10.1080/07370024.2020.1788945 (2020).
Vaidyam, A. N., Wisniewski, H., Halamka, J. D., Kashavan, M. S. & Torous, J. B. Chatbots and conversational agents in mental health: a review of the psychiatric landscape. Can. J. Psychiatry 64, 456–464 (2019).
Carvalho, M. R. D., Freire, R. C. & Nardi, A. E. Virtual reality as a mechanism for exposure therapy. World J. Biol. Psychiatry 11, 220–230 (2010).
Hershfield, H. E. et al. Increasing saving behavior through age-progressed renderings of the future self. J. Mark. Res. 48, S23–S37 (2011).
Son, G.-R., Therrien, B. & Whall, A. Implicit memory and familiarity among elders with dementia. J. Nurs. Scholarsh. 34, 263–267 (2002).
Pagnini, F. et al. Ageing as a mindset: a study protocol to rejuvenate older adults with a counterclockwise psychological intervention. BMJ Open 9, e030411 (2019).
Massimi, M., Odom, W., Kirk, D. & Banks, R. HCI at the end of life: understanding death, dying and the digital. In CHI’10 Extended Abstracts on Human Factors in Computing Systems 4477–4480 (ACM, 2010).
Ohlheiser, A. The lonely reality of Zoom funerals. MIT Technology Review (13 April 2020); https://www.technologyreview.com/2020/04/13/999348/covid-19-grief-zoom-funerals/
Massimi, M. & Baecker, R. M. Dealing with death in design: developing systems for the bereaved. In Proc. SIGCHI Conference on Human Factors in Computing Systems 1001–1010 (ACM, 2011).
Newton, C. Speak, memory. The Verge (2016).
Hamilton, I. What is wisdom? Business Insider (17 November 2018); https://www.businessinsider.com/eternime-and-replika-giving-life-to-the-dead-with-new-technology-2018-11
Hayden, S. Mother meets recreation of her deceased child in VR https://www.roadtovr.com/mother-meets-recreation-of-deceased-child-in-vr/ (accessed 24 January 2021).
Brooker, C. & Harris, O. Be right back. Episode of Black Mirror (2013).
Villaronga, E. F. in Emotional Design in Human–Robot Interaction (eds Ayanoğlu, H. & Duarte, E.) 93–110 (Springer, 2019).
Kapur, A. et al. Non-invasive silent speech recognition in multiple sclerosis with dysphonia. In Proc. Machine Learning for Health Workshop 25–38 (PMLR, 2020).
Segalov, M. ‘i choose to thrive’: the man fighting motor neurone disease with cyborg technology. The Guardian (16 August 2021); https://www.theguardian.com/society/2020/aug/16/i-choose-to-thrive-the-man-fighting-motor-neurone-disease-with-cyborg-technology
Cudeiro, D., Bolkart, T., Laidlaw, C., Ranjan, A. & Black, M. J. Capture, learning and synthesis of 3D speaking styles. In Proc. IEEE Conference on Computer Vision and Pattern Recognition (CVPR) 10093–10103 (IEEE, 2019).
Li, T., Bolkart, T., Black, M. J., Li, H. & Romero, J. Learning a model of facial shape and expression from 4D scans. ACM Trans. Graph. 36, 194 (2017).
Vougioukas, K., Petridis, S. & Pantic, M. Realistic speech-driven facial animation with GANs. Int. J. Comput. Vis. 128, 1398–1413 (2020).
Siarohin, A., Lathuilière, S., Tulyakov, S., Ricci, E. & Sebe, N. First order motion model for image animation. In Advances in Neural Information Processing Systems 32 (eds. Wallach, H. et al.) 7137–7147 (Curran Associates, 2019).
Jemine, C. et al. Automatic Multispeaker Voice Cloning. MSc thesis, Université de Liège (2019).
Sutton, R. E. Image manipulation: then and now. In Selected Readings from the Symposium of the International Visual Literacy Association (1993).
Goodfellow, I. J., Shlens, J. & Szegedy, C. Explaining and harnessing adversarial examples. Preprint at https://arxiv.org/abs/1412.6572 (2014).
Rooney, B. Women and children first: technology and moral panic. The Wall Street Journal (11 June 2011); https://www.wsj.com/articles/BL-TEB-2814
Thompson, C. Texting isn’t the first new technology thought to impair social skills. Smithsonian Magazine (March 20116); https://www.smithsonianmag.com/innovation/texting-isnt-first-new-technology-thought-impair-social-skills-180958091/
Müller, V. C. in The Stanford Encyclopedia of Philosophy summer 2021 edn (ed. Zalta, E. N.) (Stanford Univ., 2021); https://plato.stanford.edu/archives/sum2021/entries/ethics-ai/
Langlois, S. ‘Donald Trump’ explains money laundering to his son-in-law in ‘deepfake’ video. Market Watch (19 September 2019); https://www.marketwatch.com/story/donald-trump-explains-money-laundering-to-his-son-in-law-in-deepfake-video-2019-09-18
Rajendra-Nicolucci, C. Language-generating A.I. is a free speech nightmare. Slate (30 September 2020); https://slate.com/technology/2020/09/language-ai-gpt-3-free-speech-harassment.html
Meskys, E., Kalpokiene, J., Jurcys, P. & Liaudanskas, A. Regulating deep fakes: legal and ethical considerations.J. Intellect. Prop. Law Pract. 15, 24–31 (2019).
Wagner, T. L. & Blewer, A. ‘The word real is no longer real’: deepfakes, gender, and the challenges of AI-altered video. Open Inf. Sci. 3, 32–46 (2019).
Fallis, D. The epistemic threat of deepfakes. Philos. Technol. https://doi.org/10.1007/s13347-020-00419-2 (2020).
In event of moon disaster (MIT Center For Advanced Virtuality, 2020); https://moondisaster.org
Ryan, E. The intersection of the Disney princess phenomenon and eating disorders. Response The Journal of Popular and American Culture https://responsejournal.net/issue/2016-08/article/intersection-disney-princess-phenomenon-and-eating-disorders (2016).
Burroughs, B. Youtube kids: the app economy and mobile parenting. Soc. Media Soc. https://doi.org/10.1177/2056305117707189 (2017).
Primack, B. A. et al. Social media use and perceived social isolation among young adults in the US. Am. J. Prev. Med. 53, 1–8 (2017).
Umberson, D. & Karas Montez, J. Social relationships and health: a flashpoint for health policy. J. Health Soc. Behav. 51, S54–S66 (2010).
Radesky, J. S. & Christakis, D. A. Increased screen time: implications for early childhood development and behavior. Pediatr. Clin. 63, 827–839 (2016).
Shin, H. & Ryan, A. M. Friend influence on early adolescent disruptive behavior in the classroom: teacher emotional support matters. Dev. Psychol. 53, 114–125 (2017).
Chiu, T. K. Student engagement in K-12 online learning amid COVID-19: a qualitative approach from a self-determination theory perspective. Interactive Learn. Environ. https://doi.org/10.1080/10494820.2021.1926289 (2021).
Abram, C. The most urgent threat of deepfakes isn’t politics. It’s porn. Vox (8 June 2020); https://www.vox.com/2020/6/8/21284005/urgent-threat-deepfakes-politics-porn-kristen-bell
Ankel, S. Many revenge porn victims consider suicide—why aren’t schools doing more to stop it. The Guardian (7 May 2018).
Kugler, M. B. & Pace, C. Deepfake privacy: attitudes and regulation. Northwestern Public Law Research Paper, SSRN 21-04 (2021); https://ssrn.com/abstract=3781968
Malaria Must Die. David Beckham launches the world’s first voice petition to end malaria https://malariamustdie.com/news/david-beckham-launches-worlds-first-voice-petition-end-malaria (accessed 15 July 2015).
Eshraghian, J. K. Human ownership of artificial creativity. Nat. Mach. Intell. 2, 157–160 (2020).
Baek, S. Free Speech in the Digital Age: Deepfakes and the Marketplace of Ideas. Honors theses (PPE), University of Pennsylvania. Penn Libraries (2020).
Fazio, L. K., Brashier, N. M., Payne, B. K. & Marsh, E. J. Knowledge does not protect against illusory truth. J. Exp. Psychol. Gen. 144, 993–1002 (2015).
Amershi, S. et al. Guidelines for human-AI interaction. In Proc. 2019 CHI Conference on Human Factors in Computing Systems 3, 1–13 (ACM, 2019).
Naruniec, J., Helminger, L., Schroers, C. & Weber, R. M. High-resolution neural face swapping for visual effects. In Proc. Computer Graphics Forum Vol. 39.4, 173–184 (Wiley, 2020).
Ramanathan, M., Mishra, N. & Thalmann, N. M. Nadine humanoid social robotics platform. In Proc. Computer Graphics International Conference 490–496 (Springer, 2019).
Claes, P. et al. Modeling 3D facial shape from DNA. PLoS Genet. 10, e1004224 (2014).
Zhu, W., Fan, X. & Zhang, Y. Applications and research trends of digital human models in the manufacturing industry. Virtual Reality Intell. Hardware 1, 558–579 (2019).
Our first artificial human. Samsung Neon https://www.neon.life/ (accessed 10 July 2020).
SoulMachines Inc. Baby X: soul machines. SoulMachines Inc. https://www.soulmachines.com/ (accessed 10 July 2020).
Corentin, J. Real-Time Voice Cloning. MSc thesis, Université de Liège (2019).
Author information
Authors and Affiliations
Contributions
P. Pataranutaporn developed the pipeline, assisted by V.D. The literature review was conducted by P. Pataranutaporn, V.D., J.L., P. Punpongsanon and M.S., who also contributed to the writing and editing of the manuscript. All other authors reviewed the manuscript. P. Pataranutaporn designed the figures. The pipeline was tested by D.N. The work was supervised by P.M. and M.S.
Corresponding author
Ethics declarations
Competing interests
The authors declare no competing interests.
Additional information
Publisher’s note Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
Rights and permissions
About this article
Cite this article
Pataranutaporn, P., Danry, V., Leong, J. et al. AI-generated characters for supporting personalized learning and well-being. Nat Mach Intell 3, 1013–1022 (2021). https://doi.org/10.1038/s42256-021-00417-9
Received:
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1038/s42256-021-00417-9
This article is cited by
-
Empowering generative AI through mobile edge computing
Nature Reviews Electrical Engineering (2024)
-
Human detection of political speech deepfakes across transcripts, audio, and video
Nature Communications (2024)
-
Behavioral health and generative AI: a perspective on future of therapies and patient care
npj Mental Health Research (2024)
-
A deepfake-based study on facial expressiveness and social outcomes
Scientific Reports (2024)
-
Understanding how personality traits, experiences, and attitudes shape negative bias toward AI-generated artworks
Scientific Reports (2024)