Skip to main content

Thank you for visiting nature.com. You are using a browser version with limited support for CSS. To obtain the best experience, we recommend you use a more up to date browser (or turn off compatibility mode in Internet Explorer). In the meantime, to ensure continued support, we are displaying the site without styles and JavaScript.

  • Perspective
  • Published:

AI-generated characters for supporting personalized learning and well-being

Abstract

Advancements in machine learning have recently enabled the hyper-realistic synthesis of prose, images, audio and video data, in what is referred to as artificial intelligence (AI)-generated media. These techniques offer novel opportunities for creating interactions with digital portrayals of individuals that can inspire and intrigue us. AI-generated portrayals of characters can feature synthesized faces, bodies and voices of anyone, from a fictional character to a historical figure, or even a deceased family member. Although negative use cases of this technology have dominated the conversation so far, in this Perspective we highlight emerging positive use cases of AI-generated characters, specifically in supporting learning and well-being. We demonstrate an easy-to-use AI character generation pipeline to enable such outcomes and discuss ethical implications as well as the need for including traceability to help maintain trust in the generated media. As we look towards the future, we foresee generative media as a crucial part of the ever growing landscape of human–AI interaction.

This is a preview of subscription content, access via your institution

Access options

Buy this article

Prices may be subject to local taxes which are calculated during checkout

Fig. 1: Example applications of AI-generated characters that resemble contemporary and historical figures.
Fig. 2: AI-generated characters as a domain can be characterized along three axes: realism, embodiment and interactivity.
Fig. 3: Our unified pipeline allows users to provide video, voice or text as inputs to generate videos and real-time facial filters.
Fig. 4: Using AI-generated characters in a virtual classroom setting.

Similar content being viewed by others

References

  1. Boden, M. A. & Edmonds, E. A. What is generative art? Digital Creativity 20, 21–46 (2009).

    Article  Google Scholar 

  2. Goodfellow, I. et al. Generative adversarial nets. In Advances in Neural Information Processing Systems 2672–2680 (NIPS, 2014).

  3. Mirsky, Y. & Lee, W. The creation and detection of deepfakes: a survey. ACM Comput. Surveys 54, 1–41 (2021).

    Article  Google Scholar 

  4. Karras, T. et al. Analyzing and improving the image quality of StyleGAN. In Proc. IEEE/CVF Conference on Computer Vision and Pattern Recognition 8110–8119 (IEEE, 2020).

  5. Zhang, Y. et al. Learning to speak fluently in a foreign language: multilingual speech synthesis and cross-language voice cloning. Preprint at https://arxiv.org/abs/1907.04448 (2019).

  6. Isola, P., Zhu, J.-Y., Zhou, T. & Efros, A. A. Image-to-image translation with conditional adversarial networks. In Proc. IEEE Conference on Computer Vision and Pattern Recognition 1125–1134 (IEEE, 2017).

  7. Zhu, J., Shen, Y., Zhao, D. & Zhou, B. In-domain GAN inversion for real image editing. Preprint at https://arxiv.org/abs/2004.00049 (2020).

  8. Ramesh, A. Zero-shot text-to-image generation. In Proc. 38th International Conference on Machine Learning Vol. 139, 8821–8831 (PMLR, 2021).

  9. Takahashi, D. Deepdub uses AI to localize voice dubbing for foreign language films. Venture Beat (16 December 2020); https://venturebeat.com/2020/12/16/deepdub-uses-ai-to-localize-dubbing-for-foreign-language-films/

  10. Dalí lives (via artificial intelligence) (Salvador Dali Museum, 2020); https://thedali.org/exhibit/dali-lives/

  11. Westerlund, M. The emergence of deepfake technology: a review. Technol. Innov. Manag. Rev. 9, 40–53 (2019).

    Article  Google Scholar 

  12. McCammon, M. N. in The Handbook of Communication Rights, Law and Ethics Ch. 24 (Wiley, 2021); https://doi.org/10.1002/9781119719564.ch24

  13. ReFace. Swap. Share. Hype. https://reface.app/ (accessed 10 July 2020).

  14. Pinscreen. The most advanced AI-driven virtual avatars. https://www.pinscreen.com/ (accessed 8 October 2020).

  15. Emilia, P. Who is Lil Miquela, the digital avatar instagram influencer? https://www.thecut.com/2018/05/lil-miquela-digital-avatar-instagram-influencer.html (accessed 24 December 2020).

  16. Prajwal, K. R., Mukhopadhyay, R., Namboodiri, V. P. & Jawahar, C. A lip sync expert is all you need for speech to lip generation in the wild. In Proc. 28th ACM International Conference on Multimedia 484-492 (ACM, 2020); https://doi.org/10.1145/3394171.3413532

  17. Osimo, S. A., Pizarro, R., Spanlang, B. & Slater, M. Conversations between self and self as Sigmund Freud—a virtual body ownership paradigm for self counselling. Sci. Rep. 5, 13899 (2015).

    Article  Google Scholar 

  18. Slater, M. et al. Virtually being Lenin enhances presence and engagement in a scene from the Russian revolution. Front. Robot. AI 5, 91 (2018).

    Article  Google Scholar 

  19. Peck, T. C., Seinfeld, S., Aglioti, S. M. & Slater, M. Putting yourself in the skin of a black avatar reduces implicit racial bias. Conscious. Cogn. 22, 779–787 (2013).

    Article  Google Scholar 

  20. Pataranutaporn, P., Vega Gálvez, T., Yoo, L., Chhetri, A. & Maes, P. Wearable wisdom: an intelligent audio-based system for mediating wisdom and advice. In Extended Abstracts of the 2020 CHI Conference on Human Factors in Computing Systems 1–8 (ACM, 2020); https://doi.org/10.1145/3334480.3383092

  21. Soulmachines Inc. https://www.soulmachines.com/2018/08/meet-will-vectors-new-renewable-energy-educator-in-schools/ (accessed 25 April 2021).

  22. Deci, E. L., Vallerand, R. J., Pelletier, L. G. & Ryan, R. M. Motivation and education: the self-determination perspective. Educational Psychol. 26, 325–346 (1991).

    Article  Google Scholar 

  23. Skinner, E. A. & Belmont, M. J. Motivation in the classroom: reciprocal effects of teacher behavior and student engagement across the school year. J. Educ. Psychol. 85, 571–581 (1993).

    Article  Google Scholar 

  24. Alavi, M. & Leidner, D. E. Research commentary: technology-mediated learning—a call for greater depth and breadth of research. Inf. Syst. Res. 12, 1–10 (2001).

    Article  Google Scholar 

  25. Hudson, I. & Hurter, J. Avatar types matter: review of avatar literature for performance purposes. In Proc. International Conference on Virtual, Augmented and Mixed Reality 14–21 (Springer, 2016).

  26. Kosmyna, N., Gross, A. & Maes, P. ‘The thinking cap 2.0’ preliminary study on fostering growth mindset of children by means of electroencephalography and perceived magic using artifacts from fictional sci-fi universes. In Proc. Interaction Design and Children Conference 458–469 (ACM, 2020).

  27. Edwards, C. Male professor turns himself into anime schoolgirl to teach students remotely during coronavirus lockdown. The U.S. Sun (18 March 2020); https://www.the-sun.com/lifestyle/tech/556889/male-professor-turns-himself-into-anime-schoolgirl-to-teach-students-remotely-during-coronavirus-lockdown/

  28. Kilteni, K., Bergstrom, I. & Slater, M. Drumming in immersive virtual reality: the body shapes the way we play. IEEE Trans. Vis. Comput. Graph. 19, 597–605 (2013).

    Article  Google Scholar 

  29. Peck, T. C., Good, J. J. & Bourne, K. A. Inducing and mitigating stereotype threat through gendered virtual body-swap illusions. In Proc. 2020 CHI Conference on Human Factors in Computing Systems 1–13 (ACM, 2020).

  30. Guegan, J., Buisine, S., Mantelet, F., Maranzana, N. & Segonds, F. Avatar-mediated creativity: when embodying inventors makes engineers more creative. Comput. Human Behav. 61, 165–175 (2016).

    Article  Google Scholar 

  31. Banakou, D., Kishore, S. & Slater, M. Virtually being Einstein results in an improvement in cognitive task performance and a decrease in age bias. Front. Psychol. 9, 917 (2018).

    Article  Google Scholar 

  32. Leong, J. et al. Exploring the use of real-time camera filters on embodiment and creativity. In Extended Abstracts of the 2021 CHI Conference on Human Factors in Computing Systems 1–7 (ACM, 2021).

  33. Ames, C. Motivation: what teachers need to know. Teach. Coll. Rec. 91, 409–421 (1990).

    Article  Google Scholar 

  34. Henderlong, J. & Lepper, M. R. The effects of praise on children’s intrinsic motivation: a review and synthesis. Psychol. Bull. 128, 774–795 (2002).

    Article  Google Scholar 

  35. Bandura, A. Self-efficacy: toward a unifying theory of behavioral change. Psychol. Rev. 84, 191–215 (1977).

    Article  Google Scholar 

  36. Shiomi, M., Okumura, S., Kimoto, M., Iio, T. & Shimohara, K. Two is better than one: social rewards from two agents enhance offline improvements in motor skills more than single agent. PLoS ONE 15, e0240622 (2020).

    Article  Google Scholar 

  37. Kory-Westlund, J. M. & Breazeal, C. A long-term study of young children’s rapport, social emulation and language learning with a peer-like robot playmate in preschool. Front. Robot. AI 6, 81 (2019).

    Article  Google Scholar 

  38. Musek, J. & Polic, M. in Encyclopedia of Quality of Life and Well-Being Research (ed. Michalos, A. C.) 4752–4755 (Springer, 2014).

  39. National Institute of Mental Health. Mental illness; https://www.nimh.nih.gov/health/statistics/ (accessed 8 October 2021).

  40. Torous, J. & Weiss Roberts, L. Needed innovation in digital health and smartphone applications for mental health transparency and trusts. JAMA Psychiatry. 74, 437–438 (2017).

    Article  Google Scholar 

  41. Zhou, L., Gao, J., Li, D. & Shum, H.-Y. The design and implementation of Xiaoice, an empathetic social chatbot. Comput. Linguistics 46, 53–93 (2020).

    Article  Google Scholar 

  42. Laranjo, L. et al. Conversational agents in healthcare: a systematic review. J. Am. Med. Inform. Assoc. 25, 1248–1258 (2018).

    Article  Google Scholar 

  43. Gaffney, H., Mansell, W. & Tai, S. Conversational agents in the treatment of mental health problems: mixed-method systematic review. JMIR Mental Health 6, e14166 (2019).

    Article  Google Scholar 

  44. Loveys, K., Fricchione, G., Kolappa, K., Sagar, M. & Broadbent, E. Reducing patient loneliness with artificial agents: design insights from evolutionary neuropsychiatry. J. Med. Internet Res. 21, e13664 (2019).

    Article  Google Scholar 

  45. Johnson, L. A. & Caldwell, B. E. Race, gender and therapist confidence: effects on satisfaction with the therapeutic relationship in MFT. Am. J. Family Therapy 39, 307–324 (2011).

    Article  Google Scholar 

  46. Banerjee, A. & Sanyal, D. Dynamics of doctor-patient relationship: a cross-sectional study on concordance, trust and patient enablement. J. Family Community Med. 19, 12–19 (2012).

    Article  Google Scholar 

  47. Matsangidou, M. et al. ‘Now i can see me’ designing a multi-user virtual reality remote psychotherapy for body weight and shape concerns. Hum. Comput. Interact. https://doi.org/10.1080/07370024.2020.1788945 (2020).

  48. Vaidyam, A. N., Wisniewski, H., Halamka, J. D., Kashavan, M. S. & Torous, J. B. Chatbots and conversational agents in mental health: a review of the psychiatric landscape. Can. J. Psychiatry 64, 456–464 (2019).

    Article  Google Scholar 

  49. Carvalho, M. R. D., Freire, R. C. & Nardi, A. E. Virtual reality as a mechanism for exposure therapy. World J. Biol. Psychiatry 11, 220–230 (2010).

    Article  Google Scholar 

  50. Hershfield, H. E. et al. Increasing saving behavior through age-progressed renderings of the future self. J. Mark. Res. 48, S23–S37 (2011).

    Article  Google Scholar 

  51. Son, G.-R., Therrien, B. & Whall, A. Implicit memory and familiarity among elders with dementia. J. Nurs. Scholarsh. 34, 263–267 (2002).

    Article  Google Scholar 

  52. Pagnini, F. et al. Ageing as a mindset: a study protocol to rejuvenate older adults with a counterclockwise psychological intervention. BMJ Open 9, e030411 (2019).

    Article  Google Scholar 

  53. Massimi, M., Odom, W., Kirk, D. & Banks, R. HCI at the end of life: understanding death, dying and the digital. In CHI10 Extended Abstracts on Human Factors in Computing Systems 4477–4480 (ACM, 2010).

  54. Ohlheiser, A. The lonely reality of Zoom funerals. MIT Technology Review (13 April 2020); https://www.technologyreview.com/2020/04/13/999348/covid-19-grief-zoom-funerals/

  55. Massimi, M. & Baecker, R. M. Dealing with death in design: developing systems for the bereaved. In Proc. SIGCHI Conference on Human Factors in Computing Systems 1001–1010 (ACM, 2011).

  56. Newton, C. Speak, memory. The Verge (2016).

  57. Hamilton, I. What is wisdom? Business Insider (17 November 2018); https://www.businessinsider.com/eternime-and-replika-giving-life-to-the-dead-with-new-technology-2018-11

  58. Hayden, S. Mother meets recreation of her deceased child in VR https://www.roadtovr.com/mother-meets-recreation-of-deceased-child-in-vr/ (accessed 24 January 2021).

  59. Brooker, C. & Harris, O. Be right back. Episode of Black Mirror (2013).

  60. Villaronga, E. F. in Emotional Design in Human–Robot Interaction (eds Ayanoğlu, H. & Duarte, E.) 93–110 (Springer, 2019).

  61. Kapur, A. et al. Non-invasive silent speech recognition in multiple sclerosis with dysphonia. In Proc. Machine Learning for Health Workshop 25–38 (PMLR, 2020).

  62. Segalov, M. ‘i choose to thrive’: the man fighting motor neurone disease with cyborg technology. The Guardian (16 August 2021); https://www.theguardian.com/society/2020/aug/16/i-choose-to-thrive-the-man-fighting-motor-neurone-disease-with-cyborg-technology

  63. Cudeiro, D., Bolkart, T., Laidlaw, C., Ranjan, A. & Black, M. J. Capture, learning and synthesis of 3D speaking styles. In Proc. IEEE Conference on Computer Vision and Pattern Recognition (CVPR) 10093–10103 (IEEE, 2019).

  64. Li, T., Bolkart, T., Black, M. J., Li, H. & Romero, J. Learning a model of facial shape and expression from 4D scans. ACM Trans. Graph. 36, 194 (2017).

    Article  Google Scholar 

  65. Vougioukas, K., Petridis, S. & Pantic, M. Realistic speech-driven facial animation with GANs. Int. J. Comput. Vis. 128, 1398–1413 (2020).

    Article  Google Scholar 

  66. Siarohin, A., Lathuilière, S., Tulyakov, S., Ricci, E. & Sebe, N. First order motion model for image animation. In Advances in Neural Information Processing Systems 32 (eds. Wallach, H. et al.) 7137–7147 (Curran Associates, 2019).

  67. Jemine, C. et al. Automatic Multispeaker Voice Cloning. MSc thesis, Université de Liège (2019).

  68. Sutton, R. E. Image manipulation: then and now. In Selected Readings from the Symposium of the International Visual Literacy Association (1993).

  69. Goodfellow, I. J., Shlens, J. & Szegedy, C. Explaining and harnessing adversarial examples. Preprint at https://arxiv.org/abs/1412.6572 (2014).

  70. Rooney, B. Women and children first: technology and moral panic. The Wall Street Journal (11 June 2011); https://www.wsj.com/articles/BL-TEB-2814

  71. Thompson, C. Texting isn’t the first new technology thought to impair social skills. Smithsonian Magazine (March 20116); https://www.smithsonianmag.com/innovation/texting-isnt-first-new-technology-thought-impair-social-skills-180958091/

  72. Müller, V. C. in The Stanford Encyclopedia of Philosophy summer 2021 edn (ed. Zalta, E. N.) (Stanford Univ., 2021); https://plato.stanford.edu/archives/sum2021/entries/ethics-ai/

  73. Langlois, S. ‘Donald Trump’ explains money laundering to his son-in-law in ‘deepfake’ video. Market Watch (19 September 2019); https://www.marketwatch.com/story/donald-trump-explains-money-laundering-to-his-son-in-law-in-deepfake-video-2019-09-18

  74. Rajendra-Nicolucci, C. Language-generating A.I. is a free speech nightmare. Slate (30 September 2020); https://slate.com/technology/2020/09/language-ai-gpt-3-free-speech-harassment.html

  75. Meskys, E., Kalpokiene, J., Jurcys, P. & Liaudanskas, A. Regulating deep fakes: legal and ethical considerations.J. Intellect. Prop. Law Pract. 15, 24–31 (2019).

    Article  Google Scholar 

  76. Wagner, T. L. & Blewer, A. ‘The word real is no longer real’: deepfakes, gender, and the challenges of AI-altered video. Open Inf. Sci. 3, 32–46 (2019).

    Google Scholar 

  77. Fallis, D. The epistemic threat of deepfakes. Philos. Technol. https://doi.org/10.1007/s13347-020-00419-2 (2020).

  78. In event of moon disaster (MIT Center For Advanced Virtuality, 2020); https://moondisaster.org

  79. Ryan, E. The intersection of the Disney princess phenomenon and eating disorders. Response The Journal of Popular and American Culture https://responsejournal.net/issue/2016-08/article/intersection-disney-princess-phenomenon-and-eating-disorders (2016).

  80. Burroughs, B. Youtube kids: the app economy and mobile parenting. Soc. Media Soc. https://doi.org/10.1177/2056305117707189 (2017).

  81. Primack, B. A. et al. Social media use and perceived social isolation among young adults in the US. Am. J. Prev. Med. 53, 1–8 (2017).

    Article  Google Scholar 

  82. Umberson, D. & Karas Montez, J. Social relationships and health: a flashpoint for health policy. J. Health Soc. Behav. 51, S54–S66 (2010).

    Article  Google Scholar 

  83. Radesky, J. S. & Christakis, D. A. Increased screen time: implications for early childhood development and behavior. Pediatr. Clin. 63, 827–839 (2016).

    Google Scholar 

  84. Shin, H. & Ryan, A. M. Friend influence on early adolescent disruptive behavior in the classroom: teacher emotional support matters. Dev. Psychol. 53, 114–125 (2017).

    Article  Google Scholar 

  85. Chiu, T. K. Student engagement in K-12 online learning amid COVID-19: a qualitative approach from a self-determination theory perspective. Interactive Learn. Environ. https://doi.org/10.1080/10494820.2021.1926289 (2021).

  86. Abram, C. The most urgent threat of deepfakes isn’t politics. It’s porn. Vox (8 June 2020); https://www.vox.com/2020/6/8/21284005/urgent-threat-deepfakes-politics-porn-kristen-bell

  87. Ankel, S. Many revenge porn victims consider suicide—why aren’t schools doing more to stop it. The Guardian (7 May 2018).

  88. Kugler, M. B. & Pace, C. Deepfake privacy: attitudes and regulation. Northwestern Public Law Research Paper, SSRN 21-04 (2021); https://ssrn.com/abstract=3781968

  89. Malaria Must Die. David Beckham launches the world’s first voice petition to end malaria https://malariamustdie.com/news/david-beckham-launches-worlds-first-voice-petition-end-malaria (accessed 15 July 2015).

  90. Eshraghian, J. K. Human ownership of artificial creativity. Nat. Mach. Intell. 2, 157–160 (2020).

    Article  Google Scholar 

  91. Baek, S. Free Speech in the Digital Age: Deepfakes and the Marketplace of Ideas. Honors theses (PPE), University of Pennsylvania. Penn Libraries (2020).

  92. Fazio, L. K., Brashier, N. M., Payne, B. K. & Marsh, E. J. Knowledge does not protect against illusory truth. J. Exp. Psychol. Gen. 144, 993–1002 (2015).

    Article  Google Scholar 

  93. Amershi, S. et al. Guidelines for human-AI interaction. In Proc. 2019 CHI Conference on Human Factors in Computing Systems 3, 1–13 (ACM, 2019).

  94. Naruniec, J., Helminger, L., Schroers, C. & Weber, R. M. High-resolution neural face swapping for visual effects. In Proc. Computer Graphics Forum Vol. 39.4, 173–184 (Wiley, 2020).

  95. Ramanathan, M., Mishra, N. & Thalmann, N. M. Nadine humanoid social robotics platform. In Proc. Computer Graphics International Conference 490–496 (Springer, 2019).

  96. Claes, P. et al. Modeling 3D facial shape from DNA. PLoS Genet. 10, e1004224 (2014).

    Article  Google Scholar 

  97. Zhu, W., Fan, X. & Zhang, Y. Applications and research trends of digital human models in the manufacturing industry. Virtual Reality Intell. Hardware 1, 558–579 (2019).

    Article  Google Scholar 

  98. Our first artificial human. Samsung Neon https://www.neon.life/ (accessed 10 July 2020).

  99. SoulMachines Inc. Baby X: soul machines. SoulMachines Inc. https://www.soulmachines.com/ (accessed 10 July 2020).

  100. Corentin, J. Real-Time Voice Cloning. MSc thesis, Université de Liège (2019).

Download references

Author information

Authors and Affiliations

Authors

Contributions

P. Pataranutaporn developed the pipeline, assisted by V.D. The literature review was conducted by P. Pataranutaporn, V.D., J.L., P. Punpongsanon and M.S., who also contributed to the writing and editing of the manuscript. All other authors reviewed the manuscript. P. Pataranutaporn designed the figures. The pipeline was tested by D.N. The work was supervised by P.M. and M.S.

Corresponding author

Correspondence to Misha Sra.

Ethics declarations

Competing interests

The authors declare no competing interests.

Additional information

Publisher’s note Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Pataranutaporn, P., Danry, V., Leong, J. et al. AI-generated characters for supporting personalized learning and well-being. Nat Mach Intell 3, 1013–1022 (2021). https://doi.org/10.1038/s42256-021-00417-9

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1038/s42256-021-00417-9

This article is cited by

Search

Quick links

Nature Briefing

Sign up for the Nature Briefing newsletter — what matters in science, free to your inbox daily.

Get the most important science stories of the day, free in your inbox. Sign up for Nature Briefing