Using EbSynth to Create Better NeRF Facial Avatars

link管理

链接快照平台

输入网页链接，自动生成快照
标签化管理网页链接

相关文章推荐

怕老婆的西装 · 首頁 | 內壢高中 Moodle· 4 月前 ·

善良的牛腩 · 四川省公共资源交易信息网· 6 月前 ·

爱吹牛的牙膏 · minigame-unity-webgl-t ...· 8 月前 ·

不爱学习的手套 · Umeng友盟数据优化-大陆业务-灵蛋ASO ...· 11 月前 ·

深沉的菠菜 · 最新！杭城热门小学对口直升初中可能有变！· 1 年前 ·

</noscript>  <a class="skip-link screen-reader-text" href="#content"> Skip to content</a><div data-elementor-type="header" data-elementor-id="437" class="elementor elementor-437 elementor-location-header" data-elementor-post-type="elementor_library"><section class="elementor-section elementor-top-section elementor-element elementor-element-71a7926a elementor-section-height-min-height elementor-section-full_width elementor-section-height-default elementor-section-items-middle elementor-invisible" data-id="71a7926a" data-element_type="section" data-settings="{&quot;background_background&quot;:&quot;gradient&quot;,&quot;animation&quot;:&quot;fadeInLeft&quot;}"><div class="elementor-container elementor-column-gap-no"><div class="elementor-column elementor-col-33 elementor-top-column elementor-element elementor-element-557660f2" data-id="557660f2" data-element_type="column"><div class="elementor-widget-wrap elementor-element-populated"><div class="elementor-element elementor-element-7e1790fa elementor-widget elementor-widget-image" data-id="7e1790fa" data-element_type="widget" data-widget_type="image.default"><div class="elementor-widget-container"> <a href="https://metaphysic.ai"> <img loading="lazy" width="1253" height="101" src="https://blog.metaphysic.ai/wp-content/uploads/2021/06/Metaphysic_Logo-Lockup_White_RGB.png" class="attachment-full size-full wp-image-1122" alt="" srcset="https://blog.metaphysic.ai/wp-content/uploads/2021/06/Metaphysic_Logo-Lockup_White_RGB.png 1253w, https://blog.metaphysic.ai/wp-content/uploads/2021/06/Metaphysic_Logo-Lockup_White_RGB-300x24.png 300w, https://blog.metaphysic.ai/wp-content/uploads/2021/06/Metaphysic_Logo-Lockup_White_RGB-1024x83.png 1024w, https://blog.metaphysic.ai/wp-content/uploads/2021/06/Metaphysic_Logo-Lockup_White_RGB-768x62.png 768w" sizes="(max-width: 1253px) 100vw, 1253px"/> </a></div></div></div></div><div class="elementor-column elementor-col-33 elementor-top-column elementor-element elementor-element-5e8fec10" data-id="5e8fec10" data-element_type="column"><div class="elementor-widget-wrap elementor-element-populated"><div class="elementor-element elementor-element-a7dd76f elementor-widget elementor-widget-heading" data-id="a7dd76f" data-element_type="widget" data-widget_type="heading.default"><div class="elementor-widget-container"><h2 class="elementor-heading-title elementor-size-default"><a href="https://metaphysic.ai">Home</a></h2></div></div></div></div><div class="elementor-column elementor-col-33 elementor-top-column elementor-element elementor-element-78f14261 elementor-hidden-tablet elementor-hidden-phone" data-id="78f14261" data-element_type="column"><div class="elementor-widget-wrap"/></div></div></section></div><div data-elementor-type="single-post" data-elementor-id="875" class="elementor elementor-875 elementor-location-single post-8846 post type-post status-publish format-standard has-post-thumbnail hentry category-ai-machinelearning-deeplearning" data-elementor-post-type="elementor_library"><section class="elementor-section elementor-top-section elementor-element elementor-element-0b474cf elementor-section-height-min-height elementor-section-items-bottom elementor-section-boxed elementor-section-height-default" data-id="0b474cf" data-element_type="section" data-settings="{&quot;background_background&quot;:&quot;classic&quot;}"><div class="elementor-background-overlay"/><div class="elementor-container elementor-column-gap-no"><div class="elementor-column elementor-col-100 elementor-top-column elementor-element elementor-element-c0890b4" data-id="c0890b4" data-element_type="column"><div class="elementor-widget-wrap elementor-element-populated"><div class="elementor-element elementor-element-8a1df2a elementor-invisible elementor-widget elementor-widget-heading" data-id="8a1df2a" data-element_type="widget" data-settings="{&quot;motion_fx_motion_fx_scrolling&quot;:&quot;yes&quot;,&quot;motion_fx_translateY_effect&quot;:&quot;yes&quot;,&quot;motion_fx_translateY_speed&quot;:{&quot;unit&quot;:&quot;px&quot;,&quot;size&quot;:0.5,&quot;sizes&quot;:[]},&quot;_animation&quot;:&quot;fadeInUp&quot;,&quot;_animation_delay&quot;:500,&quot;motion_fx_translateY_affectedRange&quot;:{&quot;unit&quot;:&quot;%&quot;,&quot;size&quot;:&quot;&quot;,&quot;sizes&quot;:{&quot;start&quot;:0,&quot;end&quot;:100}},&quot;motion_fx_devices&quot;:[&quot;desktop&quot;,&quot;tablet&quot;,&quot;mobile&quot;]}" data-widget_type="heading.default"><div class="elementor-widget-container"><h1 class="elementor-heading-title elementor-size-default">Using EbSynth to Create Better NeRF Facial Avatars</h1></div></div></div></div></div></section><section class="elementor-section elementor-top-section elementor-element elementor-element-68102deb elementor-section-boxed elementor-section-height-default elementor-section-height-default" data-id="68102deb" data-element_type="section"><div class="elementor-container elementor-column-gap-default"><div class="elementor-column elementor-col-100 elementor-top-column elementor-element elementor-element-18822cd3" data-id="18822cd3" data-element_type="column"><div class="elementor-widget-wrap elementor-element-populated"><div class="elementor-element elementor-element-20309ab6 elementor-widget elementor-widget-theme-post-featured-image elementor-widget-image" data-id="20309ab6" data-element_type="widget" data-widget_type="theme-post-featured-image.default"><div class="elementor-widget-container"><figure class="wp-caption"> <img loading="lazy" width="800" height="503" src="https://blog.metaphysic.ai/wp-content/uploads/2023/06/Instruct-Video2Avatar-MAIN.jpg" class="attachment-large size-large wp-image-8848" alt="" srcset="https://blog.metaphysic.ai/wp-content/uploads/2023/06/Instruct-Video2Avatar-MAIN.jpg 818w, https://blog.metaphysic.ai/wp-content/uploads/2023/06/Instruct-Video2Avatar-MAIN-300x189.jpg 300w, https://blog.metaphysic.ai/wp-content/uploads/2023/06/Instruct-Video2Avatar-MAIN-768x483.jpg 768w" sizes="(max-width: 800px) 100vw, 800px"/><figcaption class="widget-image-caption wp-caption-text"/></figure></div></div><div class="elementor-element elementor-element-882f137 elementor-widget elementor-widget-spacer" data-id="882f137" data-element_type="widget" data-widget_type="spacer.default"><div class="elementor-widget-container"><div class="elementor-spacer"><div class="elementor-spacer-inner"/></div></div></div><div class="elementor-element elementor-element-b8c4c31 elementor-widget elementor-widget-post-info" data-id="b8c4c31" data-element_type="widget" data-widget_type="post-info.default"><div class="elementor-widget-container"><ul class="elementor-inline-items elementor-icon-list-items elementor-post-info"><li class="elementor-icon-list-item elementor-repeater-item-a970b2f elementor-inline-item" itemprop="datePublished"> <a href="https://blog.metaphysic.ai/2023/06/12/"> <span class="elementor-icon-list-icon"> <i aria-hidden="true" class="fas fa-calendar"/> </span> <span class="elementor-icon-list-text elementor-post-info__item elementor-post-info__item--type-date"> <time>June 12, 2023</time> </span> </a></li><li class="elementor-icon-list-item elementor-repeater-item-63768f4 elementor-inline-item"> <span class="elementor-icon-list-icon"> <i aria-hidden="true" class="far fa-clock"/> </span> <span class="elementor-icon-list-text elementor-post-info__item elementor-post-info__item--type-time"> <time>10:01 am</time> </span></li></ul></div></div><div class="elementor-element elementor-element-83a67cd elementor-widget elementor-widget-heading" data-id="83a67cd" data-element_type="widget" data-widget_type="heading.default"><div class="elementor-widget-container"><h3 class="elementor-heading-title elementor-size-default">About the author</h3></div></div><div class="elementor-element elementor-element-7323686 elementor-author-box--avatar-yes elementor-author-box--name-yes elementor-author-box--biography-yes elementor-widget elementor-widget-author-box" data-id="7323686" data-element_type="widget" data-widget_type="author-box.default"><div class="elementor-widget-container"><div class="elementor-author-box"><div class="elementor-author-box__avatar"> <img src="https://secure.gravatar.com/avatar/4d87f980a06e3c1ad01037d165d99922?s=300&amp;d=mm&amp;r=g" alt="Picture of Martin Anderson" loading="lazy"/></div><div class="elementor-author-box__text"><div><h4 class="elementor-author-box__name"> Martin Anderson</h4></div><div class="elementor-author-box__bio"> I'm Martin Anderson, a writer occupied exclusively with machine learning, artificial intelligence, big data, and closely-related topics, with an emphasis on image synthesis, computer vision, and NLP.</div></div></div></div></div><div class="elementor-element elementor-element-8ef8f3e elementor-widget elementor-widget-post-info" data-id="8ef8f3e" data-element_type="widget" data-widget_type="post-info.default"><div class="elementor-widget-container"><ul class="elementor-inline-items elementor-icon-list-items elementor-post-info"><li class="elementor-icon-list-item elementor-repeater-item-ddecf71 elementor-inline-item"> <a href="https://martinanderson.ai"> <span class="elementor-icon-list-icon"> <i aria-hidden="true" class="fas fa-link"/> </span> <span class="elementor-icon-list-text elementor-post-info__item elementor-post-info__item--type-custom"> Author Website </span> </a></li><li class="elementor-icon-list-item elementor-repeater-item-83614f2 elementor-inline-item"> <a href="https://blog.metaphysic.ai/author/metaphysicai/"> <span class="elementor-icon-list-icon"> <i aria-hidden="true" class="fas fa-archive"/> </span> <span class="elementor-icon-list-text elementor-post-info__item elementor-post-info__item--type-custom"> Author Archive </span> </a></li></ul></div></div><section class="elementor-section elementor-inner-section elementor-element elementor-element-d998ba4 elementor-section-boxed elementor-section-height-default elementor-section-height-default" data-id="d998ba4" data-element_type="section"><div class="elementor-container elementor-column-gap-default"><div class="elementor-column elementor-col-50 elementor-inner-column elementor-element elementor-element-2f7ead33" data-id="2f7ead33" data-element_type="column"><div class="elementor-widget-wrap elementor-element-populated"><div class="elementor-element elementor-element-458eb659 elementor-widget elementor-widget-heading" data-id="458eb659" data-element_type="widget" data-widget_type="heading.default"><div class="elementor-widget-container"><h2 class="elementor-heading-title elementor-size-default">Share This Post</h2></div></div></div></div><div class="elementor-column elementor-col-50 elementor-inner-column elementor-element elementor-element-2fff8efa" data-id="2fff8efa" data-element_type="column"><div class="elementor-widget-wrap elementor-element-populated"><div class="elementor-element elementor-element-7f39a758 elementor-share-buttons--view-icon elementor-share-buttons--skin-flat elementor-share-buttons--align-right elementor-share-buttons--color-custom elementor-share-buttons-mobile--align-center elementor-share-buttons--shape-square elementor-grid-0 elementor-widget elementor-widget-share-buttons" data-id="7f39a758" data-element_type="widget" data-widget_type="share-buttons.default"><div class="elementor-widget-container"><div class="elementor-grid"><div class="elementor-grid-item"><div class="elementor-share-btn elementor-share-btn_facebook" role="button" tabindex="0" aria-label="Share on facebook"> <span class="elementor-share-btn__icon"> <i class="fab fa-facebook" aria-hidden="true"/> </span></div></div><div class="elementor-grid-item"><div class="elementor-share-btn elementor-share-btn_linkedin" role="button" tabindex="0" aria-label="Share on linkedin"> <span class="elementor-share-btn__icon"> <i class="fab fa-linkedin" aria-hidden="true"/> </span></div></div><div class="elementor-grid-item"><div class="elementor-share-btn elementor-share-btn_twitter" role="button" tabindex="0" aria-label="Share on twitter"> <span class="elementor-share-btn__icon"> <i class="fab fa-twitter" aria-hidden="true"/> </span></div></div><div class="elementor-grid-item"><div class="elementor-share-btn elementor-share-btn_email" role="button" tabindex="0" aria-label="Share on email"> <span class="elementor-share-btn__icon"> <i class="fas fa-envelope" aria-hidden="true"/> </span></div></div></div></div></div></div></div></div></section><div class="elementor-element elementor-element-42825e76 elementor-widget elementor-widget-spacer" data-id="42825e76" data-element_type="widget" data-widget_type="spacer.default"><div class="elementor-widget-container"><div class="elementor-spacer"><div class="elementor-spacer-inner"/></div></div></div><div class="elementor-element elementor-element-230321ec elementor-widget elementor-widget-theme-post-content" data-id="230321ec" data-element_type="widget" data-widget_type="theme-post-content.default"><div class="elementor-widget-container"><div data-elementor-type="wp-post" data-elementor-id="8846" class="elementor elementor-8846" data-elementor-post-type="post"><section class="elementor-section elementor-top-section elementor-element elementor-element-0f5a300 elementor-section-boxed elementor-section-height-default elementor-section-height-default" data-id="0f5a300" data-element_type="section"><div class="elementor-container elementor-column-gap-default"><div class="elementor-column elementor-col-100 elementor-top-column elementor-element elementor-element-7c963f6" data-id="7c963f6" data-element_type="column"><div class="elementor-widget-wrap elementor-element-populated"><div class="elementor-element elementor-element-9abdec7 elementor-widget elementor-widget-text-editor" data-id="9abdec7" data-element_type="widget" data-widget_type="text-editor.default"><div class="elementor-widget-container"><p>New research out of Shanghai offers a novel method of generating NeRF-based facial avatars – by using <a href="https://blog.metaphysic.ai/stable-diffusion-is-video-coming-soon/">Stable Diffusion</a> and the popular tweening software <a href="https://blog.metaphysic.ai/generating-temporally-coherent-high-resolution-video-with-stable-diffusion/#ebsynth">EbSynth</a>.</p></div></div></div></div></div></section><section class="elementor-section elementor-top-section elementor-element elementor-element-9114275 elementor-section-boxed elementor-section-height-default elementor-section-height-default" data-id="9114275" data-element_type="section"><div class="elementor-container elementor-column-gap-default"><div class="elementor-column elementor-col-100 elementor-top-column elementor-element elementor-element-0232709" data-id="0232709" data-element_type="column"><div class="elementor-widget-wrap"/></div></div></section><section class="elementor-section elementor-top-section elementor-element elementor-element-d0c762d elementor-section-boxed elementor-section-height-default elementor-section-height-default" data-id="d0c762d" data-element_type="section"><div class="elementor-container elementor-column-gap-default"><div class="elementor-column elementor-col-100 elementor-top-column elementor-element elementor-element-75fb599" data-id="75fb599" data-element_type="column"><div class="elementor-widget-wrap elementor-element-populated"><div class="elementor-element elementor-element-caa4fc5 elementor-widget__width-initial elementor-widget elementor-widget-video" data-id="caa4fc5" data-element_type="widget" data-settings="{&quot;video_type&quot;:&quot;hosted&quot;,&quot;autoplay&quot;:&quot;yes&quot;,&quot;play_on_mobile&quot;:&quot;yes&quot;,&quot;mute&quot;:&quot;yes&quot;,&quot;loop&quot;:&quot;yes&quot;,&quot;controls&quot;:&quot;yes&quot;}" data-widget_type="video.default"><div class="elementor-widget-container"><div class="e-hosted-video elementor-wrapper elementor-open-inline"><video class="elementor-video lazyload" data-src="https://blog.metaphysic.ai/wp-content/uploads/2023/06/Instruct-Video2Avatar-.mp4" autoplay="" loop="" controls="" muted="muted" playsinline="" controlslist="nodownload"/></div></div></div><div class="elementor-element elementor-element-d90cf16 elementor-widget elementor-widget-text-editor" data-id="d90cf16" data-element_type="widget" data-widget_type="text-editor.default"><div class="elementor-widget-container"><p><span style="color: #999999;"><em>Instruct-Video2Avatar utilizes the InstructPix2Pix adjunct framework for Stable Diffusion, and the EbSynth (non-AI) tweening software to extrapolate temporally consistent frames from the altered images. Source: https://github.com/lsx0101/Instruct-Video2Avatar</em></span></p></div></div></div></div></div></section><section class="elementor-section elementor-top-section elementor-element elementor-element-5a2c797 elementor-section-boxed elementor-section-height-default elementor-section-height-default" data-id="5a2c797" data-element_type="section"><div class="elementor-container elementor-column-gap-default"><div class="elementor-column elementor-col-100 elementor-top-column elementor-element elementor-element-4427035" data-id="4427035" data-element_type="column"><div class="elementor-widget-wrap elementor-element-populated"><div class="elementor-element elementor-element-97a3a41 elementor-widget elementor-widget-text-editor" data-id="97a3a41" data-element_type="widget" data-widget_type="text-editor.default"><div class="elementor-widget-container"><p>The new approach is called <em><i>Instruct-Video2Avatar</i></em> (IV2A). Like some of the prior systems from which it takes inspiration, IV2A interferes with the photogrammetry process native to <a href="https://blog.metaphysic.ai/nerf-successor-deepfakes/">Neural Radiance Fields</a> (NeRF); instead of allowing real-world face images to be composed directly into an explorable neural matrix, IV2A first runs <em><i>two additional procedures</i></em> on the captured images.</p><p>First, it converts the source face image into an artistic, alternate or otherwise stylized version using Stable Diffusion and <a href="https://www.timothybrooks.com/instruct-pix2pix">InstructPix2Pix</a> (an improved image synthesis add-on framework which we <a href="https://blog.metaphysic.ai/instructpix2pix-accurate-ai-based-image-editing-with-gpt-3-and-stable-diffusion/">covered when it came out late last year</a>)…</p></div></div></div></div></div></section><section class="elementor-section elementor-top-section elementor-element elementor-element-7162b8e elementor-section-boxed elementor-section-height-default elementor-section-height-default" data-id="7162b8e" data-element_type="section"><div class="elementor-container elementor-column-gap-default"><div class="elementor-column elementor-col-100 elementor-top-column elementor-element elementor-element-4e939ae" data-id="4e939ae" data-element_type="column"><div class="elementor-widget-wrap elementor-element-populated"><div class="elementor-element elementor-element-506bc76 elementor-widget elementor-widget-image" data-id="506bc76" data-element_type="widget" data-widget_type="image.default"><div class="elementor-widget-container"><figure class="wp-caption"> <img fetchpriority="high" decoding="async" width="768" height="589" src="data:image/gif;base64,R0lGODlhAQABAIAAAAAAAP///yH5BAEAAAAALAAAAAABAAEAAAIBRAA7" data-src="https://blog.metaphysic.ai/wp-content/uploads/2023/06/replace-the-fruits-with-cake-instructpix2pix-768x589.jpg" class="attachment-medium_large size-medium_large wp-image-8862 lazyload" alt="Examples from the November release of InstructPix2Pix, which offers better compositionality and accuracy in image-to-image conversions in Stable Diffusion. Source: https://arxiv.org/pdf/2211.09800.pdf" data-srcset="https://blog.metaphysic.ai/wp-content/uploads/2023/06/replace-the-fruits-with-cake-instructpix2pix-768x589.jpg 768w, https://blog.metaphysic.ai/wp-content/uploads/2023/06/replace-the-fruits-with-cake-instructpix2pix-300x230.jpg 300w, https://blog.metaphysic.ai/wp-content/uploads/2023/06/replace-the-fruits-with-cake-instructpix2pix-1024x785.jpg 1024w, https://blog.metaphysic.ai/wp-content/uploads/2023/06/replace-the-fruits-with-cake-instructpix2pix.jpg 1400w" sizes="(max-width: 768px) 100vw, 768px"/><noscript><img decoding="async" width="768" height="589" src="https://blog.metaphysic.ai/wp-content/uploads/2023/06/replace-the-fruits-with-cake-instructpix2pix-768x589.jpg" class="attachment-medium_large size-medium_large wp-image-8862 lazyload" alt="Examples from the November release of InstructPix2Pix, which offers better compositionality and accuracy in image-to-image conversions in Stable Diffusion. Source: https://arxiv.org/pdf/2211.09800.pdf" srcset="https://blog.metaphysic.ai/wp-content/uploads/2023/06/replace-the-fruits-with-cake-instructpix2pix-768x589.jpg 768w, https://blog.metaphysic.ai/wp-content/uploads/2023/06/replace-the-fruits-with-cake-instructpix2pix-300x230.jpg 300w, https://blog.metaphysic.ai/wp-content/uploads/2023/06/replace-the-fruits-with-cake-instructpix2pix-1024x785.jpg 1024w, https://blog.metaphysic.ai/wp-content/uploads/2023/06/replace-the-fruits-with-cake-instructpix2pix.jpg 1400w" sizes="(max-width: 768px) 100vw, 768px"/></noscript><figcaption class="widget-image-caption wp-caption-text">Examples from the November release of InstructPix2Pix, which offers better compositionality and accuracy in image-to-image conversions in Stable Diffusion. Source: https://arxiv.org/pdf/2211.09800.pdf</figcaption></figure></div></div></div></div></div></section><section class="elementor-section elementor-top-section elementor-element elementor-element-eeec487 elementor-section-boxed elementor-section-height-default elementor-section-height-default" data-id="eeec487" data-element_type="section"><div class="elementor-container elementor-column-gap-default"><div class="elementor-column elementor-col-100 elementor-top-column elementor-element elementor-element-ae9f252" data-id="ae9f252" data-element_type="column"><div class="elementor-widget-wrap elementor-element-populated"><div class="elementor-element elementor-element-6e935cc elementor-widget elementor-widget-text-editor" data-id="6e935cc" data-element_type="widget" data-widget_type="text-editor.default"><div class="elementor-widget-container"><p>Then it uses EbSynth to perform <a href="https://www.adobe.com/creativecloud/video/discover/tweening.html">tweening</a> on a small number of the altered keyframes, which provides entire coverage in the chosen style: </p></div></div></div></div></div></section><section class="elementor-section elementor-top-section elementor-element elementor-element-324891f elementor-section-boxed elementor-section-height-default elementor-section-height-default" data-id="324891f" data-element_type="section"><div class="elementor-container elementor-column-gap-default"><div class="elementor-column elementor-col-100 elementor-top-column elementor-element elementor-element-e578257" data-id="e578257" data-element_type="column"><div class="elementor-widget-wrap elementor-element-populated"><div class="elementor-element elementor-element-74c1cc5 elementor-widget elementor-widget-image" data-id="74c1cc5" data-element_type="widget" data-widget_type="image.default"><div class="elementor-widget-container"><figure class="wp-caption"> <img decoding="async" width="768" height="736" src="data:image/gif;base64,R0lGODlhAQABAIAAAAAAAP///yH5BAEAAAAALAAAAAABAAEAAAIBRAA7" data-src="https://blog.metaphysic.ai/wp-content/uploads/2023/06/ebsynth-768x736.jpg" class="attachment-medium_large size-medium_large wp-image-8863 lazyload" alt="Given only two 'altered' keyframes, EbSynth can impose an interpretation of them onto a real video. This process is used in Instruct-Video2Avatar to provide a temporally coherent series of altered faces into a NeRF model. Source: https://www.facebook.com/scrtwpns/posts/ebsynth-tipwe-often-get-asked-how-to-handle-multiple-keyframes-in-ebsynth-and-i-/2760592233975591/" data-srcset="https://blog.metaphysic.ai/wp-content/uploads/2023/06/ebsynth-768x736.jpg 768w, https://blog.metaphysic.ai/wp-content/uploads/2023/06/ebsynth-300x287.jpg 300w, https://blog.metaphysic.ai/wp-content/uploads/2023/06/ebsynth.jpg 1000w" sizes="(max-width: 768px) 100vw, 768px"/><noscript><img loading="lazy" decoding="async" width="768" height="736" src="https://blog.metaphysic.ai/wp-content/uploads/2023/06/ebsynth-768x736.jpg" class="attachment-medium_large size-medium_large wp-image-8863 lazyload" alt="Given only two 'altered' keyframes, EbSynth can impose an interpretation of them onto a real video. This process is used in Instruct-Video2Avatar to provide a temporally coherent series of altered faces into a NeRF model. Source: https://www.facebook.com/scrtwpns/posts/ebsynth-tipwe-often-get-asked-how-to-handle-multiple-keyframes-in-ebsynth-and-i-/2760592233975591/" srcset="https://blog.metaphysic.ai/wp-content/uploads/2023/06/ebsynth-768x736.jpg 768w, https://blog.metaphysic.ai/wp-content/uploads/2023/06/ebsynth-300x287.jpg 300w, https://blog.metaphysic.ai/wp-content/uploads/2023/06/ebsynth.jpg 1000w" sizes="(max-width: 768px) 100vw, 768px"/></noscript><figcaption class="widget-image-caption wp-caption-text">Given only two 'altered' keyframes, EbSynth can impose an interpretation of them onto a real video. This process is used in Instruct-Video2Avatar to provide a temporally coherent series of altered faces into a NeRF model. Source: https://www.facebook.com/scrtwpns/posts/ebsynth-tipwe-often-get-asked-how-to-handle-multiple-keyframes-in-ebsynth-and-i-/2760592233975591/</figcaption></figure></div></div></div></div></div></section><section class="elementor-section elementor-top-section elementor-element elementor-element-6b5f85b elementor-section-boxed elementor-section-height-default elementor-section-height-default" data-id="6b5f85b" data-element_type="section"><div class="elementor-container elementor-column-gap-default"><div class="elementor-column elementor-col-100 elementor-top-column elementor-element elementor-element-3515e0c" data-id="3515e0c" data-element_type="column"><div class="elementor-widget-wrap elementor-element-populated"><div class="elementor-element elementor-element-891bb46 elementor-widget elementor-widget-text-editor" data-id="891bb46" data-element_type="widget" data-widget_type="text-editor.default"><div class="elementor-widget-container"><p>These consistent altered face/head images are then passed to a version of the <a href="https://zielon.github.io/insta/">INSTA</a> (<em><i>Instant Volumetric Head Avatars</i></em>) system, which rationalizes the rendered images into an explorable neural NeRF space.</p></div></div></div></div></div></section><section class="elementor-section elementor-top-section elementor-element elementor-element-afe4621 elementor-section-boxed elementor-section-height-default elementor-section-height-default" data-id="afe4621" data-element_type="section"><div class="elementor-container elementor-column-gap-default"><div class="elementor-column elementor-col-100 elementor-top-column elementor-element elementor-element-b3c9092" data-id="b3c9092" data-element_type="column"><div class="elementor-widget-wrap elementor-element-populated"><div class="elementor-element elementor-element-2aea771 elementor-widget elementor-widget-video" data-id="2aea771" data-element_type="widget" data-settings="{&quot;youtube_url&quot;:&quot;https:\/\/www.youtube.com\/watch?v=HOgaeWTih7Q&quot;,&quot;video_type&quot;:&quot;youtube&quot;,&quot;controls&quot;:&quot;yes&quot;}" data-widget_type="video.default"><div class="elementor-widget-container"><div class="elementor-wrapper elementor-open-inline"><div class="elementor-video"/></div></div></div></div></div></div></section><section class="elementor-section elementor-top-section elementor-element elementor-element-8fd5786 elementor-section-boxed elementor-section-height-default elementor-section-height-default" data-id="8fd5786" data-element_type="section"><div class="elementor-container elementor-column-gap-default"><div class="elementor-column elementor-col-100 elementor-top-column elementor-element elementor-element-337d118" data-id="337d118" data-element_type="column"><div class="elementor-widget-wrap elementor-element-populated"><div class="elementor-element elementor-element-a85ec6f elementor-widget elementor-widget-text-editor" data-id="a85ec6f" data-element_type="widget" data-widget_type="text-editor.default"><div class="elementor-widget-container"><p>The process allows for photorealistic or more fantastical transformations, though its obvious potential for <a href="https://blog.metaphysic.ai/deepfakes/">deepfakes</a>-style usage (i.e., identity transfer rather than stylization) is not closely examined in the new work.</p></div></div></div></div></div></section><section class="elementor-section elementor-top-section elementor-element elementor-element-7c94890 elementor-section-boxed elementor-section-height-default elementor-section-height-default" data-id="7c94890" data-element_type="section"><div class="elementor-container elementor-column-gap-default"><div class="elementor-column elementor-col-100 elementor-top-column elementor-element elementor-element-20f03d5" data-id="20f03d5" data-element_type="column"><div class="elementor-widget-wrap elementor-element-populated"><div class="elementor-element elementor-element-8a07976 elementor-widget elementor-widget-image" data-id="8a07976" data-element_type="widget" data-widget_type="image.default"><div class="elementor-widget-container"><figure class="wp-caption"> <img loading="lazy" decoding="async" width="768" height="584" src="data:image/gif;base64,R0lGODlhAQABAIAAAAAAAP///yH5BAEAAAAALAAAAAABAAEAAAIBRAA7" data-src="https://blog.metaphysic.ai/wp-content/uploads/2023/06/biden-768x584.jpg" class="attachment-medium_large size-medium_large wp-image-8872 lazyload" alt="Joe Biden 'Hulks out' with IV2A." data-srcset="https://blog.metaphysic.ai/wp-content/uploads/2023/06/biden-768x584.jpg 768w, https://blog.metaphysic.ai/wp-content/uploads/2023/06/biden-300x228.jpg 300w, https://blog.metaphysic.ai/wp-content/uploads/2023/06/biden.jpg 1000w" sizes="(max-width: 768px) 100vw, 768px"/><noscript><img loading="lazy" decoding="async" width="768" height="584" src="https://blog.metaphysic.ai/wp-content/uploads/2023/06/biden-768x584.jpg" class="attachment-medium_large size-medium_large wp-image-8872 lazyload" alt="Joe Biden 'Hulks out' with IV2A." srcset="https://blog.metaphysic.ai/wp-content/uploads/2023/06/biden-768x584.jpg 768w, https://blog.metaphysic.ai/wp-content/uploads/2023/06/biden-300x228.jpg 300w, https://blog.metaphysic.ai/wp-content/uploads/2023/06/biden.jpg 1000w" sizes="(max-width: 768px) 100vw, 768px"/></noscript><figcaption class="widget-image-caption wp-caption-text">Joe Biden 'Hulks out' with IV2A.</figcaption></figure></div></div></div></div></div></section><section class="elementor-section elementor-top-section elementor-element elementor-element-c1a2ab9 elementor-section-boxed elementor-section-height-default elementor-section-height-default" data-id="c1a2ab9" data-element_type="section"><div class="elementor-container elementor-column-gap-default"><div class="elementor-column elementor-col-100 elementor-top-column elementor-element elementor-element-29b215d" data-id="29b215d" data-element_type="column"><div class="elementor-widget-wrap elementor-element-populated"><div class="elementor-element elementor-element-10d370b elementor-widget elementor-widget-text-editor" data-id="10d370b" data-element_type="widget" data-widget_type="text-editor.default"><div class="elementor-widget-container"><p>However, the smooth interpretations of EbSynth also allow for an unusual level of temporal coherence in more simplistic animated styles – one of the most sought-after results in the Stable Diffusion community.</p></div></div></div></div></div></section><section class="elementor-section elementor-top-section elementor-element elementor-element-b01d00b elementor-section-boxed elementor-section-height-default elementor-section-height-default" data-id="b01d00b" data-element_type="section"><div class="elementor-container elementor-column-gap-default"><div class="elementor-column elementor-col-100 elementor-top-column elementor-element elementor-element-4571076" data-id="4571076" data-element_type="column"><div class="elementor-widget-wrap elementor-element-populated"><div class="elementor-element elementor-element-9e6e3f3 elementor-widget__width-initial elementor-widget elementor-widget-video" data-id="9e6e3f3" data-element_type="widget" data-settings="{&quot;video_type&quot;:&quot;hosted&quot;,&quot;autoplay&quot;:&quot;yes&quot;,&quot;play_on_mobile&quot;:&quot;yes&quot;,&quot;mute&quot;:&quot;yes&quot;,&quot;loop&quot;:&quot;yes&quot;,&quot;controls&quot;:&quot;yes&quot;}" data-widget_type="video.default"><div class="elementor-widget-container"><div class="e-hosted-video elementor-wrapper elementor-open-inline"><video class="elementor-video lazyload" data-src="https://blog.metaphysic.ai/wp-content/uploads/2023/06/ebsynth-demo.mp4" autoplay="" loop="" controls="" muted="muted" playsinline="" controlslist="nodownload"/></div></div></div></div></div></div></section><section class="elementor-section elementor-top-section elementor-element elementor-element-8ad9221 elementor-section-boxed elementor-section-height-default elementor-section-height-default" data-id="8ad9221" data-element_type="section"><div class="elementor-container elementor-column-gap-default"><div class="elementor-column elementor-col-100 elementor-top-column elementor-element elementor-element-9cf779e" data-id="9cf779e" data-element_type="column"><div class="elementor-widget-wrap elementor-element-populated"><div class="elementor-element elementor-element-756cf5b elementor-widget elementor-widget-text-editor" data-id="756cf5b" data-element_type="widget" data-widget_type="text-editor.default"><div class="elementor-widget-container"><p><span style="color: #999999;"><em>EbSynth’s interpretive abilities mean that the images provided to the NeRF system are consistent, leading to smooth video interpretations.</em> </span></p></div></div></div></div></div></section><section class="elementor-section elementor-top-section elementor-element elementor-element-63a30f6 elementor-section-boxed elementor-section-height-default elementor-section-height-default" data-id="63a30f6" data-element_type="section"><div class="elementor-container elementor-column-gap-default"><div class="elementor-column elementor-col-100 elementor-top-column elementor-element elementor-element-209196c" data-id="209196c" data-element_type="column"><div class="elementor-widget-wrap elementor-element-populated"><div class="elementor-element elementor-element-07c756c elementor-widget elementor-widget-text-editor" data-id="07c756c" data-element_type="widget" data-widget_type="text-editor.default"><div class="elementor-widget-container"><p>The resulting NeRF is a <em><i>deformable</i></em> radiance field, which means that the <a href="https://blog.metaphysic.ai/real-time-photorealistic-hands-for-neural-environments/#canon">canonical</a> ‘default’ disposition is used as a baseline from which morphs and deviations are generated, such as turning the head, tilting it, and changing facial expressions.</p><p>Though several research initiatives of the last six months have leveraged Stable Diffusion innovations <a href="https://blog.metaphysic.ai/bringing-temporal-coherence-to-stable-diffusion-with-flow-maps/">such as ControlNet</a> and <a href="https://blog.metaphysic.ai/fine-tuning-in-machine-learning/#dreambooth">DreamBooth</a>, IV2A is the first to come to our attention that has used EbSynth, which is a popular method in the Stable Diffusion community of providing fluid and coherent temporal motion to a system that has no native mechanism to provide such functionality.</p><p>EbSynth has a number of shortcomings in respect to this objective, not least that it can require many keyframes in order to provide the smoothest motion, and at the same time limits the number of keyframes usable for any one clip – which means that long productions require the stitching together of multiple EbSynth projects.</p><p>In terms of overall consistency, these requirements also oblige the Stable Diffusion user to create a consistent series of keyframes, so that the characteristics of the material do not subtly change as the video progresses (which is in itself difficult to achieve).</p><p>By using EbSynth in the more limited way outlined in the new project, the user need only obtain consistency for a <em><i>small number of keyframes</i></em>, with EbSynth generating interstitial frames, and NeRF thereafter handling temporal consistency in a predictable and relatively convincing manner.</p><p>The use of EbSynth has been a hobbyist or artisanal pursuit for some years, but the recent beta trial of the <a href="https://www.thevfxmedia.com/articles/ebsynth-studio-1-0-optimized-for-studio-pipelines">Studio 1.0 version</a> offers a CLI-driven ‘version optimized for studio pipelines’, perhaps signifying that tweening will remain the primary method of overcoming Stable Diffusion’s temporal shortcomings in the near future. IV2A is the first notable academic initiative to incorporate EbSynth into a rational neural synthesis pipeline.</p><p>The <a href="https://arxiv.org/abs/2306.02903">new paper</a> is titled <em><i>Instruct-Video2Avatar: Video-to-Avatar Generation with Instructions</i></em>, and comes from Shaoxu Li of the John Hopcroft Center for Computer Science at Shanghai Jiao Tong University.</p></div></div></div></div></div></section><section class="elementor-section elementor-top-section elementor-element elementor-element-81f1b4d elementor-section-boxed elementor-section-height-default elementor-section-height-default" data-id="81f1b4d" data-element_type="section"><div class="elementor-container elementor-column-gap-default"><div class="elementor-column elementor-col-100 elementor-top-column elementor-element elementor-element-fb4da40" data-id="fb4da40" data-element_type="column"><div class="elementor-widget-wrap elementor-element-populated"><div class="elementor-element elementor-element-b6743cd elementor-widget elementor-widget-heading" data-id="b6743cd" data-element_type="widget" data-widget_type="heading.default"><div class="elementor-widget-container"><h2 class="elementor-heading-title elementor-size-default">Approach</h2></div></div></div></div></div></section><section class="elementor-section elementor-top-section elementor-element elementor-element-10f0550 elementor-section-boxed elementor-section-height-default elementor-section-height-default" data-id="10f0550" data-element_type="section"><div class="elementor-container elementor-column-gap-default"><div class="elementor-column elementor-col-100 elementor-top-column elementor-element elementor-element-eee4a46" data-id="eee4a46" data-element_type="column"><div class="elementor-widget-wrap elementor-element-populated"><div class="elementor-element elementor-element-9f14a8e elementor-widget elementor-widget-text-editor" data-id="9f14a8e" data-element_type="widget" data-widget_type="text-editor.default"><div class="elementor-widget-container"><p>As with <a href="https://blog.metaphysic.ai/creating-state-of-the-art-nerf-head-avatars-in-minutes/">typical NeRF workflows</a> in avatar creation, IV2A takes an input video as source material from which to build up a neural reconstruction, which can then be subject to deformations that represent natural movement. The text-to-image component comes in the form of Stable Diffusion image-to-image manipulations of each frame, facilitated by InstructPix2Pix.</p><p>One ‘exemplar’ image is initially fed into the system, an image altered by text instructions such as<em> ‘Make him older’</em>, <em>‘make him an elf’</em>, etc. If one were to run InstructPix2Pix sequentially and unaided over the extracted video frames, the results would exhibit inconsistencies typical of Stable Diffusion’s inability to reproduce any solution perfectly twice; but the EbSynth tweening instead takes the previous frame as the starting point for the next frame, supporting the necessary continuity of appearance.</p></div></div></div></div></div></section><section class="elementor-section elementor-top-section elementor-element elementor-element-003143b elementor-section-boxed elementor-section-height-default elementor-section-height-default" data-id="003143b" data-element_type="section"><div class="elementor-container elementor-column-gap-default"><div class="elementor-column elementor-col-100 elementor-top-column elementor-element elementor-element-1bcdde3" data-id="1bcdde3" data-element_type="column"><div class="elementor-widget-wrap elementor-element-populated"><div class="elementor-element elementor-element-25b2701 elementor-widget elementor-widget-image" data-id="25b2701" data-element_type="widget" data-widget_type="image.default"><div class="elementor-widget-container"><figure class="wp-caption"> <img loading="lazy" decoding="async" width="1200" height="526" src="data:image/gif;base64,R0lGODlhAQABAIAAAAAAAP///yH5BAEAAAAALAAAAAABAAEAAAIBRAA7" data-src="https://blog.metaphysic.ai/wp-content/uploads/2023/06/instruct-video-2-avatar-architecture.jpg" class="attachment-full size-full wp-image-8874 lazyload" alt="IV2A iteratively updates an animatable head avatar. Source: https://arxiv.org/pdf/2306.02903.pdf" data-srcset="https://blog.metaphysic.ai/wp-content/uploads/2023/06/instruct-video-2-avatar-architecture.jpg 1200w, https://blog.metaphysic.ai/wp-content/uploads/2023/06/instruct-video-2-avatar-architecture-300x132.jpg 300w, https://blog.metaphysic.ai/wp-content/uploads/2023/06/instruct-video-2-avatar-architecture-1024x449.jpg 1024w, https://blog.metaphysic.ai/wp-content/uploads/2023/06/instruct-video-2-avatar-architecture-768x337.jpg 768w" sizes="(max-width: 1200px) 100vw, 1200px"/><noscript><img loading="lazy" decoding="async" width="1200" height="526" src="https://blog.metaphysic.ai/wp-content/uploads/2023/06/instruct-video-2-avatar-architecture.jpg" class="attachment-full size-full wp-image-8874 lazyload" alt="IV2A iteratively updates an animatable head avatar. Source: https://arxiv.org/pdf/2306.02903.pdf" srcset="https://blog.metaphysic.ai/wp-content/uploads/2023/06/instruct-video-2-avatar-architecture.jpg 1200w, https://blog.metaphysic.ai/wp-content/uploads/2023/06/instruct-video-2-avatar-architecture-300x132.jpg 300w, https://blog.metaphysic.ai/wp-content/uploads/2023/06/instruct-video-2-avatar-architecture-1024x449.jpg 1024w, https://blog.metaphysic.ai/wp-content/uploads/2023/06/instruct-video-2-avatar-architecture-768x337.jpg 768w" sizes="(max-width: 1200px) 100vw, 1200px"/></noscript><figcaption class="widget-image-caption wp-caption-text">IV2A iteratively updates an animatable head avatar. Source: https://arxiv.org/pdf/2306.02903.pdf</figcaption></figure></div></div></div></div></div></section><section class="elementor-section elementor-top-section elementor-element elementor-element-3e65585 elementor-section-boxed elementor-section-height-default elementor-section-height-default" data-id="3e65585" data-element_type="section"><div class="elementor-container elementor-column-gap-default"><div class="elementor-column elementor-col-100 elementor-top-column elementor-element elementor-element-be1b3e2" data-id="be1b3e2" data-element_type="column"><div class="elementor-widget-wrap elementor-element-populated"><div class="elementor-element elementor-element-2af2647 elementor-widget elementor-widget-text-editor" data-id="2af2647" data-element_type="widget" data-widget_type="text-editor.default"><div class="elementor-widget-container"><p>Though the paper diligently documents the process through formal academic method, there really isn’t a lot more to the system than has been outlined here.</p><p>One additional requirement was to ensure that the output from EbSynth maintains adequate consistency. Though EbSynth takes the previous frame as input for the next, various factors can contrive to warp or deviate from the original design as the tweening continues. Therefore the author performs some additional processing on the EbSynth output. He states:</p></div></div></div></div></div></section><section class="elementor-section elementor-top-section elementor-element elementor-element-3b58134 elementor-section-boxed elementor-section-height-default elementor-section-height-default" data-id="3b58134" data-element_type="section"><div class="elementor-container elementor-column-gap-default"><div class="elementor-column elementor-col-100 elementor-top-column elementor-element elementor-element-c612b91" data-id="c612b91" data-element_type="column"><div class="elementor-widget-wrap elementor-element-populated"><div class="elementor-element elementor-element-2ab4d67 elementor-widget elementor-widget-text-editor" data-id="2ab4d67" data-element_type="widget" data-widget_type="text-editor.default"><div class="elementor-widget-container"><p><em><i>‘For high-quality synthesis, we propose an iterative dataset update. We only edit the sampler image once and execute iterations on other images. In the first training, the editing is carried on the head images from the original video. In the later training cycle, the editing is carried out on the rendered images from the optimized head avatar.’</i></em></p></div></div></div></div></div></section><section class="elementor-section elementor-top-section elementor-element elementor-element-0120df2 elementor-section-boxed elementor-section-height-default elementor-section-height-default" data-id="0120df2" data-element_type="section"><div class="elementor-container elementor-column-gap-default"><div class="elementor-column elementor-col-100 elementor-top-column elementor-element elementor-element-043a407" data-id="043a407" data-element_type="column"><div class="elementor-widget-wrap elementor-element-populated"><div class="elementor-element elementor-element-c389cb5 elementor-widget elementor-widget-text-editor" data-id="c389cb5" data-element_type="widget" data-widget_type="text-editor.default"><div class="elementor-widget-container"><p>In line with more traditional VFX CGI pipelines, the ‘core’ image chosen should ideally be one with the mouth open, since this facial movement cannot be easily inferred from a closed-mouth image, but the mouth can be ‘re-sealed’ as necessary, once the system has some knowledge of the mouth interior (tongue, teeth, facial mouth disposition, etc.).</p><p>The rest of IV2A system relies on the avatar reconstruction abilities of Max Planck’s prior (2023) work INSTA (see video above, and the <a href="https://arxiv.org/pdf/2211.12499v2.pdf">source paper</a>), to effect a NeRF synthesis of the input images.</p></div></div></div></div></div></section><section class="elementor-section elementor-top-section elementor-element elementor-element-32efd73 elementor-section-boxed elementor-section-height-default elementor-section-height-default" data-id="32efd73" data-element_type="section"><div class="elementor-container elementor-column-gap-default"><div class="elementor-column elementor-col-100 elementor-top-column elementor-element elementor-element-085ff90" data-id="085ff90" data-element_type="column"><div class="elementor-widget-wrap elementor-element-populated"><div class="elementor-element elementor-element-3b7f810 elementor-widget elementor-widget-heading" data-id="3b7f810" data-element_type="widget" data-widget_type="heading.default"><div class="elementor-widget-container"><h2 class="elementor-heading-title elementor-size-default">Data and Experiments</h2></div></div></div></div></div></section><section class="elementor-section elementor-top-section elementor-element elementor-element-992a5be elementor-section-boxed elementor-section-height-default elementor-section-height-default" data-id="992a5be" data-element_type="section"><div class="elementor-container elementor-column-gap-default"><div class="elementor-column elementor-col-100 elementor-top-column elementor-element elementor-element-c57b443" data-id="c57b443" data-element_type="column"><div class="elementor-widget-wrap elementor-element-populated"><div class="elementor-element elementor-element-891b557 elementor-widget elementor-widget-text-editor" data-id="891b557" data-element_type="widget" data-widget_type="text-editor.default"><div class="elementor-widget-container"><p>The researcher conducted various qualitative and quantitative experiments of IV2A, comparing it across various methodologies, and also with the use of <a href="https://github.com/Ha0Tang/DAGAN">Dual Attention GAN</a> (DaGAN), using the extracted images from diverse videos as datasets.</p><p>The Windows-only standard EbSynth executable was used for the tweening process, with other experiments carried out on Ubuntu Linux on a NVIDIA 3090 GPU with 24GB of VRAM.</p><p>The method used in the tests were: ‘InstructPix2Pix+One Seed’, wherein a single fixed seed and guidance weights value informed the facial image alteration; ‘ InstructPix2Pix+EbSynth’, wherein the extra rounds of sampling were not carried out; and ‘InstructPix2Pix+DaGAN’, using the prior framework.</p></div></div></div></div></div></section><section class="elementor-section elementor-top-section elementor-element elementor-element-ba74309 elementor-section-boxed elementor-section-height-default elementor-section-height-default" data-id="ba74309" data-element_type="section"><div class="elementor-container elementor-column-gap-default"><div class="elementor-column elementor-col-100 elementor-top-column elementor-element elementor-element-aaff554" data-id="aaff554" data-element_type="column"><div class="elementor-widget-wrap elementor-element-populated"><div class="elementor-element elementor-element-dd60bbd elementor-widget elementor-widget-image" data-id="dd60bbd" data-element_type="widget" data-widget_type="image.default"><div class="elementor-widget-container"><figure class="wp-caption"> <img loading="lazy" decoding="async" width="1200" height="718" src="data:image/gif;base64,R0lGODlhAQABAIAAAAAAAP///yH5BAEAAAAALAAAAAABAAEAAAIBRAA7" data-src="https://blog.metaphysic.ai/wp-content/uploads/2023/06/results.jpg" class="attachment-full size-full wp-image-8875 lazyload" alt="A comparison between IV2A and the nearest available SOTA methods in this pursuit. Please refer to the original paper for better resolution and detail." data-srcset="https://blog.metaphysic.ai/wp-content/uploads/2023/06/results.jpg 1200w, https://blog.metaphysic.ai/wp-content/uploads/2023/06/results-300x180.jpg 300w, https://blog.metaphysic.ai/wp-content/uploads/2023/06/results-1024x613.jpg 1024w, https://blog.metaphysic.ai/wp-content/uploads/2023/06/results-768x460.jpg 768w" sizes="(max-width: 1200px) 100vw, 1200px"/><noscript><img loading="lazy" decoding="async" width="1200" height="718" src="https://blog.metaphysic.ai/wp-content/uploads/2023/06/results.jpg" class="attachment-full size-full wp-image-8875 lazyload" alt="A comparison between IV2A and the nearest available SOTA methods in this pursuit. Please refer to the original paper for better resolution and detail." srcset="https://blog.metaphysic.ai/wp-content/uploads/2023/06/results.jpg 1200w, https://blog.metaphysic.ai/wp-content/uploads/2023/06/results-300x180.jpg 300w, https://blog.metaphysic.ai/wp-content/uploads/2023/06/results-1024x613.jpg 1024w, https://blog.metaphysic.ai/wp-content/uploads/2023/06/results-768x460.jpg 768w" sizes="(max-width: 1200px) 100vw, 1200px"/></noscript><figcaption class="widget-image-caption wp-caption-text">A comparison between IV2A and the nearest available SOTA methods in this pursuit. Please refer to the original paper for better resolution and detail.</figcaption></figure></div></div></div></div></div></section><section class="elementor-section elementor-top-section elementor-element elementor-element-89f6c4f elementor-section-boxed elementor-section-height-default elementor-section-height-default" data-id="89f6c4f" data-element_type="section"><div class="elementor-container elementor-column-gap-default"><div class="elementor-column elementor-col-100 elementor-top-column elementor-element elementor-element-adb3d1a" data-id="adb3d1a" data-element_type="column"><div class="elementor-widget-wrap elementor-element-populated"><div class="elementor-element elementor-element-e25b5d8 elementor-widget elementor-widget-text-editor" data-id="e25b5d8" data-element_type="widget" data-widget_type="text-editor.default"><div class="elementor-widget-container"><p>We can observe in the left section of the second row from the top that the single-seed method does not accomplish the text instruction to convert the image to an anime style, rather producing a more deepfake-style effect, while the iterative method (bottom row) appears to most faithfully adhere to the prompt.</p><p>In terms of temporal stability, the paper refers to supplementary videos which may not yet be available, or may be the animated GIFs supplied at the project site (which have been concatenated in their entirety for this article). We have reached out to the author for access to any additional information, and for clarification, but have not heard back at this time.</p><p>Regarding one particular section of these tests, the author states:</p></div></div></div></div></div></section><section class="elementor-section elementor-top-section elementor-element elementor-element-a5f9457 elementor-section-boxed elementor-section-height-default elementor-section-height-default" data-id="a5f9457" data-element_type="section"><div class="elementor-container elementor-column-gap-default"><div class="elementor-column elementor-col-100 elementor-top-column elementor-element elementor-element-7cf7ded" data-id="7cf7ded" data-element_type="column"><div class="elementor-widget-wrap elementor-element-populated"><div class="elementor-element elementor-element-ba53f06 elementor-widget elementor-widget-text-editor" data-id="ba53f06" data-element_type="widget" data-widget_type="text-editor.default"><div class="elementor-widget-container"><p><em><i>‘With DaGAN, the edited image consistency increases a lot. But the image quality is inferior and there are significant inconsistencies before and after editing. For example, ”The Hulk” can hardly open his mouth and the eyes of the ”17 years old man” open unexpectedly. </i></em></p><p><em><i>‘With EbSynth, the edited images are sharpest with good quality and are consistent with the original images. Some noises exist in the edited results. For example, there are noises in the mouth of the ”anime man”. Our method produces images with good quality. Some shadow noises exist around the avatar head, which are caused by the radiance field. The mouth expressions vary with DaGAN, EbSynth, and our method.’</i></em></p></div></div></div></div></div></section><section class="elementor-section elementor-top-section elementor-element elementor-element-c4f110a elementor-section-boxed elementor-section-height-default elementor-section-height-default" data-id="c4f110a" data-element_type="section"><div class="elementor-container elementor-column-gap-default"><div class="elementor-column elementor-col-100 elementor-top-column elementor-element elementor-element-de7c009" data-id="de7c009" data-element_type="column"><div class="elementor-widget-wrap elementor-element-populated"><div class="elementor-element elementor-element-9e30e16 elementor-widget elementor-widget-text-editor" data-id="9e30e16" data-element_type="widget" data-widget_type="text-editor.default"><div class="elementor-widget-container"><p>The paper emphasizes the importance of the aforementioned iterative updates when processing the facial images, and provide comparisons to illustrate the effect of this:</p></div></div></div></div></div></section><section class="elementor-section elementor-top-section elementor-element elementor-element-a1da340 elementor-section-boxed elementor-section-height-default elementor-section-height-default" data-id="a1da340" data-element_type="section"><div class="elementor-container elementor-column-gap-default"><div class="elementor-column elementor-col-100 elementor-top-column elementor-element elementor-element-07fdacc" data-id="07fdacc" data-element_type="column"><div class="elementor-widget-wrap elementor-element-populated"><div class="elementor-element elementor-element-18f3ae3 elementor-widget elementor-widget-image" data-id="18f3ae3" data-element_type="widget" data-widget_type="image.default"><div class="elementor-widget-container"><figure class="wp-caption"> <img loading="lazy" decoding="async" width="1200" height="735" src="data:image/gif;base64,R0lGODlhAQABAIAAAAAAAP///yH5BAEAAAAALAAAAAABAAEAAAIBRAA7" data-src="https://blog.metaphysic.ai/wp-content/uploads/2023/06/updates-difference.jpg" class="attachment-full size-full wp-image-8876 lazyload" alt="The effects of updating the facial synthesis across various methods." data-srcset="https://blog.metaphysic.ai/wp-content/uploads/2023/06/updates-difference.jpg 1200w, https://blog.metaphysic.ai/wp-content/uploads/2023/06/updates-difference-300x184.jpg 300w, https://blog.metaphysic.ai/wp-content/uploads/2023/06/updates-difference-1024x627.jpg 1024w, https://blog.metaphysic.ai/wp-content/uploads/2023/06/updates-difference-768x470.jpg 768w" sizes="(max-width: 1200px) 100vw, 1200px"/><noscript><img loading="lazy" decoding="async" width="1200" height="735" src="https://blog.metaphysic.ai/wp-content/uploads/2023/06/updates-difference.jpg" class="attachment-full size-full wp-image-8876 lazyload" alt="The effects of updating the facial synthesis across various methods." srcset="https://blog.metaphysic.ai/wp-content/uploads/2023/06/updates-difference.jpg 1200w, https://blog.metaphysic.ai/wp-content/uploads/2023/06/updates-difference-300x184.jpg 300w, https://blog.metaphysic.ai/wp-content/uploads/2023/06/updates-difference-1024x627.jpg 1024w, https://blog.metaphysic.ai/wp-content/uploads/2023/06/updates-difference-768x470.jpg 768w" sizes="(max-width: 1200px) 100vw, 1200px"/></noscript><figcaption class="widget-image-caption wp-caption-text">The effects of updating the facial synthesis across various methods.</figcaption></figure></div></div></div></div></div></section><section class="elementor-section elementor-top-section elementor-element elementor-element-2d5fb49 elementor-section-boxed elementor-section-height-default elementor-section-height-default" data-id="2d5fb49" data-element_type="section"><div class="elementor-container elementor-column-gap-default"><div class="elementor-column elementor-col-100 elementor-top-column elementor-element elementor-element-2d2a421" data-id="2d2a421" data-element_type="column"><div class="elementor-widget-wrap elementor-element-populated"><div class="elementor-element elementor-element-fc0a903 elementor-widget elementor-widget-text-editor" data-id="fc0a903" data-element_type="widget" data-widget_type="text-editor.default"><div class="elementor-widget-container"><p>The author additionally carried out a user study, wherein 20 participants were asked to score 10 edited videos (not as yet published) demonstrating all the aforementioned methods. Regarding these results, the author states:</p></div></div></div></div></div></section><section class="elementor-section elementor-top-section elementor-element elementor-element-3595026 elementor-section-boxed elementor-section-height-default elementor-section-height-default" data-id="3595026" data-element_type="section"><div class="elementor-container elementor-column-gap-default"><div class="elementor-column elementor-col-100 elementor-top-column elementor-element elementor-element-1d3d97c" data-id="1d3d97c" data-element_type="column"><div class="elementor-widget-wrap elementor-element-populated"><div class="elementor-element elementor-element-a3971da elementor-widget elementor-widget-text-editor" data-id="a3971da" data-element_type="widget" data-widget_type="text-editor.default"><div class="elementor-widget-container"><p><em><i>‘For “High Definition”, InstructPix2Pix+EbSynth gets the highest score and ours is the second-highest. For “Temporal Consistency”, ours gets the highest score and One time Dataset Update with EbSynth is the second-highest.’</i></em></p></div></div></div></div></div></section><section class="elementor-section elementor-top-section elementor-element elementor-element-c9d4e23 elementor-section-boxed elementor-section-height-default elementor-section-height-default" data-id="c9d4e23" data-element_type="section"><div class="elementor-container elementor-column-gap-default"><div class="elementor-column elementor-col-100 elementor-top-column elementor-element elementor-element-eb5337d" data-id="eb5337d" data-element_type="column"><div class="elementor-widget-wrap elementor-element-populated"><div class="elementor-element elementor-element-8727693 elementor-widget elementor-widget-image" data-id="8727693" data-element_type="widget" data-widget_type="image.default"><div class="elementor-widget-container"><figure class="wp-caption"> <img loading="lazy" decoding="async" width="1000" height="259" src="data:image/gif;base64,R0lGODlhAQABAIAAAAAAAP///yH5BAEAAAAALAAAAAABAAEAAAIBRAA7" data-src="https://blog.metaphysic.ai/wp-content/uploads/2023/06/user-study.jpg" class="attachment-full size-full wp-image-8877 lazyload" alt="Results of the user study." data-srcset="https://blog.metaphysic.ai/wp-content/uploads/2023/06/user-study.jpg 1000w, https://blog.metaphysic.ai/wp-content/uploads/2023/06/user-study-300x78.jpg 300w, https://blog.metaphysic.ai/wp-content/uploads/2023/06/user-study-768x199.jpg 768w" sizes="(max-width: 1000px) 100vw, 1000px"/><noscript><img loading="lazy" decoding="async" width="1000" height="259" src="https://blog.metaphysic.ai/wp-content/uploads/2023/06/user-study.jpg" class="attachment-full size-full wp-image-8877 lazyload" alt="Results of the user study." srcset="https://blog.metaphysic.ai/wp-content/uploads/2023/06/user-study.jpg 1000w, https://blog.metaphysic.ai/wp-content/uploads/2023/06/user-study-300x78.jpg 300w, https://blog.metaphysic.ai/wp-content/uploads/2023/06/user-study-768x199.jpg 768w" sizes="(max-width: 1000px) 100vw, 1000px"/></noscript><figcaption class="widget-image-caption wp-caption-text">Results of the user study.</figcaption></figure></div></div></div></div></div></section><section class="elementor-section elementor-top-section elementor-element elementor-element-a87391b elementor-section-boxed elementor-section-height-default elementor-section-height-default" data-id="a87391b" data-element_type="section"><div class="elementor-container elementor-column-gap-default"><div class="elementor-column elementor-col-100 elementor-top-column elementor-element elementor-element-e877df8" data-id="e877df8" data-element_type="column"><div class="elementor-widget-wrap elementor-element-populated"><div class="elementor-element elementor-element-8072cec elementor-widget elementor-widget-text-editor" data-id="8072cec" data-element_type="widget" data-widget_type="text-editor.default"><div class="elementor-widget-container"><p>In ablation tests, the study found that EbSynth ‘significantly enhances’ consistency in per-frame editing results, and that the 3x refinement process notably improves the quality of output.</p><p>The author suggests that the method proposed can be extended eventually into an effective pipeline for arbitrary video editing, in which text-to-image instructions could be used to directly manipulate rasterized video content with one (admittedly resource-intensive) pass through the neural pipeline indicated in the new work.</p></div></div></div></div></div></section><section class="elementor-section elementor-top-section elementor-element elementor-element-d7b5ef9 elementor-section-boxed elementor-section-height-default elementor-section-height-default" data-id="d7b5ef9" data-element_type="section"><div class="elementor-container elementor-column-gap-default"><div class="elementor-column elementor-col-100 elementor-top-column elementor-element elementor-element-6e96cea" data-id="6e96cea" data-element_type="column"><div class="elementor-widget-wrap elementor-element-populated"><div class="elementor-element elementor-element-6cf321d elementor-widget elementor-widget-heading" data-id="6cf321d" data-element_type="widget" data-widget_type="heading.default"><div class="elementor-widget-container"><h2 class="elementor-heading-title elementor-size-default">Conclusion</h2></div></div></div></div></div></section><section class="elementor-section elementor-top-section elementor-element elementor-element-90b4d04 elementor-section-boxed elementor-section-height-default elementor-section-height-default" data-id="90b4d04" data-element_type="section"><div class="elementor-container elementor-column-gap-default"><div class="elementor-column elementor-col-100 elementor-top-column elementor-element elementor-element-f14d5eb" data-id="f14d5eb" data-element_type="column"><div class="elementor-widget-wrap elementor-element-populated"><div class="elementor-element elementor-element-1bc1385 elementor-widget elementor-widget-text-editor" data-id="1bc1385" data-element_type="widget" data-widget_type="text-editor.default"><div class="elementor-widget-container"><p>The main takeaway from this paper is the leveraging of EbSynth’s algorithmic (rather than AI-based) tweening method as way of tackling Stable Diffusion’s shortcomings in terms of temporal stability. The additional use of NeRF, which is naturally stable ( since it is effectively a neural analog to older CGI approaches), indicates the possible extent of the growing desperation of the research community.</p></div></div></div></div></div></section><section class="elementor-section elementor-top-section elementor-element elementor-element-c8f2014 elementor-section-boxed elementor-section-height-default elementor-section-height-default" data-id="c8f2014" data-element_type="section"><div class="elementor-container elementor-column-gap-default"><div class="elementor-column elementor-col-100 elementor-top-column elementor-element elementor-element-51e7ccb" data-id="51e7ccb" data-element_type="column"><div class="elementor-widget-wrap elementor-element-populated"><div class="elementor-element elementor-element-ec1208a elementor-widget__width-initial elementor-widget elementor-widget-video" data-id="ec1208a" data-element_type="widget" data-settings="{&quot;video_type&quot;:&quot;hosted&quot;,&quot;autoplay&quot;:&quot;yes&quot;,&quot;play_on_mobile&quot;:&quot;yes&quot;,&quot;mute&quot;:&quot;yes&quot;,&quot;loop&quot;:&quot;yes&quot;,&quot;controls&quot;:&quot;yes&quot;}" data-widget_type="video.default"><div class="elementor-widget-container"><div class="e-hosted-video elementor-wrapper elementor-open-inline"><video class="elementor-video lazyload" data-src="https://blog.metaphysic.ai/wp-content/uploads/2023/06/Ageing.mp4" autoplay="" loop="" controls="" muted="muted" playsinline="" controlslist="nodownload"/></div></div></div></div></div></div></section><section class="elementor-section elementor-top-section elementor-element elementor-element-6e605c7 elementor-section-boxed elementor-section-height-default elementor-section-height-default" data-id="6e605c7" data-element_type="section"><div class="elementor-container elementor-column-gap-default"><div class="elementor-column elementor-col-100 elementor-top-column elementor-element elementor-element-c56ccad" data-id="c56ccad" data-element_type="column"><div class="elementor-widget-wrap elementor-element-populated"><div class="elementor-element elementor-element-428327b elementor-widget elementor-widget-text-editor" data-id="428327b" data-element_type="widget" data-widget_type="text-editor.default"><div class="elementor-widget-container"><p><span style="color: #999999;"><em>Ageing a subject using the new system.</em></span></p></div></div></div></div></div></section><section class="elementor-section elementor-top-section elementor-element elementor-element-6ffb7c2 elementor-section-boxed elementor-section-height-default elementor-section-height-default" data-id="6ffb7c2" data-element_type="section"><div class="elementor-container elementor-column-gap-default"><div class="elementor-column elementor-col-100 elementor-top-column elementor-element elementor-element-dbe7ede" data-id="dbe7ede" data-element_type="column"><div class="elementor-widget-wrap elementor-element-populated"><div class="elementor-element elementor-element-018f82d elementor-widget elementor-widget-text-editor" data-id="018f82d" data-element_type="widget" data-widget_type="text-editor.default"><div class="elementor-widget-container"><p>It seems hard to believe that a generative system as powerful as Latent Diffusion Models (LDMs) cannot be coaxed, in some more intrinsic and fundamental way, into producing temporally coherent output. Yet all indications are that Stable Diffusion and similar LDM-based systems will need to rely on secondary technologies to perform this functionality, which will in effect turn LDMs into mere skinning or texture-based content systems for entirely discrete temporal systems.</p><p>This unsatisfactory state of affairs is essentially a repeat of the slow and disappointing process by which the community eventually came to realize that the similarly astounding reconstructive potential of GANs could not be made to produce coherent movement and temporal consistency without adjunct and ‘bolt on’ technologies such as <a href="https://blog.metaphysic.ai/3d-morphable-models-3dmms/">3DMM, SMPL</a>, and various other CGI-based methods.</p><p>It could be that the emergent Studio 1.0 version of EbSynth will enable similar projects to this one to be able to construct entirely Linux-based VFX pipelines that use algorithmic tweening to produce consistent results, leading to less clunky, multi-platform methodologies.</p><p>As it stands, the output of new text-to-video systems continues to leverage the same kind of ‘cheap tricks’ that proponents of autoencoder-based deepfakes used for years, to make the systems seem more capable and versatile than they really were.</p><p>For instance, in the past week, RunwayML has <a href="https://www.linkedin.com/feed/update/urn:li:activity:7072483570841210881/">dazzled less sophisticated users</a> with a new video demonstrating the power of its LDM T2V generative system, and a clip showing extraordinary transformations occurring within a Scorsese-style POV shot:</p></div></div></div></div></div></section><section class="elementor-section elementor-top-section elementor-element elementor-element-5c87573 elementor-section-boxed elementor-section-height-default elementor-section-height-default" data-id="5c87573" data-element_type="section"><div class="elementor-container elementor-column-gap-default"><div class="elementor-column elementor-col-100 elementor-top-column elementor-element elementor-element-f99b661" data-id="f99b661" data-element_type="column"><div class="elementor-widget-wrap elementor-element-populated"><div class="elementor-element elementor-element-4cf679e elementor-widget elementor-widget-menu-anchor" data-id="4cf679e" data-element_type="widget" data-widget_type="menu-anchor.default"><div class="elementor-widget-container"><div class="elementor-menu-anchor" id="runway"/></div></div></div></div></div></section><section class="elementor-section elementor-top-section elementor-element elementor-element-41121ba elementor-section-boxed elementor-section-height-default elementor-section-height-default" data-id="41121ba" data-element_type="section"><div class="elementor-container elementor-column-gap-default"><div class="elementor-column elementor-col-100 elementor-top-column elementor-element elementor-element-14ad35b" data-id="14ad35b" data-element_type="column"><div class="elementor-widget-wrap elementor-element-populated"><div class="elementor-element elementor-element-0cbc6a0 elementor-widget__width-initial elementor-widget elementor-widget-video" data-id="0cbc6a0" data-element_type="widget" data-settings="{&quot;video_type&quot;:&quot;hosted&quot;,&quot;autoplay&quot;:&quot;yes&quot;,&quot;play_on_mobile&quot;:&quot;yes&quot;,&quot;mute&quot;:&quot;yes&quot;,&quot;loop&quot;:&quot;yes&quot;,&quot;controls&quot;:&quot;yes&quot;}" data-widget_type="video.default"><div class="elementor-widget-container"><div class="e-hosted-video elementor-wrapper elementor-open-inline"><video class="elementor-video lazyload" data-src="https://blog.metaphysic.ai/wp-content/uploads/2023/06/runway.mp4" autoplay="" loop="" controls="" muted="muted" playsinline="" controlslist="nodownload"/></div></div></div></div></div></div></section><section class="elementor-section elementor-top-section elementor-element elementor-element-3450203 elementor-section-boxed elementor-section-height-default elementor-section-height-default" data-id="3450203" data-element_type="section"><div class="elementor-container elementor-column-gap-default"><div class="elementor-column elementor-col-100 elementor-top-column elementor-element elementor-element-92478a9" data-id="92478a9" data-element_type="column"><div class="elementor-widget-wrap elementor-element-populated"><div class="elementor-element elementor-element-b315281 elementor-widget elementor-widget-text-editor" data-id="b315281" data-element_type="widget" data-widget_type="text-editor.default"><div class="elementor-widget-container"><p><span style="color: #999999;"><em>Runway’s promoted generative video, created solely with the Runway Gen1 system.</em> </span></p></div></div></div></div></div></section><section class="elementor-section elementor-top-section elementor-element elementor-element-44a2bc8 elementor-section-boxed elementor-section-height-default elementor-section-height-default" data-id="44a2bc8" data-element_type="section"><div class="elementor-container elementor-column-gap-default"><div class="elementor-column elementor-col-100 elementor-top-column elementor-element elementor-element-958fa27" data-id="958fa27" data-element_type="column"><div class="elementor-widget-wrap elementor-element-populated"><div class="elementor-element elementor-element-e7f1c7b elementor-widget elementor-widget-spacer" data-id="e7f1c7b" data-element_type="widget" data-widget_type="spacer.default"><div class="elementor-widget-container"><div class="elementor-spacer"><div class="elementor-spacer-inner"/></div></div></div></div></div></div></section><section class="elementor-section elementor-top-section elementor-element elementor-element-d3b8d8f elementor-section-boxed elementor-section-height-default elementor-section-height-default" data-id="d3b8d8f" data-element_type="section"><div class="elementor-container elementor-column-gap-default"><div class="elementor-column elementor-col-100 elementor-top-column elementor-element elementor-element-d42037e" data-id="d42037e" data-element_type="column"><div class="elementor-widget-wrap elementor-element-populated"><div class="elementor-element elementor-element-a19355b elementor-widget elementor-widget-text-editor" data-id="a19355b" data-element_type="widget" data-widget_type="text-editor.default"><div class="elementor-widget-container"><p>However, many users have observed that this impressive shot pre-constrains the generative requirements by keeping the person in the picture practically unmoving throughout – a shortcoming that’s common to EbSynth itself, which does not handle ‘wild’ or fast motion very well, since it is difficult to guess a successive transformed frame from a prior frame that is radically different, in terms of positioning.</p><p>Thus, we haven’t necessarily got further than those cheap tricks yet, either in terms of the new paper or the latest T2V demos. The fundamental capabilities are missing from the target system; therefore they are inevitably also missing in the output.</p></div></div></div></div></div></section></div></div></div><div class="elementor-element elementor-element-1edbe8a6 elementor-post-navigation-borders-yes elementor-widget elementor-widget-post-navigation" data-id="1edbe8a6" data-element_type="widget" data-widget_type="post-navigation.default"><div class="elementor-widget-container"><div class="elementor-post-navigation"><div class="elementor-post-navigation__prev elementor-post-navigation__link"> <a href="https://blog.metaphysic.ai/improving-human-pose-extraction-with-transformers/" rel="prev"><span class="post-navigation__arrow-wrapper post-navigation__arrow-prev"><i class="fa fa-arrow-left" aria-hidden="true"/><span class="elementor-screen-only">Prev</span></span><span class="elementor-post-navigation__link__prev"><span class="post-navigation__prev--label">Previous</span><span class="post-navigation__prev--title">Improving Human Pose Extraction With Transformers</span></span></a></div><div class="elementor-post-navigation__separator-wrapper"><div class="elementor-post-navigation__separator"/></div><div class="elementor-post-navigation__next elementor-post-navigation__link"> <a href="https://blog.metaphysic.ai/combating-identity-bleed-in-deepfakes/" rel="next"><span class="elementor-post-navigation__link__next"><span class="post-navigation__next--label">Next</span><span class="post-navigation__next--title">Combating ‘Identity Bleed’ in Deepfakes</span></span><span class="post-navigation__arrow-wrapper post-navigation__arrow-next"><i class="fa fa-arrow-right" aria-hidden="true"/><span class="elementor-screen-only">Next</span></span></a></div></div></div></div><div class="elementor-element elementor-element-5e5b1fa8 elementor-widget elementor-widget-heading" data-id="5e5b1fa8" data-element_type="widget" data-widget_type="heading.default"><div class="elementor-widget-container"><h2 class="elementor-heading-title elementor-size-default">More To Explore</h2></div></div><div class="elementor-element elementor-element-63f62bf8 elementor-grid-2 elementor-posts--align-center elementor-grid-tablet-2 elementor-grid-mobile-1 elementor-posts--thumbnail-top elementor-card-shadow-yes elementor-posts__hover-gradient elementor-widget elementor-widget-posts" data-id="63f62bf8" data-element_type="widget" data-settings="{&quot;cards_columns&quot;:&quot;2&quot;,&quot;cards_row_gap&quot;:{&quot;unit&quot;:&quot;px&quot;,&quot;size&quot;:&quot;30&quot;,&quot;sizes&quot;:[]},&quot;cards_columns_tablet&quot;:&quot;2&quot;,&quot;cards_columns_mobile&quot;:&quot;1&quot;,&quot;cards_row_gap_tablet&quot;:{&quot;unit&quot;:&quot;px&quot;,&quot;size&quot;:&quot;&quot;,&quot;sizes&quot;:[]},&quot;cards_row_gap_mobile&quot;:{&quot;unit&quot;:&quot;px&quot;,&quot;size&quot;:&quot;&quot;,&quot;sizes&quot;:[]}}" data-widget_type="posts.cards"><div class="elementor-widget-container"><div class="elementor-posts-container elementor-posts elementor-posts--skin-cards elementor-grid"><article class="elementor-post elementor-grid-item post-16565 post type-post status-publish format-standard has-post-thumbnail hentry category-ai-machinelearning-deeplearning"><div class="elementor-post__card"> <a class="elementor-post__thumbnail__link" href="https://blog.metaphysic.ai/improving-pose-estimation-for-generative-ai/" tabindex="-1"><div class="elementor-post__thumbnail"><img loading="lazy" width="300" height="189" src="https://blog.metaphysic.ai/wp-content/uploads/2024/06/stable-pose-MAIN-300x189.jpg" class="attachment-medium size-medium wp-image-16567" alt="Sources: https://unsplash.com/photos/a-man-in-a-blue-shirt-is-doing-a-yoga-pose-ak3bgRHt3zY | https://huggingface.co/spaces/hysts/mediapipe-pose-estimation" decoding="async" srcset="https://blog.metaphysic.ai/wp-content/uploads/2024/06/stable-pose-MAIN-300x189.jpg 300w, https://blog.metaphysic.ai/wp-content/uploads/2024/06/stable-pose-MAIN-768x483.jpg 768w, https://blog.metaphysic.ai/wp-content/uploads/2024/06/stable-pose-MAIN.jpg 818w" sizes="(max-width: 300px) 100vw, 300px"/></div></a><div class="elementor-post__badge">AI ML DL</div><div class="elementor-post__text"><h3 class="elementor-post__title"> <a href="https://blog.metaphysic.ai/improving-pose-estimation-for-generative-ai/"> Improving Pose Estimation for Generative AI </a></h3><div class="elementor-post__excerpt"><p>Turning human poses into skeletal stick-figures and back into new images via generative AI is fraught with pitfalls, particularly when the pose in question is uncommon, taken from an unusual angle, or in some way ‘out of distribution’ for what the target generative system is expecting. Among systems that perform these tasks for Stable Diffusion, ControlNet’s openpose module has become very popular in the last year or so – but new research has improved on it, bringing us nearer to the dream of purely generative video generation.</p></div></div><div class="elementor-post__meta-data"> <span class="elementor-post-author"> Martin Anderson </span> <span class="elementor-post-date"> June 6, 2024 </span></div></div></article><article class="elementor-post elementor-grid-item post-16491 post type-post status-publish format-standard has-post-thumbnail hentry category-ai-machinelearning-deeplearning"><div class="elementor-post__card"> <a class="elementor-post__thumbnail__link" href="https://blog.metaphysic.ai/better-human-facial-synthesis-with-gaussian-splatting-and-parametric-heads/" tabindex="-1"><div class="elementor-post__thumbnail"><img loading="lazy" width="300" height="189" src="https://blog.metaphysic.ai/wp-content/uploads/2024/05/ngpa-MAIN-300x189.jpg" class="attachment-medium size-medium wp-image-16493" alt="NPGA: Neural Parametric Gaussian Avatars - https://arxiv.org/pdf/2405.19331" decoding="async" srcset="https://blog.metaphysic.ai/wp-content/uploads/2024/05/ngpa-MAIN-300x189.jpg 300w, https://blog.metaphysic.ai/wp-content/uploads/2024/05/ngpa-MAIN-768x483.jpg 768w, https://blog.metaphysic.ai/wp-content/uploads/2024/05/ngpa-MAIN.jpg 818w" sizes="(max-width: 300px) 100vw, 300px"/></div></a><div class="elementor-post__badge">AI ML DL</div><div class="elementor-post__text"><h3 class="elementor-post__title"> <a href="https://blog.metaphysic.ai/better-human-facial-synthesis-with-gaussian-splatting-and-parametric-heads/"> Better Human Facial Synthesis With Gaussian Splatting and Parametric Heads </a></h3><div class="elementor-post__excerpt"><p>Gaussian Splatting has taken the VFX scene by storm over the last 6-8 months, and new research backed by Synthesia has produced some of the most impressive Splat-based human avatars ever seen, overtaking the state-of-the-art in tests. But is the burden of complexity, in adding neural systems, too high a price to pay for the improvements?</p></div></div><div class="elementor-post__meta-data"> <span class="elementor-post-author"> Martin Anderson </span> <span class="elementor-post-date"> May 31, 2024 </span></div></div></article></div></div></div><div class="elementor-element elementor-element-b80d83a elementor-widget elementor-widget-spacer" data-id="b80d83a" data-element_type="widget" data-widget_type="spacer.default"><div class="elementor-widget-container"><div class="elementor-spacer"><div class="elementor-spacer-inner"/></div></div></div></div></div></div></section><section class="elementor-section elementor-top-section elementor-element elementor-element-272797e elementor-section-boxed elementor-section-height-default elementor-section-height-default" data-id="272797e" data-element_type="section" data-settings="{&quot;background_background&quot;:&quot;gradient&quot;}"><div class="elementor-container elementor-column-gap-default"><div class="elementor-column elementor-col-100 elementor-top-column elementor-element elementor-element-ef6d222" data-id="ef6d222" data-element_type="column"><div class="elementor-widget-wrap elementor-element-populated"><div class="elementor-element elementor-element-43cfd11 elementor-widget elementor-widget-heading" data-id="43cfd11" data-element_type="widget" data-widget_type="heading.default"><div class="elementor-widget-container"><div class="elementor-heading-title elementor-size-default">“</div></div></div><div class="elementor-element elementor-element-fedf26a elementor-widget elementor-widget-heading" data-id="fedf26a" data-element_type="widget" data-widget_type="heading.default"><div class="elementor-widget-container"><h2 class="elementor-heading-title elementor-size-default">It is the mark of an educated mind to be able to entertain a thought without accepting it.</h2></div></div><div class="elementor-element elementor-element-5e09cfc elementor-widget elementor-widget-heading" data-id="5e09cfc" data-element_type="widget" data-widget_type="heading.default"><div class="elementor-widget-container"><p class="elementor-heading-title elementor-size-default">Aristotle</p></div></div></div></div></div></section></div><div data-elementor-type="footer" data-elementor-id="655" class="elementor elementor-655 elementor-location-footer" data-elementor-post-type="elementor_library"><section class="elementor-section elementor-top-section elementor-element elementor-element-514bf1de elementor-section-boxed elementor-section-height-default elementor-section-height-default" data-id="514bf1de" data-element_type="section"><div class="elementor-container elementor-column-gap-default"><div class="elementor-column elementor-col-25 elementor-top-column elementor-element elementor-element-515bf84f" data-id="515bf84f" data-element_type="column"><div class="elementor-widget-wrap elementor-element-populated"><div class="elementor-element elementor-element-77f7638 elementor-invisible elementor-widget elementor-widget-image" data-id="77f7638" data-element_type="widget" data-settings="{&quot;_animation&quot;:&quot;bounce&quot;}" data-widget_type="image.default"><div class="elementor-widget-container"> <img loading="lazy" width="111" height="110" src="https://blog.metaphysic.ai/wp-content/uploads/2021/06/Vector-2.png" class="attachment-large size-large wp-image-1510" alt=""/></div></div><div class="elementor-element elementor-element-f3685d7 elementor-widget elementor-widget-image" data-id="f3685d7" data-element_type="widget" data-widget_type="image.default"><div class="elementor-widget-container"> <img loading="lazy" width="1253" height="101" src="https://blog.metaphysic.ai/wp-content/uploads/2021/06/Metaphysic_Logo-Lockup_Black_RGB.png" class="attachment-full size-full wp-image-651" alt="" srcset="https://blog.metaphysic.ai/wp-content/uploads/2021/06/Metaphysic_Logo-Lockup_Black_RGB.png 1253w, https://blog.metaphysic.ai/wp-content/uploads/2021/06/Metaphysic_Logo-Lockup_Black_RGB-300x24.png 300w, https://blog.metaphysic.ai/wp-content/uploads/2021/06/Metaphysic_Logo-Lockup_Black_RGB-1024x83.png 1024w, https://blog.metaphysic.ai/wp-content/uploads/2021/06/Metaphysic_Logo-Lockup_Black_RGB-768x62.png 768w" sizes="(max-width: 1253px) 100vw, 1253px"/></div></div><div class="elementor-element elementor-element-5d9e3b02 elementor-widget elementor-widget-text-editor" data-id="5d9e3b02" data-element_type="widget" data-widget_type="text-editor.default"><div class="elementor-widget-container"><p>Copyright © 2023. All rights reserved.<br/><a href="https://blog.metaphysic.ai/privacy-policy/">Privacy Policy</a></p></div></div></div></div><div class="elementor-column elementor-col-25 elementor-top-column elementor-element elementor-element-1efd8e56" data-id="1efd8e56" data-element_type="column"><div class="elementor-widget-wrap elementor-element-populated"><div class="elementor-element elementor-element-40b4a9bf elementor-widget elementor-widget-heading" data-id="40b4a9bf" data-element_type="widget" data-widget_type="heading.default"><div class="elementor-widget-container"><h3 class="elementor-heading-title elementor-size-default">Quick Links</h3></div></div><div class="elementor-element elementor-element-3e8f4956 elementor-mobile-align-center elementor-icon-list--layout-traditional elementor-list-item-link-full_width elementor-widget elementor-widget-icon-list" data-id="3e8f4956" data-element_type="widget" data-widget_type="icon-list.default"><div class="elementor-widget-container"><ul class="elementor-icon-list-items"><li class="elementor-icon-list-item"> <a href="https://metaphysic.ai/"> <span class="elementor-icon-list-text">Home</span> </a></li><li class="elementor-icon-list-item"> <a href="https://everyany.one" target="_blank"> <span class="elementor-icon-list-text">Every Anyone</span> </a></li><li class="elementor-icon-list-item"> <a href="https://syntheticfutures.org" target="_blank"> <span class="elementor-icon-list-text">Synthetic Futures</span> </a></li></ul></div></div></div></div><div class="elementor-column elementor-col-25 elementor-top-column elementor-element elementor-element-1555be72" data-id="1555be72" data-element_type="column"><div class="elementor-widget-wrap elementor-element-populated"><div class="elementor-element elementor-element-846a2be elementor-widget elementor-widget-heading" data-id="846a2be" data-element_type="widget" data-widget_type="heading.default"><div class="elementor-widget-container"><h3 class="elementor-heading-title elementor-size-default">Connect with us</h3></div></div><div class="elementor-element elementor-element-70e6bfaa elementor-mobile-align-center elementor-icon-list--layout-traditional elementor-list-item-link-full_width elementor-widget elementor-widget-icon-list" data-id="70e6bfaa" data-element_type="widget" data-widget_type="icon-list.default"><div class="elementor-widget-container"><ul class="elementor-icon-list-items"><li class="elementor-icon-list-item"> <a href="https://discord.gg/5vshCNWTuw"> <span class="elementor-icon-list-icon"> <i aria-hidden="true" class="fab fa-discord"/> </span> <span class="elementor-icon-list-text">Discord</span> </a></li><li class="elementor-icon-list-item"> <a href="https://www.tiktok.com/@deeptomcruise"> <span class="elementor-icon-list-icon"> <i aria-hidden="true" class="fab fa-tiktok"/> </span> <span class="elementor-icon-list-text">Tiktok</span> </a></li><li class="elementor-icon-list-item"> <a href="https://twitter.com/Metaphysic_ai"> <span class="elementor-icon-list-icon"> <i aria-hidden="true" class="fab fa-twitter"/> </span> <span class="elementor-icon-list-text">Twitter</span> </a></li><li class="elementor-icon-list-item"> <a href="https://www.youtube.com/channel/UClbSYyDnUCa6NzLjLqPdMoA"> <span class="elementor-icon-list-icon"> <i aria-hidden="true" class="fab fa-youtube"/> </span> <span class="elementor-icon-list-text">Youtube</span> </a></li><li class="elementor-icon-list-item"> <a href="https://www.instagram.com/metaphysic.ai/"> <span class="elementor-icon-list-icon"> <i aria-hidden="true" class="fab fa-instagram"/> </span> <span class="elementor-icon-list-text">Instagram</span> </a></li><li class="elementor-icon-list-item"> <a href="https://github.com/Metaphysic-ai"> <span class="elementor-icon-list-icon"> <i aria-hidden="true" class="icon icon-github"/> </span> <span class="elementor-icon-list-text">Github</span> </a></li><li class="elementor-icon-list-item"> <a href="http://www.linkedin.com/company/metaphysic-ai/"> <span class="elementor-icon-list-icon"> <i aria-hidden="true" class="fab fa-linkedin"/> </span> <span class="elementor-icon-list-text">Linkedin</span> </a></li></ul></div></div></div></div><div class="elementor-column elementor-col-25 elementor-top-column elementor-element elementor-element-675fdec2" data-id="675fdec2" data-element_type="column"><div class="elementor-widget-wrap elementor-element-populated"><div class="elementor-element elementor-element-6fe3527e elementor-widget elementor-widget-heading" data-id="6fe3527e" data-element_type="widget" data-widget_type="heading.default"><div class="elementor-widget-container"><h3 class="elementor-heading-title elementor-size-default">Contact Info</h3></div></div><div class="elementor-element elementor-element-2b928ae8 elementor-mobile-align-center elementor-icon-list--layout-traditional elementor-list-item-link-full_width elementor-widget elementor-widget-icon-list" data-id="2b928ae8" data-element_type="widget" data-widget_type="icon-list.default"><div class="elementor-widget-container"><ul class="elementor-icon-list-items"><li class="elementor-icon-list-item"> <span class="elementor-icon-list-icon"> <i aria-hidden="true" class="fas fa-envelope"/> </span> <span class="elementor-icon-list-text">info@metaphysic.ai</span></li><li class="elementor-icon-list-item"> <span class="elementor-icon-list-icon"> <i aria-hidden="true" class="fas fa-envelope"/> </span> <span class="elementor-icon-list-text">press@metaphysic.ai</span></li></ul></div></div></div></div></div></section></div> </body>