Integrating MongoDB with Debezium and Redpanda for CDC

link管理

链接快照平台

输入网页链接，自动生成快照
标签化管理网页链接

相关文章推荐

仗义的炒面 · TCL file mkdir - CSDN文库· 1 年前 ·

有腹肌的夕阳 · 在缓存和数据库双写场景下，一致性是如何保证的 ...· 2 年前 ·

一直单身的饭盒 · 简单介绍python编程之文件读写_ITPUB博客· 3 年前 ·

讲道义的高山 · PyQt5实现点击菜单栏切换页面（非弹出新窗 ...· 3 年前 ·

刀枪不入的熊猫 · Error in ...· 3 年前 ·

</noscript><div id="__next" data-reactroot=""><div class="styles_Layout__MHDWL" style="padding-top:0"><div class="styles_Header__T_z_x"><div class="styles_Header__banner__5Gg7b">Save your seat for our August 23 Masterclass: <a href="https://go.redpanda.com/masterclass-aug23-2023">Day 2 Operations with Redpanda</a></div><header role="contentinfo" class="styles_DesktopHeader__EY9hX"><a class="styles_DesktopHeader__logo__S6_Zn" aria-label="Red Panda Logo" href="/"><img src="https://images.ctfassets.net/paqvtpyf8rwu/5wODDUzLP7teOGDS7M5m39/466f646e44c2ccc4e78236b17d7d68ac/Logo.svg" alt="" title="Redpanda Logo"/></a><nav class="styles_DesktopHeader__desktopNav__K2ueX"><ul class="styles_DesktopHeader__navList__xVQM2"><li class="styles_DesktopHeader__navItem__8yJw5 mr-0" aria-label="Toggle PLATFORM subnav dropdown"><div><button class="styles_DesktopHeader__navButton__gVr05">PLATFORM<svg width="12" height="7" viewbox="0 0 12 7" fill="none" xmlns="http://www.w3.org/2000/svg" class=""><path d="M1 1L6 6L11 1" stroke="#667085" stroke-width="1.67" stroke-linecap="round" stroke-linejoin="round"/></svg></button><div class="styles_DesktopHeader__subNav__LuUHn"><ul class="styles_DesktopHeader__subNavList__xd4nt"><li class="styles_DesktopHeader__subNavItem__KgS5R"><a class="styles_Header__subNavLink__HCqh7" target="_self" rel="" href="/what-is-redpanda"><div class="styles_Header__subNavLinkTextAndIcon__YD6kl"><img class="styles_Header__subNavLinkIcon__ORxzC" src="https://images.ctfassets.net/paqvtpyf8rwu/5VKIqaX8JhLWcL9mPUnpXg/019d6f9d8594c781ab0cb4401b7c45a7/r-icon.svg" alt="Overview"/><div><div class="styles_Header__subNavLinkGroup__FAyTJ">Overview<div class="styles_Header__subNavLinkArrow___dJm0"><svg width="17" height="13" viewbox="0 0 17 16" fill="none" xmlns="http://www.w3.org/2000/svg"><path d="M10.7077 10.313L9.42138 9.02034L12.4023 6.05306L-6.79749e-05 6.04121L0.00175553 4.218L12.3704 4.22985L9.43961 1.28536L10.7314 -2.249e-07L15.8765 5.16881L10.7077 10.313Z" fill="#E2401B"/></svg></div></div>Technical tour of the Redpanda platform</div></div></a></li><li class="styles_DesktopHeader__subNavItem__KgS5R"><a class="styles_Header__subNavLink__HCqh7" target="_self" rel="" href="/platform-capabilities"><div class="styles_Header__subNavLinkTextAndIcon__YD6kl"><img class="styles_Header__subNavLinkIcon__ORxzC" src="https://images.ctfassets.net/paqvtpyf8rwu/3QzpsfG9Ba6u6Zu14Zmb0X/06873b28b5459a166879833affc7923d/capabilities.svg" alt="Core capabilities"/><div><div class="styles_Header__subNavLinkGroup__FAyTJ">Core capabilities<div class="styles_Header__subNavLinkArrow___dJm0"><svg width="17" height="13" viewbox="0 0 17 16" fill="none" xmlns="http://www.w3.org/2000/svg"><path d="M10.7077 10.313L9.42138 9.02034L12.4023 6.05306L-6.79749e-05 6.04121L0.00175553 4.218L12.3704 4.22985L9.43961 1.28536L10.7314 -2.249e-07L15.8765 5.16881L10.7077 10.313Z" fill="#E2401B"/></svg></div></div>Explore all features and functions</div></div></a></li><li class="styles_DesktopHeader__subNavItem__KgS5R"><a class="styles_Header__subNavLink__HCqh7" target="_self" rel="" href="/platform-tco/"><div class="styles_Header__subNavLinkTextAndIcon__YD6kl"><img class="styles_Header__subNavLinkIcon__ORxzC" src="https://images.ctfassets.net/paqvtpyf8rwu/4xqo30nw0MEDIcpVos5hmD/6e2e148a35c59fa4d85c97f2d8186234/tco-icon.svg" alt="Platform TCO"/><div><div class="styles_Header__subNavLinkGroup__FAyTJ">Platform TCO<div class="styles_Header__subNavLinkArrow___dJm0"><svg width="17" height="13" viewbox="0 0 17 16" fill="none" xmlns="http://www.w3.org/2000/svg"><path d="M10.7077 10.313L9.42138 9.02034L12.4023 6.05306L-6.79749e-05 6.04121L0.00175553 4.218L12.3704 4.22985L9.43961 1.28536L10.7314 -2.249e-07L15.8765 5.16881L10.7077 10.313Z" fill="#E2401B"/></svg></div></div>6x lower TCO - see how</div></div></a></li><li class="styles_DesktopHeader__subNavItem__KgS5R"><a class="styles_Header__subNavLink__HCqh7" target="_self" rel="" href="/redpanda-console-kafka-ui"><div class="styles_Header__subNavLinkTextAndIcon__YD6kl"><img class="styles_Header__subNavLinkIcon__ORxzC" src="https://images.ctfassets.net/paqvtpyf8rwu/2JdeLBOezvJgJ1SNId6FFZ/ba1a95da69557a00cb7c792e862f97ba/console.svg" alt="Redpanda Console"/><div><div class="styles_Header__subNavLinkGroup__FAyTJ">Redpanda Console<div class="styles_Header__subNavLinkArrow___dJm0"><svg width="17" height="13" viewbox="0 0 17 16" fill="none" xmlns="http://www.w3.org/2000/svg"><path d="M10.7077 10.313L9.42138 9.02034L12.4023 6.05306L-6.79749e-05 6.04121L0.00175553 4.218L12.3704 4.22985L9.43961 1.28536L10.7314 -2.249e-07L15.8765 5.16881L10.7077 10.313Z" fill="#E2401B"/></svg></div></div>An interactive web UI for Kafka</div></div></a></li><li class="styles_DesktopHeader__subNavItem__KgS5R"><a class="styles_Header__subNavLink__HCqh7" target="_self" rel="" href="/deployment-options"><div class="styles_Header__subNavLinkTextAndIcon__YD6kl"><img class="styles_Header__subNavLinkIcon__ORxzC" src="https://images.ctfassets.net/paqvtpyf8rwu/4j3qyEo0tUk3PDFqrgUmhe/b7efe7f8ff072a9d31ac8a7638dfb7c0/deployment.svg" alt="Deployment options"/><div><div class="styles_Header__subNavLinkGroup__FAyTJ">Deployment options<div class="styles_Header__subNavLinkArrow___dJm0"><svg width="17" height="13" viewbox="0 0 17 16" fill="none" xmlns="http://www.w3.org/2000/svg"><path d="M10.7077 10.313L9.42138 9.02034L12.4023 6.05306L-6.79749e-05 6.04121L0.00175553 4.218L12.3704 4.22985L9.43961 1.28536L10.7314 -2.249e-07L15.8765 5.16881L10.7077 10.313Z" fill="#E2401B"/></svg></div></div>Compare self-hosted and cloud options</div></div></a></li></ul></div></div></li><li class="styles_DesktopHeader__navItem__8yJw5 mr-0" aria-label="Toggle CLOUD subnav dropdown"><a class="styles_DesktopHeader__navButton__gVr05" href="/redpanda-cloud">CLOUD</a></li><li class="styles_DesktopHeader__navItem__8yJw5 mr-0" aria-label="Toggle CUSTOMERS subnav dropdown"><a class="styles_DesktopHeader__navButton__gVr05" href="/customers">CUSTOMERS</a></li><li class="styles_DesktopHeader__navItem__8yJw5 mr-0" aria-label="Toggle BLOG subnav dropdown"><a class="styles_DesktopHeader__navButton__gVr05" href="/blog">BLOG</a></li><li class="styles_DesktopHeader__navItem__8yJw5 mr-0" aria-label="Toggle LEARN subnav dropdown"><div><button class="styles_DesktopHeader__navButton__gVr05">LEARN<svg width="12" height="7" viewbox="0 0 12 7" fill="none" xmlns="http://www.w3.org/2000/svg" class=""><path d="M1 1L6 6L11 1" stroke="#667085" stroke-width="1.67" stroke-linecap="round" stroke-linejoin="round"/></svg></button><div class="styles_DesktopHeader__subNav__LuUHn"><ul class="styles_DesktopHeader__subNavList__xd4nt"><li class="styles_DesktopHeader__subNavItem__KgS5R"><a class="styles_Header__subNavLink__HCqh7" target="_blank" rel="noopener noreferrer" href="https://docs.redpanda.com/docs/home"><div class="styles_Header__subNavLinkTextAndIcon__YD6kl"><img class="styles_Header__subNavLinkIcon__ORxzC" src="https://images.ctfassets.net/paqvtpyf8rwu/2pOpTO3vvTj6lfkIgOzfqU/9e34792be89009df7cb22f4f599c8f00/docs-icon.svg" alt="Documentation"/><div><div class="styles_Header__subNavLinkGroup__FAyTJ">Documentation<div class="styles_Header__subNavLinkArrow___dJm0"><svg width="17" height="13" viewbox="0 0 17 16" fill="none" xmlns="http://www.w3.org/2000/svg"><path d="M10.7077 10.313L9.42138 9.02034L12.4023 6.05306L-6.79749e-05 6.04121L0.00175553 4.218L12.3704 4.22985L9.43961 1.28536L10.7314 -2.249e-07L15.8765 5.16881L10.7077 10.313Z" fill="#E2401B"/></svg></div></div>Dive deep into features and APIs.</div></div></a></li><li class="styles_DesktopHeader__subNavItem__KgS5R"><a class="styles_Header__subNavLink__HCqh7" target="_blank" rel="noopener noreferrer" href="https://university.redpanda.com"><div class="styles_Header__subNavLinkTextAndIcon__YD6kl"><img class="styles_Header__subNavLinkIcon__ORxzC" src="https://images.ctfassets.net/paqvtpyf8rwu/78Ji8ECFrOzxOqvMWwYdy/403c48279d4a645e63cec6b476c8c761/redpanda-u.svg" alt="Redpanda University"/><div><div class="styles_Header__subNavLinkGroup__FAyTJ">Redpanda University<div class="styles_Header__subNavLinkArrow___dJm0"><svg width="17" height="13" viewbox="0 0 17 16" fill="none" xmlns="http://www.w3.org/2000/svg"><path d="M10.7077 10.313L9.42138 9.02034L12.4023 6.05306L-6.79749e-05 6.04121L0.00175553 4.218L12.3704 4.22985L9.43961 1.28536L10.7314 -2.249e-07L15.8765 5.16881L10.7077 10.313Z" fill="#E2401B"/></svg></div></div>Guided courses on building real-time apps.</div></div></a></li><li class="styles_DesktopHeader__subNavItem__KgS5R"><a class="styles_Header__subNavLink__HCqh7" target="_self" rel="" href="/resources"><div class="styles_Header__subNavLinkTextAndIcon__YD6kl"><img class="styles_Header__subNavLinkIcon__ORxzC" src="https://images.ctfassets.net/paqvtpyf8rwu/5tRzRjoeG3Ny4TwWaF1WAv/41485d30c174cd9753e00d0cb34d3354/resources.svg" alt="On-demand resources"/><div><div class="styles_Header__subNavLinkGroup__FAyTJ">On-demand resources<div class="styles_Header__subNavLinkArrow___dJm0"><svg width="17" height="13" viewbox="0 0 17 16" fill="none" xmlns="http://www.w3.org/2000/svg"><path d="M10.7077 10.313L9.42138 9.02034L12.4023 6.05306L-6.79749e-05 6.04121L0.00175553 4.218L12.3704 4.22985L9.43961 1.28536L10.7314 -2.249e-07L15.8765 5.16881L10.7077 10.313Z" fill="#E2401B"/></svg></div></div>Tech talks, workshops, reports, and more.</div></div></a></li><li class="styles_DesktopHeader__subNavItem__KgS5R"><a class="styles_Header__subNavLink__HCqh7" target="_self" rel="" href="https://redpanda.com/slack"><div class="styles_Header__subNavLinkTextAndIcon__YD6kl"><img class="styles_Header__subNavLinkIcon__ORxzC" src="https://images.ctfassets.net/paqvtpyf8rwu/PMcVPLOUI853NUus1Ofqy/41306403501d2129761387a0291cc60d/slack.svg" alt="Community Slack"/><div><div class="styles_Header__subNavLinkGroup__FAyTJ">Community Slack<div class="styles_Header__subNavLinkArrow___dJm0"><svg width="17" height="13" viewbox="0 0 17 16" fill="none" xmlns="http://www.w3.org/2000/svg"><path d="M10.7077 10.313L9.42138 9.02034L12.4023 6.05306L-6.79749e-05 6.04121L0.00175553 4.218L12.3704 4.22985L9.43961 1.28536L10.7314 -2.249e-07L15.8765 5.16881L10.7077 10.313Z" fill="#E2401B"/></svg></div></div>Ask our dev community your burning questions.</div></div></a></li></ul></div></div></li><li class="styles_DesktopHeader__navItem__8yJw5 xl:mr-12" aria-label="Toggle COMPANY subnav dropdown"><div><button class="styles_DesktopHeader__navButton__gVr05">COMPANY<svg width="12" height="7" viewbox="0 0 12 7" fill="none" xmlns="http://www.w3.org/2000/svg" class=""><path d="M1 1L6 6L11 1" stroke="#667085" stroke-width="1.67" stroke-linecap="round" stroke-linejoin="round"/></svg></button><div class="styles_DesktopHeader__subNav__LuUHn"><ul class="styles_DesktopHeader__subNavList__xd4nt"><li class="styles_DesktopHeader__subNavItem__KgS5R"><a class="styles_Header__subNavLink__HCqh7" target="_self" rel="" href="/about-us/"><div class="styles_Header__subNavLinkTextAndIcon__YD6kl"><img class="styles_Header__subNavLinkIcon__ORxzC" src="https://images.ctfassets.net/paqvtpyf8rwu/5y3RsyLnrJb66KaVzyZ91d/d095abcc315fc62cd052936996a2eae3/about-icon.svg" alt="About us"/><div><div class="styles_Header__subNavLinkGroup__FAyTJ">About us<div class="styles_Header__subNavLinkArrow___dJm0"><svg width="17" height="13" viewbox="0 0 17 16" fill="none" xmlns="http://www.w3.org/2000/svg"><path d="M10.7077 10.313L9.42138 9.02034L12.4023 6.05306L-6.79749e-05 6.04121L0.00175553 4.218L12.3704 4.22985L9.43961 1.28536L10.7314 -2.249e-07L15.8765 5.16881L10.7077 10.313Z" fill="#E2401B"/></svg></div></div>Meet our team and investors.</div></div></a></li><li class="styles_DesktopHeader__subNavItem__KgS5R"><a class="styles_Header__subNavLink__HCqh7" target="_self" rel="" href="/careers"><div class="styles_Header__subNavLinkTextAndIcon__YD6kl"><img class="styles_Header__subNavLinkIcon__ORxzC" src="https://images.ctfassets.net/paqvtpyf8rwu/7wxiMVSCkCzeIlskXUh02h/5c66960f5ae6a43634e68872d64ca243/careers.svg" alt="Careers"/><div><div class="styles_Header__subNavLinkGroup__FAyTJ">Careers<div class="styles_Header__subNavLinkArrow___dJm0"><svg width="17" height="13" viewbox="0 0 17 16" fill="none" xmlns="http://www.w3.org/2000/svg"><path d="M10.7077 10.313L9.42138 9.02034L12.4023 6.05306L-6.79749e-05 6.04121L0.00175553 4.218L12.3704 4.22985L9.43961 1.28536L10.7314 -2.249e-07L15.8765 5.16881L10.7077 10.313Z" fill="#E2401B"/></svg></div></div>Explore Redpanda opportunities and culture.</div></div></a></li></ul></div></div></li><li class="styles_DesktopHeader__navItem__8yJw5 mr-0" aria-label="Toggle CONTACT subnav dropdown"><div><button class="styles_DesktopHeader__navButton__gVr05">CONTACT<svg width="12" height="7" viewbox="0 0 12 7" fill="none" xmlns="http://www.w3.org/2000/svg" class=""><path d="M1 1L6 6L11 1" stroke="#667085" stroke-width="1.67" stroke-linecap="round" stroke-linejoin="round"/></svg></button><div class="styles_DesktopHeader__subNav__LuUHn"><ul class="styles_DesktopHeader__subNavList__xd4nt"><li class="styles_DesktopHeader__subNavItem__KgS5R"><a class="styles_Header__subNavLink__HCqh7" target="_self" rel="" href="/contact"><div class="styles_Header__subNavLinkTextAndIcon__YD6kl"><img class="styles_Header__subNavLinkIcon__ORxzC" src="https://images.ctfassets.net/paqvtpyf8rwu/2kOTXYFsqojwB0lKISpmsA/9983226adbaeb5fa01bf31fd0f18d93a/inquiries.svg" alt="Request demo"/><div><div class="styles_Header__subNavLinkGroup__FAyTJ">Request demo<div class="styles_Header__subNavLinkArrow___dJm0"><svg width="17" height="13" viewbox="0 0 17 16" fill="none" xmlns="http://www.w3.org/2000/svg"><path d="M10.7077 10.313L9.42138 9.02034L12.4023 6.05306L-6.79749e-05 6.04121L0.00175553 4.218L12.3704 4.22985L9.43961 1.28536L10.7314 -2.249e-07L15.8765 5.16881L10.7077 10.313Z" fill="#E2401B"/></svg></div></div>Request private tech talks, pricing, product resources.</div></div></a></li><li class="styles_DesktopHeader__subNavItem__KgS5R"><a class="styles_Header__subNavLink__HCqh7" target="_blank" rel="noopener noreferrer" href="/slack"><div class="styles_Header__subNavLinkTextAndIcon__YD6kl"><img class="styles_Header__subNavLinkIcon__ORxzC" src="https://images.ctfassets.net/paqvtpyf8rwu/PMcVPLOUI853NUus1Ofqy/41306403501d2129761387a0291cc60d/slack.svg" alt="Join our Slack"/><div><div class="styles_Header__subNavLinkGroup__FAyTJ">Join our Slack<div class="styles_Header__subNavLinkArrow___dJm0"><svg width="17" height="13" viewbox="0 0 17 16" fill="none" xmlns="http://www.w3.org/2000/svg"><path d="M10.7077 10.313L9.42138 9.02034L12.4023 6.05306L-6.79749e-05 6.04121L0.00175553 4.218L12.3704 4.22985L9.43961 1.28536L10.7314 -2.249e-07L15.8765 5.16881L10.7077 10.313Z" fill="#E2401B"/></svg></div></div>Get answers from our community.</div></div></a></li><li class="styles_DesktopHeader__subNavItem__KgS5R"><a class="styles_Header__subNavLink__HCqh7" target="_blank" rel="noopener noreferrer" href="https://support.redpanda.com/"><div class="styles_Header__subNavLinkTextAndIcon__YD6kl"><img class="styles_Header__subNavLinkIcon__ORxzC" src="https://images.ctfassets.net/paqvtpyf8rwu/A9rREbqZidrb8bRNQXRk5/1f19c2ebac528c6e7e970481f28c64f3/support-requests.svg" alt="Support request"/><div><div class="styles_Header__subNavLinkGroup__FAyTJ">Support request<div class="styles_Header__subNavLinkArrow___dJm0"><svg width="17" height="13" viewbox="0 0 17 16" fill="none" xmlns="http://www.w3.org/2000/svg"><path d="M10.7077 10.313L9.42138 9.02034L12.4023 6.05306L-6.79749e-05 6.04121L0.00175553 4.218L12.3704 4.22985L9.43961 1.28536L10.7314 -2.249e-07L15.8765 5.16881L10.7077 10.313Z" fill="#E2401B"/></svg></div></div>Submit technical support ticket.</div></div></a></li></ul></div></div></li><li class="styles_DesktopHeader__navItem__8yJw5 mr-0" aria-label="Toggle Start for free subnav dropdown"><a href="/try-redpanda" tabindex="0" role="link" aria-label="Start for free" class="styles_DesktopHeader__button__fD4_O styles_Button__znVlr styles_Button__textCenter__mmDEU styles_Button__Small__Chq2W styles_Button__WhiteTextRedBg__Qn_Zp styles_Button--centered__vjrFc styles_Button--uppercase__lkHlN">Start for free </a></li><li class="styles_DesktopHeader__navItem__8yJw5 mr-0" aria-label="Toggle subnav dropdown"><a class="styles_DesktopHeader__socialLink__4vjjL styles_DesktopHeader__navButton__gVr05" target="_blank" rel="noopener noreferrer" href="https://redpanda.com/slack"><img src="https://images.ctfassets.net/paqvtpyf8rwu/69HyyrugepthrFF25coB86/bbc52835d1aa2c5714b30f62a957133a/slack__1_.svg" alt="Slack (homepage header)"/> </a></li><li class="styles_DesktopHeader__navItem__8yJw5 mr-0" aria-label="Toggle 4.3K subnav dropdown"><a class="styles_DesktopHeader__socialLink__4vjjL styles_DesktopHeader__navButton__gVr05" target="_blank" rel="noopener noreferrer" href="https://github.com/redpanda-data/redpanda/"><img src="https://images.ctfassets.net/paqvtpyf8rwu/6ZoZUb18LuxNlX5qyhD1vI/08a798d68104fe6a7f9b14d7251a45a3/github__1_.svg" alt="Github"/>4.3K</a></li></ul></nav></header><header role="contentinfo" class="styles_MobileHeader__QkweI"><div class="styles_MobileHeader__wrapper__S5jaI"><a aria-label="Red Panda Logo" href="/"><img src="https://images.ctfassets.net/paqvtpyf8rwu/5wODDUzLP7teOGDS7M5m39/466f646e44c2ccc4e78236b17d7d68ac/Logo.svg" alt="" title="Redpanda Logo"/></a><button class="styles_MobileHeader__mobileNavToggle__Jlbag" role="button" tabindex="0" aria-label="Toggle mobile navigation"><svg class="styles_MobileHeader__mobileNavToggleSvg__z4rWs" xmlns="http://www.w3.org/2000/svg" viewbox="0 0 100 100"><g fill="none" fill-rule="evenodd" stroke="#344054" stroke-linecap="square" stroke-width="10"><line x1="9" y1="22" x2="91" y2="22" id="top"/><line x1="9" y1="50" x2="91" y2="50" id="middle"/><line x1="9" y1="78" x2="91" y2="78" id="bottom"/></g></svg></button></div><nav class="styles_MobileHeader__mobileNavListWrapper__rdJoG"/></header></div><main class="styles_Main__Lod2n"><div class="styles_BlogPostTemplate__xziin"><section aria-labelledby="Blog header section." class="styles_SectionContainer__peieG" id="undefined"><div class="styles_SectionContainer__innerContainer__fsrG8 styles_SectionContainer__container--relative__rLVsP styles_SectionContainer__yPaddingOff__5AMpG"><div class="styles_BlogPostTemplate__inlineNavigation__mjkO4"><a class="styles_BlogPostTemplate__inlineNavigationItem__O_5gT" href="/"><img src="/images/home-icon.svg" alt="Red Panda logo"/></a><img src="/images/blog-nav-arrow.svg" alt="Chevron right icon"/><a class="styles_BlogPostTemplate__inlineNavigationItem__O_5gT" href="/blog">Blog</a></div><div class="styles_BlogPostTemplate__header__xETQr"><div class="styles_BlogPostTemplate__headerCategoriesAndTimeRead__Jkjmz"><a class="styles_BlogPostTemplate__headerCategory__gJTTK undefined" href="/blog/categories/tutorial">Tutorial</a><a class="styles_BlogPostTemplate__headerCategory__gJTTK undefined" href="/blog/categories/community">Community</a><a class="styles_BlogPostTemplate__headerCategory__gJTTK undefined" href="/blog/categories/ecosystem">Ecosystem</a></div><h1 class="styles_RenderBreakLineText__nTFTN styles_BlogPostTemplate__headerTitle__tPZrK"/>Learn how to use MongoDB with Debezium and Redpanda for real-time change data capture.<div class="styles_BlogPostTemplate__headerAuthorsAndDate__6EY9N">By<a class="styles_BlogPostTemplate__headerAuthor__xNOCU" href="/blog/authors/jakkie-koekemoer">Jakkie Koekemoer</a>onSeptember 8, 2022</div><div class="styles_BlogPostTemplate__headerImageWrapper__xPKAF"><img src="https://images.ctfassets.net/paqvtpyf8rwu/1tsHm7rgLNrYm0fk3WxrQ5/3da330e2c64ea56a3c1dc7c9e0e48166/redpanda-debezium-mongodb.png" alt="Integrating MongoDB with Debezium and Redpanda for CDC"/></div><div class="styles_BlogPostSocialShare__8KTAp"><div class="styles_BlogPostSocialShare__shareButtons__Zu0HF"><a href="https://github.com/redpanda-data/redpanda/" target="_blank" rel="noopener noreferrer" class="styles_BlogPostSocialShare__shareButton__3UdFX"><img src="/images/share-buttons-icons/github.svg" alt="Social icon github" title="Share on github"/></a><a href="https://redpandacommunity.slack.com/join/shared_invite/zt-ng2ze1uv-l5VMWSGQHB9gp47~kNnYGA#/shared-invite/email" target="_blank" rel="noopener noreferrer" class="styles_BlogPostSocialShare__shareButton__3UdFX"><img src="/images/share-buttons-icons/slack.svg" alt="Social icon slack" title="Share on slack"/></a><a href="http://twitter.com/share?url=https%3A%2F%2Fredpanda.com%2Fblog%2Fchange-data-capture-mongodb-debezium&amp;text=Integrating MongoDB with Debezium and Redpanda for CDC by @redpandadata #redpanda #apachekafka" target="_blank" rel="noopener noreferrer" class="styles_BlogPostSocialShare__shareButton__3UdFX"><img src="/images/share-buttons-icons/twitter.svg" alt="Social icon twitter" title="Share on twitter"/></a><a href="https://www.facebook.com/sharer/sharer.php?u=https%3A%2F%2Fredpanda.com%2Fblog%2Fchange-data-capture-mongodb-debezium" target="_blank" rel="noopener noreferrer" class="styles_BlogPostSocialShare__shareButton__3UdFX"><img src="/images/share-buttons-icons/facebook.svg" alt="Social icon facebook" title="Share on facebook"/></a><a href="https://www.linkedin.com/shareArticle?mini=true&amp;url=https%3A%2F%2Fredpanda.com%2Fblog%2Fchange-data-capture-mongodb-debezium" target="_blank" rel="noopener noreferrer" class="styles_BlogPostSocialShare__shareButton__3UdFX"><img src="/images/share-buttons-icons/linkedin.svg" alt="Social icon linkedin" title="Share on linkedin"/></a></div><div class="styles_BlogPostSocialShare__shareEmailAndCopy__yRire"><a href="mailto:?subject=Integrating MongoDB with Debezium and Redpanda for CDC&amp;body=https%3A%2F%2Fredpanda.com%2Fblog%2Fchange-data-capture-mongodb-debezium" class="styles_BlogPostSocialShare__shareButton__3UdFX"><img src="/images/share-buttons-icons/email.svg" alt="email icon" title="Share by email"/></a><button class="styles_BlogPostSocialShare__shareCopy__OseKa styles_BlogPostSocialShare__shareButton__3UdFX" role="button" aria-label="Copy link to clipboard"><img src="/images/share-buttons-icons/copy.svg" alt="copy icon" title="Copy link"/>Copy link</button></div></div></div></div></section><section aria-labelledby="Blog post content section." class="styles_SectionContainer__peieG styles_BlogPostTemplate__content__wrapper__6DRW_" id="undefined"><div class="styles_SectionContainer__innerContainer__fsrG8 styles_SectionContainer__container--relative__rLVsP styles_SectionContainer__yPaddingOff__5AMpG"><div class="styles_BlogPostTemplate__content__with_sidebar__pzJ5V"><div class="col-span-2"><div class="styles_Marked__pncG4 styles_Marked__container__T_BSl">Data sensitivity is becoming more and more important in the cloud-native era. Data is the key to success for any business, whether it is private user information that needs to be secured or metrics that make it possible to predict business trends. Thus, being able to access data in real time is particularly beneficial as you can use, consume, and process “fresh” data. <a href="/platform" node="[object Object]">Redpanda</a>, with its Apache KafkaⓇ-compatible API and fast mechanism, provides a data-pipeline backbone for streaming real-time data to be processed flawlessly. Real-time data streaming generally needs faster and scalable persistence layers such as NoSQL databases. <a href="https://www.mongodb.com/" node="[object Object]">MongoDB</a> is a modern NoSQL database that supports transactions, searching, and analytics. It’s horizontally scalable and easily manageable and configurable, making MongoDB one of the best NoSQL technologies. Note: In this tutorial we will be integrating MongoDB with Redpanda via Debezium. This may be preferred in cases where users already have operational patterns in place with Debezium. If you prefer to use MongoDB’s Kafka source connector, this is also an option. Redpanda’s features and MongoDB complement each other well in many use cases, especially those that require real-time data streaming. Managing the data is easy with MongoDB while Redpanda simplifies the streaming. You can keep your important, actively growing data like transactional logs, user-behavior data, or audit data on MongoDB and stream it to Redpanda to expose the data to other platforms for monitoring, tracing, or event triggering. In this tutorial, you’ll learn how to do the following: <li>Run a Redpanda cluster in a containerized way by using Docker</li> <li>Create a topic within Redpanda by using its <code class="styles_Marked__inlineCode__2XDzC">rpk</code> CLI</li> <li>Run MongoDB as a Docker container</li> <li>Run <a href="https://kafka.apache.org/documentation/#connect" node="[object Object]">Kafka ConnectⓇ</a> as a Docker container and configure it with the MongoDB Debezium connector</li> You can refer to <a href="https://github.com/redpanda-data-blog/2022-debezium-mongodb-cdc-tutorial" node="[object Object]">this GitHub repo</a> for all code used in this demo at any time throughout the tutorial. <h2 id="prerequisites" level="2" node="[object Object]">Prerequisites</h2> You’ll need the following to get started: <li>A recent version of <a href="https://docs.docker.com/get-docker/" node="[object Object]">Docker</a> installed on your machine. (Docker Desktop 4.6.1 was used for this article)</li> <li>A command-line interface or terminal to run Docker commands</li> <li>A docker network called <code class="styles_Marked__inlineCode__2XDzC">pandonline</code> (you can create the network by running this command on your terminal: <code class="styles_Marked__inlineCode__2XDzC">docker network create pandonline</code>)</li> <h2 id="scenario:-tracking-users’-classified-ad-visits" level="2" node="[object Object]">Scenario: Tracking users’ classified ad visits</h2> Through an example scenario with Pandonline Corp., a fictional web-based classified ad company, this tutorial demonstrates how you can integrate Redpanda with Debezium for CDC in MongoDB. In this scenario, Pandonline’s website has a huge number of daily visitors, and they want to track and log these visitors’ classified ad visiting activity. They save each user’s classified ad visit activity in MongoDB, and they want to expose this activity to tracking services by streaming it through Redpanda. They prefer to use CDC with Debezium to capture the changes in MongoDB and send them to Redpanda. You are one of Pandonline’s experienced developers and must implement this architecture. They need a prototype from you before discussing any further implementation points with the architecture team, so they have asked you to produce a quick prototype that runs in Docker containers. <h2 id="running-redpanda" level="2" node="[object Object]">Running Redpanda</h2> There are many options for preparing a Redpanda instance. For information on installing or running Redpanda on a variety of platforms, refer to <a href="https://docs.redpanda.com/docs/quickstart/" node="[object Object]">this documentation</a>. For the purposes of this tutorial, Redpanda runs in a Docker container, which you can learn how to do <a href="https://docs.redpanda.com/docs/quickstart/quick-start-docker/" node="[object Object]">here</a>. An example command for running Redpanda on Docker in the <code class="styles_Marked__inlineCode__2XDzC">pandonline</code> network is as follows: <pre><div style="text-align:left;white-space:pre;word-spacing:normal;word-break:normal;word-wrap:normal;color:#eee;background:#2f2f2f;font-family:Roboto Mono, monospace;font-size:1em;line-height:1.5em;-moz-tab-size:4;-o-tab-size:4;tab-size:4;-webkit-hyphens:none;-moz-hyphens:none;-ms-hyphens:none;hyphens:none;overflow:auto;position:relative;margin:0.5em 0;padding:1.25em 1em"><code class="language-bash" style="text-align:left;white-space:pre;word-spacing:normal;word-break:normal;word-wrap:normal;color:#eee;background:#2f2f2f;font-family:Roboto Mono, monospace;font-size:1em;line-height:1.5em;-moz-tab-size:4;-o-tab-size:4;tab-size:4;-webkit-hyphens:none;-moz-hyphens:none;-ms-hyphens:none;hyphens:none">docker run -d --name=redpanda-1 --rm \ --network pandonline \ -p 9092:9092 \ -p 9644:9644 \ docker.vectorized.io/vectorized/redpanda:latest \ redpanda start \ --advertise-kafka-addr redpanda-1 \ --overprovisioned \ --smp 1 \ --memory 1G \ --reserve-memory 500M \ --node-id 0 \ --check=false</code></div></pre> The Docker network is required to make containers communicate. <h2 id="running-and-configuring-mongodb" level="2" node="[object Object]">Running and configuring MongoDB</h2> Execute the following command to run MongoDB with the container instance name <code class="styles_Marked__inlineCode__2XDzC">mongo-1</code> and replica set <code class="styles_Marked__inlineCode__2XDzC">pandonline-mongo-rs</code>: <pre><div style="text-align:left;white-space:pre;word-spacing:normal;word-break:normal;word-wrap:normal;color:#eee;background:#2f2f2f;font-family:Roboto Mono, monospace;font-size:1em;line-height:1.5em;-moz-tab-size:4;-o-tab-size:4;tab-size:4;-webkit-hyphens:none;-moz-hyphens:none;-ms-hyphens:none;hyphens:none;overflow:auto;position:relative;margin:0.5em 0;padding:1.25em 1em"><code class="language-bash" style="text-align:left;white-space:pre;word-spacing:normal;word-break:normal;word-wrap:normal;color:#eee;background:#2f2f2f;font-family:Roboto Mono, monospace;font-size:1em;line-height:1.5em;-moz-tab-size:4;-o-tab-size:4;tab-size:4;-webkit-hyphens:none;-moz-hyphens:none;-ms-hyphens:none;hyphens:none">docker run -d --rm --network pandonline -p 27017:27017 --name mongo-1 mongo mongod --replSet pandonline-mongo-rs</code></div></pre> As the Debezium Kafka connector uses the operational logs of MongoDB, you must configure the MongoDB instance as a replica set. The operation logs (oplog) are only enabled in the replica-set mode. To configure the MongoDB instance, you need to access the MongoDB shell, which you can do by running the following command: <pre><div style="text-align:left;white-space:pre;word-spacing:normal;word-break:normal;word-wrap:normal;color:#eee;background:#2f2f2f;font-family:Roboto Mono, monospace;font-size:1em;line-height:1.5em;-moz-tab-size:4;-o-tab-size:4;tab-size:4;-webkit-hyphens:none;-moz-hyphens:none;-ms-hyphens:none;hyphens:none;overflow:auto;position:relative;margin:0.5em 0;padding:1.25em 1em"><code class="language-bash" style="text-align:left;white-space:pre;word-spacing:normal;word-break:normal;word-wrap:normal;color:#eee;background:#2f2f2f;font-family:Roboto Mono, monospace;font-size:1em;line-height:1.5em;-moz-tab-size:4;-o-tab-size:4;tab-size:4;-webkit-hyphens:none;-moz-hyphens:none;-ms-hyphens:none;hyphens:none">docker exec -it mongo-1 mongo</code></div></pre> Execute the following commands on the MongoDB shell: <pre><div style="text-align:left;white-space:pre;word-spacing:normal;word-break:normal;word-wrap:normal;color:#eee;background:#2f2f2f;font-family:Roboto Mono, monospace;font-size:1em;line-height:1.5em;-moz-tab-size:4;-o-tab-size:4;tab-size:4;-webkit-hyphens:none;-moz-hyphens:none;-ms-hyphens:none;hyphens:none;overflow:auto;position:relative;margin:0.5em 0;padding:1.25em 1em"><code class="language-bash" style="text-align:left;white-space:pre;word-spacing:normal;word-break:normal;word-wrap:normal;color:#eee;background:#2f2f2f;font-family:Roboto Mono, monospace;font-size:1em;line-height:1.5em;-moz-tab-size:4;-o-tab-size:4;tab-size:4;-webkit-hyphens:none;-moz-hyphens:none;-ms-hyphens:none;hyphens:none">config = {"_id":"pandonline-mongo-rs","members":[{"_id":0,"host":"mongo-1:27017"}]}</code></div></pre> <pre><div style="text-align:left;white-space:pre;word-spacing:normal;word-break:normal;word-wrap:normal;color:#eee;background:#2f2f2f;font-family:Roboto Mono, monospace;font-size:1em;line-height:1.5em;-moz-tab-size:4;-o-tab-size:4;tab-size:4;-webkit-hyphens:none;-moz-hyphens:none;-ms-hyphens:none;hyphens:none;overflow:auto;position:relative;margin:0.5em 0;padding:1.25em 1em"><code class="language-bash" style="text-align:left;white-space:pre;word-spacing:normal;word-break:normal;word-wrap:normal;color:#eee;background:#2f2f2f;font-family:Roboto Mono, monospace;font-size:1em;line-height:1.5em;-moz-tab-size:4;-o-tab-size:4;tab-size:4;-webkit-hyphens:none;-moz-hyphens:none;-ms-hyphens:none;hyphens:none">rs.initiate(config)</code></div></pre> The first command assigns a variable called <code class="styles_Marked__inlineCode__2XDzC">config</code> that has the MongoDB replica set configuration. Then with the second command, you initiate the replica set using the configuration variable you set. Type <code class="styles_Marked__inlineCode__2XDzC">exit</code> to exit the Mongo shell. <h2 id="running-kafka-connect-for-debezium" level="2" node="[object Object]">Running Kafka Connect for Debezium</h2> Debezium provides a Kafka Connect container image that can be used with container technologies such as Docker. It is preconfigured, so it has all the needed connector binaries, including the one for Debezium’s MongoDB connector. Execute the following command on your terminal to run Kafka Connect on Docker as a container: <pre><div style="text-align:left;white-space:pre;word-spacing:normal;word-break:normal;word-wrap:normal;color:#eee;background:#2f2f2f;font-family:Roboto Mono, monospace;font-size:1em;line-height:1.5em;-moz-tab-size:4;-o-tab-size:4;tab-size:4;-webkit-hyphens:none;-moz-hyphens:none;-ms-hyphens:none;hyphens:none;overflow:auto;position:relative;margin:0.5em 0;padding:1.25em 1em"><code class="language-bash" style="text-align:left;white-space:pre;word-spacing:normal;word-break:normal;word-wrap:normal;color:#eee;background:#2f2f2f;font-family:Roboto Mono, monospace;font-size:1em;line-height:1.5em;-moz-tab-size:4;-o-tab-size:4;tab-size:4;-webkit-hyphens:none;-moz-hyphens:none;-ms-hyphens:none;hyphens:none">docker run -d -it --rm --name connect-1 \ --network pandonline \ -p 8083:8083 -e GROUP_ID=1 \ -e CONFIG_STORAGE_TOPIC=my_connect_configs \ -e OFFSET_STORAGE_TOPIC=my_connect_offsets \ -e STATUS_STORAGE_TOPIC=my_connect_statuses \ -e BOOTSTRAP_SERVERS=redpanda-1:9092 \ debezium/connect:1.9.5.Final</code></div></pre> This Kafka Connect instance runs in the distributed mode, so it accepts connector configurations via a REST API. To retrieve a list of connectors via a REST API, you can run the following command: <pre><div style="text-align:left;white-space:pre;word-spacing:normal;word-break:normal;word-wrap:normal;color:#eee;background:#2f2f2f;font-family:Roboto Mono, monospace;font-size:1em;line-height:1.5em;-moz-tab-size:4;-o-tab-size:4;tab-size:4;-webkit-hyphens:none;-moz-hyphens:none;-ms-hyphens:none;hyphens:none;overflow:auto;position:relative;margin:0.5em 0;padding:1.25em 1em"><code class="language-bash" style="text-align:left;white-space:pre;word-spacing:normal;word-break:normal;word-wrap:normal;color:#eee;background:#2f2f2f;font-family:Roboto Mono, monospace;font-size:1em;line-height:1.5em;-moz-tab-size:4;-o-tab-size:4;tab-size:4;-webkit-hyphens:none;-moz-hyphens:none;-ms-hyphens:none;hyphens:none">curl -H "Accept:application/json" localhost:8083/connectors</code></div></pre> As no connector has been configured yet, you should see the message “curl: (52) Empty reply from server.” <h3 id="configuring-the-connector" level="3" node="[object Object]">Configuring the connector</h3> To capture data changes on MongoDB and send them to any Redpanda topic, you must configure a Debezium connector for MongoDB. The following JSON is an example of a configured connector with the name <code class="styles_Marked__inlineCode__2XDzC">classifieds-connector</code>: "name": "classifieds-connector", "config": { "connector.class": "io.debezium.connector.mongodb.MongoDbConnector", "mongodb.hosts": "pandonline-mongo-rs/mongo-1:27017", "mongodb.name": "mongo-1", "collection.include.list": "testdb.classifieds" Note that the configuration includes <code class="styles_Marked__inlineCode__2XDzC">io.debezium.connector.mongodb.MongoDbConnector</code> as the connector class, which points to the Java binaries for MongoDB Debezium Connector in the connect cluster instance running in the Docker container. To capture changes from a specific MongoDB database and a schema, you must define them in the <code class="styles_Marked__inlineCode__2XDzC">mongodb.hosts</code> and <code class="styles_Marked__inlineCode__2XDzC">collection.include.list</code>. The configuration above makes it possible to access a MongoDB instance on host <code class="styles_Marked__inlineCode__2XDzC">mongo-1</code> and port <code class="styles_Marked__inlineCode__2XDzC">27017</code> on a replica set called <code class="styles_Marked__inlineCode__2XDzC">pandonline-mongo-rs</code>. The <code class="styles_Marked__inlineCode__2XDzC">collection.include.list</code> is a kind of allow list that defines the databases and the schemas to be connected. Run the following <code class="styles_Marked__inlineCode__2XDzC">curl</code> command to apply the above configuration via Kafka Connect’s REST API: <pre><div style="text-align:left;white-space:pre;word-spacing:normal;word-break:normal;word-wrap:normal;color:#eee;background:#2f2f2f;font-family:Roboto Mono, monospace;font-size:1em;line-height:1.5em;-moz-tab-size:4;-o-tab-size:4;tab-size:4;-webkit-hyphens:none;-moz-hyphens:none;-ms-hyphens:none;hyphens:none;overflow:auto;position:relative;margin:0.5em 0;padding:1.25em 1em"><code class="language-bash" style="text-align:left;white-space:pre;word-spacing:normal;word-break:normal;word-wrap:normal;color:#eee;background:#2f2f2f;font-family:Roboto Mono, monospace;font-size:1em;line-height:1.5em;-moz-tab-size:4;-o-tab-size:4;tab-size:4;-webkit-hyphens:none;-moz-hyphens:none;-ms-hyphens:none;hyphens:none">curl -i -X POST -H "Accept:application/json" -H "Content-Type:application/json" \ localhost:8083/connectors \ -d '{"name":"classifieds-connector","config":{"connector.class":"io.debezium.connector.mongodb.MongoDbConnector","mongodb.hosts":"pandonline-mongo-rs/mongo-1:27017","mongodb.name":"mongo-1","collection.include.list": "testdb.classifieds"}}'</code></div></pre> You can examine the created connectors using the following code: <pre><div style="text-align:left;white-space:pre;word-spacing:normal;word-break:normal;word-wrap:normal;color:#eee;background:#2f2f2f;font-family:Roboto Mono, monospace;font-size:1em;line-height:1.5em;-moz-tab-size:4;-o-tab-size:4;tab-size:4;-webkit-hyphens:none;-moz-hyphens:none;-ms-hyphens:none;hyphens:none;overflow:auto;position:relative;margin:0.5em 0;padding:1.25em 1em"><code class="language-bash" style="text-align:left;white-space:pre;word-spacing:normal;word-break:normal;word-wrap:normal;color:#eee;background:#2f2f2f;font-family:Roboto Mono, monospace;font-size:1em;line-height:1.5em;-moz-tab-size:4;-o-tab-size:4;tab-size:4;-webkit-hyphens:none;-moz-hyphens:none;-ms-hyphens:none;hyphens:none">curl -H "Accept:application/json" localhost:8083/connectors</code></div></pre> The output should be as follows: <pre><div style="text-align:left;white-space:pre;word-spacing:normal;word-break:normal;word-wrap:normal;color:#eee;background:#2f2f2f;font-family:Roboto Mono, monospace;font-size:1em;line-height:1.5em;-moz-tab-size:4;-o-tab-size:4;tab-size:4;-webkit-hyphens:none;-moz-hyphens:none;-ms-hyphens:none;hyphens:none;overflow:auto;position:relative;margin:0.5em 0;padding:1.25em 1em"><code class="language-bash" style="text-align:left;white-space:pre;word-spacing:normal;word-break:normal;word-wrap:normal;color:#eee;background:#2f2f2f;font-family:Roboto Mono, monospace;font-size:1em;line-height:1.5em;-moz-tab-size:4;-o-tab-size:4;tab-size:4;-webkit-hyphens:none;-moz-hyphens:none;-ms-hyphens:none;hyphens:none">["classifieds-connector"]</code></div></pre> If you are getting this output, it means your connector task has been successfully created in the Kafka Connect instance. <h2 id="creating-the-redpanda-topic-and-consuming-data" level="2" node="[object Object]">Creating the Redpanda topic and consuming data</h2> By default, the Debezium MongoDB connector sends the captured data to a topic with a specific name: it uses <code class="styles_Marked__inlineCode__2XDzC">mongodb.name</code> and <code class="styles_Marked__inlineCode__2XDzC">collection.include.list</code>, which consists of <code class="styles_Marked__inlineCode__2XDzC">[database_name].[schema_name]</code> pairs. So the overall topic name should consist of <code class="styles_Marked__inlineCode__2XDzC">[mongodb_instance_name].[database_name].[schema_name]</code>. For the example in this tutorial, the topic name is <code class="styles_Marked__inlineCode__2XDzC">mongo-1.testdb.classifieds</code>, and you can use the following command to create this topic in the Redpanda cluster: <pre><div style="text-align:left;white-space:pre;word-spacing:normal;word-break:normal;word-wrap:normal;color:#eee;background:#2f2f2f;font-family:Roboto Mono, monospace;font-size:1em;line-height:1.5em;-moz-tab-size:4;-o-tab-size:4;tab-size:4;-webkit-hyphens:none;-moz-hyphens:none;-ms-hyphens:none;hyphens:none;overflow:auto;position:relative;margin:0.5em 0;padding:1.25em 1em"><code class="language-bash" style="text-align:left;white-space:pre;word-spacing:normal;word-break:normal;word-wrap:normal;color:#eee;background:#2f2f2f;font-family:Roboto Mono, monospace;font-size:1em;line-height:1.5em;-moz-tab-size:4;-o-tab-size:4;tab-size:4;-webkit-hyphens:none;-moz-hyphens:none;-ms-hyphens:none;hyphens:none">docker exec -it redpanda-1 \ rpk topic create mongo-1.testdb.classifieds</code></div></pre> Validate the <code class="styles_Marked__inlineCode__2XDzC">mongo-1.testdb.classifieds</code> topic with the following command: <pre><div style="text-align:left;white-space:pre;word-spacing:normal;word-break:normal;word-wrap:normal;color:#eee;background:#2f2f2f;font-family:Roboto Mono, monospace;font-size:1em;line-height:1.5em;-moz-tab-size:4;-o-tab-size:4;tab-size:4;-webkit-hyphens:none;-moz-hyphens:none;-ms-hyphens:none;hyphens:none;overflow:auto;position:relative;margin:0.5em 0;padding:1.25em 1em"><code class="language-bash" style="text-align:left;white-space:pre;word-spacing:normal;word-break:normal;word-wrap:normal;color:#eee;background:#2f2f2f;font-family:Roboto Mono, monospace;font-size:1em;line-height:1.5em;-moz-tab-size:4;-o-tab-size:4;tab-size:4;-webkit-hyphens:none;-moz-hyphens:none;-ms-hyphens:none;hyphens:none">docker exec -it redpanda-1 \ rpk cluster info</code></div></pre> This returns the following output: <pre><div style="text-align:left;white-space:pre;word-spacing:normal;word-break:normal;word-wrap:normal;color:#eee;background:#2f2f2f;font-family:Roboto Mono, monospace;font-size:1em;line-height:1.5em;-moz-tab-size:4;-o-tab-size:4;tab-size:4;-webkit-hyphens:none;-moz-hyphens:none;-ms-hyphens:none;hyphens:none;overflow:auto;position:relative;margin:0.5em 0;padding:1.25em 1em"><code class="language-bash" style="text-align:left;white-space:pre;word-spacing:normal;word-break:normal;word-wrap:normal;color:#eee;background:#2f2f2f;font-family:Roboto Mono, monospace;font-size:1em;line-height:1.5em;-moz-tab-size:4;-o-tab-size:4;tab-size:4;-webkit-hyphens:none;-moz-hyphens:none;-ms-hyphens:none;hyphens:none">BROKERS ======= ID HOST PORT 0* redpanda-1 9092 TOPICS ====== NAME PARTITIONS REPLICAS ...output omitted... mongo-1.testdb.classifieds 1 1 ...output omitted...</code></div></pre> In the same terminal window, run the following command to start consuming from the <code class="styles_Marked__inlineCode__2XDzC">mongo-1.testdb.classifieds</code> topic: <pre><div style="text-align:left;white-space:pre;word-spacing:normal;word-break:normal;word-wrap:normal;color:#eee;background:#2f2f2f;font-family:Roboto Mono, monospace;font-size:1em;line-height:1.5em;-moz-tab-size:4;-o-tab-size:4;tab-size:4;-webkit-hyphens:none;-moz-hyphens:none;-ms-hyphens:none;hyphens:none;overflow:auto;position:relative;margin:0.5em 0;padding:1.25em 1em"><code class="language-bash" style="text-align:left;white-space:pre;word-spacing:normal;word-break:normal;word-wrap:normal;color:#eee;background:#2f2f2f;font-family:Roboto Mono, monospace;font-size:1em;line-height:1.5em;-moz-tab-size:4;-o-tab-size:4;tab-size:4;-webkit-hyphens:none;-moz-hyphens:none;-ms-hyphens:none;hyphens:none">docker exec -it redpanda-1 \ rpk topic consume mongo-1.testdb.classifieds</code></div></pre> Leave the terminal window open to watch the consumed messages for the next steps. <h2 id="capturing-the-change-events" level="2" node="[object Object]">Capturing the change events</h2> Pandonline has provided you with a simple containerized application for testing the CDC. It creates random classified ad records and saves them to your MongoDB instance in five-second intervals. The following is an example classified-ad record: "classifiedId":2696, "userId":8649, "url":"www.pandonline-classified-ads.com/adlgfxzccr" To run the containerized application, open a new terminal window and run the following command: <pre><div style="text-align:left;white-space:pre;word-spacing:normal;word-break:normal;word-wrap:normal;color:#eee;background:#2f2f2f;font-family:Roboto Mono, monospace;font-size:1em;line-height:1.5em;-moz-tab-size:4;-o-tab-size:4;tab-size:4;-webkit-hyphens:none;-moz-hyphens:none;-ms-hyphens:none;hyphens:none;overflow:auto;position:relative;margin:0.5em 0;padding:1.25em 1em"><code class="language-bash" style="text-align:left;white-space:pre;word-spacing:normal;word-break:normal;word-wrap:normal;color:#eee;background:#2f2f2f;font-family:Roboto Mono, monospace;font-size:1em;line-height:1.5em;-moz-tab-size:4;-o-tab-size:4;tab-size:4;-webkit-hyphens:none;-moz-hyphens:none;-ms-hyphens:none;hyphens:none">docker logs -f $(docker run -d -it --rm \ --network pandonline \ quay.io/systemcraftsman/visited_classified_ads_gateway)</code></div></pre> You should see the generated logs as the application saves the records to the MongoDB <code class="styles_Marked__inlineCode__2XDzC">classifieds</code> schema: <pre><div style="text-align:left;white-space:pre;word-spacing:normal;word-break:normal;word-wrap:normal;color:#eee;background:#2f2f2f;font-family:Roboto Mono, monospace;font-size:1em;line-height:1.5em;-moz-tab-size:4;-o-tab-size:4;tab-size:4;-webkit-hyphens:none;-moz-hyphens:none;-ms-hyphens:none;hyphens:none;overflow:auto;position:relative;margin:0.5em 0;padding:1.25em 1em"><code class="language-bash" style="text-align:left;white-space:pre;word-spacing:normal;word-break:normal;word-wrap:normal;color:#eee;background:#2f2f2f;font-family:Roboto Mono, monospace;font-size:1em;line-height:1.5em;-moz-tab-size:4;-o-tab-size:4;tab-size:4;-webkit-hyphens:none;-moz-hyphens:none;-ms-hyphens:none;hyphens:none">...output omitted... quay.io/systemcraftsman/visited_classified_ads_gateway) Classified data is saved: {'classifiedId': 3938, 'userId': 7575, 'url': 'www.pandonline-classified-ads.com/ixulfgglbv', '_id': ObjectId('62cba65993884cdcaea95d50')} Classified data is saved: {'classifiedId': 2704, 'userId': 6630, 'url': 'www.pandonline-classified-ads.com/jpchradmuf', '_id': ObjectId('62cba65e93884cdcaea95d51')} ...output omitted...</code></div></pre> Return to your Redpanda consumer terminal, where you should see the changes captured and streamed into the relevant Redpanda topic: <pre><div style="text-align:left;white-space:pre;word-spacing:normal;word-break:normal;word-wrap:normal;color:#eee;background:#2f2f2f;font-family:Roboto Mono, monospace;font-size:1em;line-height:1.5em;-moz-tab-size:4;-o-tab-size:4;tab-size:4;-webkit-hyphens:none;-moz-hyphens:none;-ms-hyphens:none;hyphens:none;overflow:auto;position:relative;margin:0.5em 0;padding:1.25em 1em"><code class="language-bash" style="text-align:left;white-space:pre;word-spacing:normal;word-break:normal;word-wrap:normal;color:#eee;background:#2f2f2f;font-family:Roboto Mono, monospace;font-size:1em;line-height:1.5em;-moz-tab-size:4;-o-tab-size:4;tab-size:4;-webkit-hyphens:none;-moz-hyphens:none;-ms-hyphens:none;hyphens:none">...output omitted... "topic":"mongo-1.testdb.classifieds", "key":"{\"schema\":{\"type\":\"struct\",\"fields\":[{\"type\":\"string\",\"optional\":false,\"field\":\"id\"}],\"optional\":false,\"name\":\"mongo_1.testdb.classifieds.Key\"},\"payload\":{\"id\":\"{\\\"$oid\\\": \\\"62cba96293884cdcaea95deb\\\"}\"}}", "value":"{\"schema\":{\"type\":\"struct\",\"fields\":[{\"type\":\"string\",\"optional\":true,\"name\":\"io.debezium.data.Json\",\"version\":1,\"field\":\"after\"},{\"type\":\"string\",\"optional\":true,\"name\":\"io.debezium.data.Json\",\"version\":1,\"field\":\"patch\"},{\"type\":\"string\",\"optional\":true,\"name\":\"io.debezium.data.Json\",\"version\":1,\"field\":\"filter\"},{\"type\":\"struct\",\"fields\":[{\"type\":\"string\",\"optional\":false,\"field\":\"version\"},{\"type\":\"string\",\"optional\":false,\"field\":\"connector\"},{\"type\":\"string\",\"optional\":false,\"field\":\"name\"},{\"type\":\"int64\",\"optional\":false,\"field\":\"ts_ms\"},{\"type\":\"string\",\"optional\":true,\"name\":\"io.debezium.data.Enum\",\"version\":1,\"parameters\":{\"allowed\":\"true,last,false\"},\"default\":\"false\",\"field\":\"snapshot\"},{\"type\":\"string\",\"optional\":false,\"field\":\"db\"},{\"type\":\"string\",\"optional\":true,\"field\":\"sequence\"},{\"type\":\"string\",\"optional\":false,\"field\":\"rs\"},{\"type\":\"string\",\"optional\":false,\"field\":\"collection\"},{\"type\":\"int32\",\"optional\":false,\"field\":\"ord\"},{\"type\":\"int64\",\"optional\":true,\"field\":\"h\"},{\"type\":\"int64\",\"optional\":true,\"field\":\"tord\"},{\"type\":\"string\",\"optional\":true,\"field\":\"stxnid\"}],\"optional\":false,\"name\":\"io.debezium.connector.mongo.Source\",\"field\":\"source\"},{\"type\":\"string\",\"optional\":true,\"field\":\"op\"},{\"type\":\"int64\",\"optional\":true,\"field\":\"ts_ms\"},{\"type\":\"struct\",\"fields\":[{\"type\":\"string\",\"optional\":false,\"field\":\"id\"},{\"type\":\"int64\",\"optional\":false,\"field\":\"total_order\"},{\"type\":\"int64\",\"optional\":false,\"field\":\"data_collection_order\"}],\"optional\":true,\"field\":\"transaction\"}],\"optional\":false,\"name\":\"mongo_1.testdb.classifieds.Envelope\"},\"payload\":{\"after\":\"{\\\"_id\\\": {\\\"$oid\\\": \\\"62cba96293884cdcaea95deb\\\"},\\\"classifiedId\\\": 3697,\\\"userId\\\": 8867,\\\"url\\\": \\\"www.pandonline-classified-ads.com/camywtnsft\\\"}\",\"patch\":null,\"filter\":null,\"source\":{\"version\":\"1.6.3.Final\",\"connector\":\"mongodb\",\"name\":\"mongo-1\",\"ts_ms\":1657514338000,\"snapshot\":\"false\",\"db\":\"testdb\",\"sequence\":null,\"rs\":\"my-mongo-set\",\"collection\":\"classifieds\",\"ord\":2,\"h\":null,\"tord\":null,\"stxnid\":\"cbcaccdd-579a-389f-88b8-a23cef4618fc:156\"},\"op\":\"c\",\"ts_ms\":1657514338571,\"transaction\":null}}", "timestamp":1657514338842, "partition":0, "offset":2097 "topic":"mongo-1.testdb.classifieds", "key":"{\"schema\":{\"type\":\"struct\",\"fields\":[{\"type\":\"string\",\"optional\":false,\"field\":\"id\"}],\"optional\":false,\"name\":\"mongo_1.testdb.classifieds.Key\"},\"payload\":{\"id\":\"{\\\"$oid\\\": \\\"62cba96492bb209e6ce7ff1a\\\"}\"}}", "value":"{\"schema\":{\"type\":\"struct\",\"fields\":[{\"type\":\"string\",\"optional\":true,\"name\":\"io.debezium.data.Json\",\"version\":1,\"field\":\"after\"},{\"type\":\"string\",\"optional\":true,\"name\":\"io.debezium.data.Json\",\"version\":1,\"field\":\"patch\"},{\"type\":\"string\",\"optional\":true,\"name\":\"io.debezium.data.Json\",\"version\":1,\"field\":\"filter\"},{\"type\":\"struct\",\"fields\":[{\"type\":\"string\",\"optional\":false,\"field\":\"version\"},{\"type\":\"string\",\"optional\":false,\"field\":\"connector\"},{\"type\":\"string\",\"optional\":false,\"field\":\"name\"},{\"type\":\"int64\",\"optional\":false,\"field\":\"ts_ms\"},{\"type\":\"string\",\"optional\":true,\"name\":\"io.debezium.data.Enum\",\"version\":1,\"parameters\":{\"allowed\":\"true,last,false\"},\"default\":\"false\",\"field\":\"snapshot\"},{\"type\":\"string\",\"optional\":false,\"field\":\"db\"},{\"type\":\"string\",\"optional\":true,\"field\":\"sequence\"},{\"type\":\"string\",\"optional\":false,\"field\":\"rs\"},{\"type\":\"string\",\"optional\":false,\"field\":\"collection\"},{\"type\":\"int32\",\"optional\":false,\"field\":\"ord\"},{\"type\":\"int64\",\"optional\":true,\"field\":\"h\"},{\"type\":\"int64\",\"optional\":true,\"field\":\"tord\"},{\"type\":\"string\",\"optional\":true,\"field\":\"stxnid\"}],\"optional\":false,\"name\":\"io.debezium.connector.mongo.Source\",\"field\":\"source\"},{\"type\":\"string\",\"optional\":true,\"field\":\"op\"},{\"type\":\"int64\",\"optional\":true,\"field\":\"ts_ms\"},{\"type\":\"struct\",\"fields\":[{\"type\":\"string\",\"optional\":false,\"field\":\"id\"},{\"type\":\"int64\",\"optional\":false,\"field\":\"total_order\"},{\"type\":\"int64\",\"optional\":false,\"field\":\"data_collection_order\"}],\"optional\":true,\"field\":\"transaction\"}],\"optional\":false,\"name\":\"mongo_1.testdb.classifieds.Envelope\"},\"payload\":{\"after\":\"{\\\"_id\\\": {\\\"$oid\\\": \\\"62cba96492bb209e6ce7ff1a\\\"},\\\"classifiedId\\\": 4354,\\\"userId\\\": 2,\\\"url\\\": \\\"www.pandonline-classified-ads.com/ackvkzggjv\\\"}\",\"patch\":null,\"filter\":null,\"source\":{\"version\":\"1.6.3.Final\",\"connector\":\"mongodb\",\"name\":\"mongo-1\",\"ts_ms\":1657514340000,\"snapshot\":\"false\",\"db\":\"testdb\",\"sequence\":null,\"rs\":\"my-mongo-set\",\"collection\":\"classifieds\",\"ord\":1,\"h\":null,\"tord\":null,\"stxnid\":\"b243f8ff-7aca-34e1-91ec-0ffb88f14468:207\"},\"op\":\"c\",\"ts_ms\":1657514340721,\"transaction\":null}}", "timestamp":1657514340842, "partition":0, "offset":2098 ...output omitted...</code></div></pre> This is the data captured by Debezium and sent to the Redpanda topic. Because this data consists of the JSON schema structure as well, you might not notice the actual captured data <code class="styles_Marked__inlineCode__2XDzC">payload</code> at the first sight among the huge JSON output. Schemas make sure that the data you are transferring has a structure. However, depending on the use case, you might not need a schema along with your actual payload. As you are only completing this tutorial for prototyping purposes for Pandonline Corp., you can skip using the JSON schema and just stream the payloads: "name": "classifieds-connector", "config": { "connector.class": "io.debezium.connector.mongodb.MongoDbConnector", "mongodb.hosts": "pandonline-mongo-rs/mongo-1:27017", "mongodb.name": "mongo-1", "collection.include.list": "testdb.classifieds", "key.converter": "org.apache.kafka.connect.json.JsonConverter", "key.converter.schemas.enable":"false", "value.converter": "org.apache.kafka.connect.json.JsonConverter", "value.converter.schemas.enable":"false" In a new terminal window, run the following command to update the connector configuration with the preceding one: <pre><div style="text-align:left;white-space:pre;word-spacing:normal;word-break:normal;word-wrap:normal;color:#eee;background:#2f2f2f;font-family:Roboto Mono, monospace;font-size:1em;line-height:1.5em;-moz-tab-size:4;-o-tab-size:4;tab-size:4;-webkit-hyphens:none;-moz-hyphens:none;-ms-hyphens:none;hyphens:none;overflow:auto;position:relative;margin:0.5em 0;padding:1.25em 1em"><code class="language-bash" style="text-align:left;white-space:pre;word-spacing:normal;word-break:normal;word-wrap:normal;color:#eee;background:#2f2f2f;font-family:Roboto Mono, monospace;font-size:1em;line-height:1.5em;-moz-tab-size:4;-o-tab-size:4;tab-size:4;-webkit-hyphens:none;-moz-hyphens:none;-ms-hyphens:none;hyphens:none">curl -i -X PUT -H "Accept:application/json" -H "Content-Type:application/json" \ localhost:8083/connectors/classifieds-connector/config \ -d '{"connector.class":"io.debezium.connector.mongodb.MongoDbConnector","mongodb.hosts":"pandonline-mongo-rs/mongo-1:27017","mongodb.name":"mongo-1","collection.include.list": "testdb.classifieds","key.converter": "org.apache.kafka.connect.json.JsonConverter","key.converter.schemas.enable":"false","value.converter": "org.apache.kafka.connect.json.JsonConverter","value.converter.schemas.enable":"false"}'</code></div></pre> The output should be as follows: <pre><div style="text-align:left;white-space:pre;word-spacing:normal;word-break:normal;word-wrap:normal;color:#eee;background:#2f2f2f;font-family:Roboto Mono, monospace;font-size:1em;line-height:1.5em;-moz-tab-size:4;-o-tab-size:4;tab-size:4;-webkit-hyphens:none;-moz-hyphens:none;-ms-hyphens:none;hyphens:none;overflow:auto;position:relative;margin:0.5em 0;padding:1.25em 1em"><code class="language-bash" style="text-align:left;white-space:pre;word-spacing:normal;word-break:normal;word-wrap:normal;color:#eee;background:#2f2f2f;font-family:Roboto Mono, monospace;font-size:1em;line-height:1.5em;-moz-tab-size:4;-o-tab-size:4;tab-size:4;-webkit-hyphens:none;-moz-hyphens:none;-ms-hyphens:none;hyphens:none">...output omitted... "topic": "mongo-1.testdb.classifieds", "key": "{\"id\":\"{\\\"$oid\\\": \\\"62cbba4e920e0661fc10e660\\\"}\"}", "value": "{\"after\":\"{\\\"_id\\\": {\\\"$oid\\\": \\\"62cbba4e920e0661fc10e660\\\"},\\\"classifiedId\\\": 2878,\\\"userId\\\": 6975,\\\"url\\\": \\\"www.pandonline-classified-ads.com/zsxzrqolgm\\\"}\",\"patch\":null,\"filter\":null,\"source\":{\"version\":\"1.6.3.Final\",\"connector\":\"mongodb\",\"name\":\"mongo-1\",\"ts_ms\":1657518670000,\"snapshot\":\"false\",\"db\":\"testdb\",\"sequence\":null,\"rs\":\"my-mongo-set\",\"collection\":\"classifieds\",\"ord\":1,\"h\":null,\"tord\":null,\"stxnid\":\"1e0e101d-13bd-39bd-a697-184cdd9e9ce6:380\"},\"op\":\"c\",\"ts_ms\":1657518670243,\"transaction\":null}", "timestamp": 1657518670395, "partition": 0, "offset": 5557 "topic": "mongo-1.testdb.classifieds", "key": "{\"id\":\"{\\\"$oid\\\": \\\"62cbba4e4b1acb3174a70406\\\"}\"}", "value": "{\"after\":\"{\\\"_id\\\": {\\\"$oid\\\": \\\"62cbba4e4b1acb3174a70406\\\"},\\\"classifiedId\\\": 2163,\\\"userId\\\": 5195,\\\"url\\\": \\\"www.pandonline-classified-ads.com/nlkgrpbxiy\\\"}\",\"patch\":null,\"filter\":null,\"source\":{\"version\":\"1.6.3.Final\",\"connector\":\"mongodb\",\"name\":\"mongo-1\",\"ts_ms\":1657518670000,\"snapshot\":\"false\",\"db\":\"testdb\",\"sequence\":null,\"rs\":\"my-mongo-set\",\"collection\":\"classifieds\",\"ord\":2,\"h\":null,\"tord\":null,\"stxnid\":\"e1937986-4b8a-3560-8109-df94b3c3c770:382\"},\"op\":\"c\",\"ts_ms\":1657518670261,\"transaction\":null}", "timestamp": 1657518670395, "partition": 0, "offset": 5558 ...output omitted...</code></div></pre>