Solving challenges caused by Out Of Memory (OOM) Killer in Linux

link管理

链接快照平台

输入网页链接，自动生成快照
标签化管理网页链接

相关文章推荐

鼻子大的脆皮肠 · 簡單的 CUDA ...· 4 月前 ·

活泼的芹菜 · Using SystemTap ...· 4 月前 ·

豪情万千的牛肉面 · Allocate memory in ...· 1 月前 ·

阳光的蚂蚁 · 全球法规网-中国商务法规· 2 年前 ·

坚强的面包 · 中国电建团委：为青年成长引路搭台--专题报道 ...· 2 年前 ·

乐观的煎鸡蛋 · 咖喱芋圆儿花开富贵儿的个人空间-咖喱芋圆儿花 ...· 2 年前 ·

害羞的海龟 · 学子故事 | 在路上的西浦电影人 - ...· 2 年前 ·

千年单身的咖啡 · 值得一看的小说推荐《穿越成反派要如何活命》_ ...· 2 年前 ·

</noscript><div id="__next" data-reactroot=""><div class="styles_Layout__MHDWL" style="padding-top:0"><div class="styles_Header__T_z_x"><header role="contentinfo" class="styles_DesktopHeader__EY9hX"><a class="styles_DesktopHeader__logo__S6_Zn" aria-label="Red Panda Logo" href="/"><img src="https://images.ctfassets.net/paqvtpyf8rwu/5wODDUzLP7teOGDS7M5m39/466f646e44c2ccc4e78236b17d7d68ac/Logo.svg" alt="" title="Redpanda Logo" width="155" height="36"/></a><nav class="styles_DesktopHeader__desktopNav__K2ueX"><ul class="styles_DesktopHeader__navList__xVQM2"><li class="styles_DesktopHeader__navItem__8yJw5 mr-0" aria-label="Toggle PLATFORM subnav dropdown"><div><button class="styles_DesktopHeader__navButton__gVr05">PLATFORM<span><svg width="12" height="7" viewbox="0 0 12 7" fill="none" xmlns="http://www.w3.org/2000/svg" class=""><path d="M1 1L6 6L11 1" stroke="#667085" stroke-width="1.67" stroke-linecap="round" stroke-linejoin="round"/></svg></span></button><div class="styles_DesktopHeader__subNav__LuUHn"><ul class="styles_DesktopHeader__subNavList__xd4nt"><li class="styles_DesktopHeader__subNavItem__KgS5R"><a class="styles_Header__subNavLink__HCqh7" target="_self" rel="" href="/what-is-redpanda"><div class="styles_Header__subNavLinkTextAndIcon__YD6kl"><img class="styles_Header__subNavLinkIcon__ORxzC" src="https://images.ctfassets.net/paqvtpyf8rwu/5VKIqaX8JhLWcL9mPUnpXg/019d6f9d8594c781ab0cb4401b7c45a7/r-icon.svg" alt="Overview"/><div><div class="styles_Header__subNavLinkGroup__FAyTJ"><p class="styles_Header__subNavLinkText__P9brF">Overview</p><div class="styles_Header__subNavLinkArrow___dJm0"><svg width="17" height="13" viewbox="0 0 17 16" fill="none" xmlns="http://www.w3.org/2000/svg"><path d="M10.7077 10.313L9.42138 9.02034L12.4023 6.05306L-6.79749e-05 6.04121L0.00175553 4.218L12.3704 4.22985L9.43961 1.28536L10.7314 -2.249e-07L15.8765 5.16881L10.7077 10.313Z" fill="#E2401B"/></svg></div></div><span class="styles_Header__subNavLinkSubText__T9G0C">Technical tour of the Redpanda platform</span></div></div></a></li><li class="styles_DesktopHeader__subNavItem__KgS5R"><a class="styles_Header__subNavLink__HCqh7" target="_self" rel="" href="/platform-capabilities"><div class="styles_Header__subNavLinkTextAndIcon__YD6kl"><img class="styles_Header__subNavLinkIcon__ORxzC" src="https://images.ctfassets.net/paqvtpyf8rwu/3QzpsfG9Ba6u6Zu14Zmb0X/06873b28b5459a166879833affc7923d/capabilities.svg" alt="Core capabilities"/><div><div class="styles_Header__subNavLinkGroup__FAyTJ"><p class="styles_Header__subNavLinkText__P9brF">Core capabilities</p><div class="styles_Header__subNavLinkArrow___dJm0"><svg width="17" height="13" viewbox="0 0 17 16" fill="none" xmlns="http://www.w3.org/2000/svg"><path d="M10.7077 10.313L9.42138 9.02034L12.4023 6.05306L-6.79749e-05 6.04121L0.00175553 4.218L12.3704 4.22985L9.43961 1.28536L10.7314 -2.249e-07L15.8765 5.16881L10.7077 10.313Z" fill="#E2401B"/></svg></div></div><span class="styles_Header__subNavLinkSubText__T9G0C">Explore all features and functions</span></div></div></a></li><li class="styles_DesktopHeader__subNavItem__KgS5R"><a class="styles_Header__subNavLink__HCqh7" target="_self" rel="" href="/redpanda-console-kafka-ui"><div class="styles_Header__subNavLinkTextAndIcon__YD6kl"><img class="styles_Header__subNavLinkIcon__ORxzC" src="https://images.ctfassets.net/paqvtpyf8rwu/2JdeLBOezvJgJ1SNId6FFZ/ba1a95da69557a00cb7c792e862f97ba/console.svg" alt="Redpanda Console"/><div><div class="styles_Header__subNavLinkGroup__FAyTJ"><p class="styles_Header__subNavLinkText__P9brF">Redpanda Console</p><div class="styles_Header__subNavLinkArrow___dJm0"><svg width="17" height="13" viewbox="0 0 17 16" fill="none" xmlns="http://www.w3.org/2000/svg"><path d="M10.7077 10.313L9.42138 9.02034L12.4023 6.05306L-6.79749e-05 6.04121L0.00175553 4.218L12.3704 4.22985L9.43961 1.28536L10.7314 -2.249e-07L15.8765 5.16881L10.7077 10.313Z" fill="#E2401B"/></svg></div></div><span class="styles_Header__subNavLinkSubText__T9G0C">An interactive web UI for Kafka</span></div></div></a></li><li class="styles_DesktopHeader__subNavItem__KgS5R"><a class="styles_Header__subNavLink__HCqh7" target="_self" rel="" href="/compare-platform-editions"><div class="styles_Header__subNavLinkTextAndIcon__YD6kl"><img class="styles_Header__subNavLinkIcon__ORxzC" src="https://images.ctfassets.net/paqvtpyf8rwu/5xWZuYA6eAr8uVmfC8GpnX/26a09a1b162c0878cddd4852b3ba4023/comparison.svg" alt="Compare editions"/><div><div class="styles_Header__subNavLinkGroup__FAyTJ"><p class="styles_Header__subNavLinkText__P9brF">Compare editions</p><div class="styles_Header__subNavLinkArrow___dJm0"><svg width="17" height="13" viewbox="0 0 17 16" fill="none" xmlns="http://www.w3.org/2000/svg"><path d="M10.7077 10.313L9.42138 9.02034L12.4023 6.05306L-6.79749e-05 6.04121L0.00175553 4.218L12.3704 4.22985L9.43961 1.28536L10.7314 -2.249e-07L15.8765 5.16881L10.7077 10.313Z" fill="#E2401B"/></svg></div></div><span class="styles_Header__subNavLinkSubText__T9G0C">Understand Redpanda Enterprise &amp; Community Editions</span></div></div></a></li><li class="styles_DesktopHeader__subNavItem__KgS5R"><a class="styles_Header__subNavLink__HCqh7" target="_self" rel="" href="/platform-tco/"><div class="styles_Header__subNavLinkTextAndIcon__YD6kl"><img class="styles_Header__subNavLinkIcon__ORxzC" src="https://images.ctfassets.net/paqvtpyf8rwu/4xqo30nw0MEDIcpVos5hmD/6e2e148a35c59fa4d85c97f2d8186234/tco-icon.svg" alt="Redpanda TCO"/><div><div class="styles_Header__subNavLinkGroup__FAyTJ"><p class="styles_Header__subNavLinkText__P9brF">Redpanda TCO</p><div class="styles_Header__subNavLinkArrow___dJm0"><svg width="17" height="13" viewbox="0 0 17 16" fill="none" xmlns="http://www.w3.org/2000/svg"><path d="M10.7077 10.313L9.42138 9.02034L12.4023 6.05306L-6.79749e-05 6.04121L0.00175553 4.218L12.3704 4.22985L9.43961 1.28536L10.7314 -2.249e-07L15.8765 5.16881L10.7077 10.313Z" fill="#E2401B"/></svg></div></div><span class="styles_Header__subNavLinkSubText__T9G0C">6x lower TCO vs Kafka - see how</span></div></div></a></li><li class="styles_DesktopHeader__subNavItem__KgS5R"><a class="styles_Header__subNavLink__HCqh7" target="_self" rel="" href="/deployment-options"><div class="styles_Header__subNavLinkTextAndIcon__YD6kl"><img class="styles_Header__subNavLinkIcon__ORxzC" src="https://images.ctfassets.net/paqvtpyf8rwu/4j3qyEo0tUk3PDFqrgUmhe/b7efe7f8ff072a9d31ac8a7638dfb7c0/deployment.svg" alt="All deployment options"/><div><div class="styles_Header__subNavLinkGroup__FAyTJ"><p class="styles_Header__subNavLinkText__P9brF">All deployment options</p><div class="styles_Header__subNavLinkArrow___dJm0"><svg width="17" height="13" viewbox="0 0 17 16" fill="none" xmlns="http://www.w3.org/2000/svg"><path d="M10.7077 10.313L9.42138 9.02034L12.4023 6.05306L-6.79749e-05 6.04121L0.00175553 4.218L12.3704 4.22985L9.43961 1.28536L10.7314 -2.249e-07L15.8765 5.16881L10.7077 10.313Z" fill="#E2401B"/></svg></div></div><span class="styles_Header__subNavLinkSubText__T9G0C">Compare self-hosted and cloud options</span></div></div></a></li></ul></div></div></li><li class="styles_DesktopHeader__navItem__8yJw5 mr-0" aria-label="Toggle CLOUD subnav dropdown"><a class="styles_DesktopHeader__navButton__gVr05" href="/redpanda-cloud">CLOUD</a></li><li class="styles_DesktopHeader__navItem__8yJw5 mr-0" aria-label="Toggle CUSTOMERS subnav dropdown"><a class="styles_DesktopHeader__navButton__gVr05" href="/customers">CUSTOMERS</a></li><li class="styles_DesktopHeader__navItem__8yJw5 mr-0" aria-label="Toggle BLOG subnav dropdown"><a class="styles_DesktopHeader__navButton__gVr05" href="/blog">BLOG</a></li><li class="styles_DesktopHeader__navItem__8yJw5 mr-0" aria-label="Toggle LEARN subnav dropdown"><div><button class="styles_DesktopHeader__navButton__gVr05">LEARN<span><svg width="12" height="7" viewbox="0 0 12 7" fill="none" xmlns="http://www.w3.org/2000/svg" class=""><path d="M1 1L6 6L11 1" stroke="#667085" stroke-width="1.67" stroke-linecap="round" stroke-linejoin="round"/></svg></span></button><div class="styles_DesktopHeader__subNav__LuUHn"><ul class="styles_DesktopHeader__subNavList__xd4nt"><li class="styles_DesktopHeader__subNavItem__KgS5R"><a class="styles_Header__subNavLink__HCqh7" target="_blank" rel="noopener noreferrer" href="https://docs.redpanda.com/docs/home"><div class="styles_Header__subNavLinkTextAndIcon__YD6kl"><img class="styles_Header__subNavLinkIcon__ORxzC" src="https://images.ctfassets.net/paqvtpyf8rwu/2pOpTO3vvTj6lfkIgOzfqU/9e34792be89009df7cb22f4f599c8f00/docs-icon.svg" alt="Documentation"/><div><div class="styles_Header__subNavLinkGroup__FAyTJ"><p class="styles_Header__subNavLinkText__P9brF">Documentation</p><div class="styles_Header__subNavLinkArrow___dJm0"><svg width="17" height="13" viewbox="0 0 17 16" fill="none" xmlns="http://www.w3.org/2000/svg"><path d="M10.7077 10.313L9.42138 9.02034L12.4023 6.05306L-6.79749e-05 6.04121L0.00175553 4.218L12.3704 4.22985L9.43961 1.28536L10.7314 -2.249e-07L15.8765 5.16881L10.7077 10.313Z" fill="#E2401B"/></svg></div></div><span class="styles_Header__subNavLinkSubText__T9G0C">Dive deep into features and APIs.</span></div></div></a></li><li class="styles_DesktopHeader__subNavItem__KgS5R"><a class="styles_Header__subNavLink__HCqh7" target="_blank" rel="noopener noreferrer" href="https://university.redpanda.com"><div class="styles_Header__subNavLinkTextAndIcon__YD6kl"><img class="styles_Header__subNavLinkIcon__ORxzC" src="https://images.ctfassets.net/paqvtpyf8rwu/78Ji8ECFrOzxOqvMWwYdy/c5a3551114aff387f238a2720326f40d/redpanda-u.svg" alt="Redpanda University"/><div><div class="styles_Header__subNavLinkGroup__FAyTJ"><p class="styles_Header__subNavLinkText__P9brF">Redpanda University</p><div class="styles_Header__subNavLinkArrow___dJm0"><svg width="17" height="13" viewbox="0 0 17 16" fill="none" xmlns="http://www.w3.org/2000/svg"><path d="M10.7077 10.313L9.42138 9.02034L12.4023 6.05306L-6.79749e-05 6.04121L0.00175553 4.218L12.3704 4.22985L9.43961 1.28536L10.7314 -2.249e-07L15.8765 5.16881L10.7077 10.313Z" fill="#E2401B"/></svg></div></div><span class="styles_Header__subNavLinkSubText__T9G0C">Guided courses on building real-time apps.</span></div></div></a></li><li class="styles_DesktopHeader__subNavItem__KgS5R"><a class="styles_Header__subNavLink__HCqh7" target="_self" rel="" href="/events"><div class="styles_Header__subNavLinkTextAndIcon__YD6kl"><img class="styles_Header__subNavLinkIcon__ORxzC" src="https://images.ctfassets.net/paqvtpyf8rwu/3Tq6HBZ1W3blN9ht3otKir/fdd4b4a56218173c4199424d1f4eb776/events.svg" alt="Upcoming events"/><div><div class="styles_Header__subNavLinkGroup__FAyTJ"><p class="styles_Header__subNavLinkText__P9brF">Upcoming events</p><div class="styles_Header__subNavLinkArrow___dJm0"><svg width="17" height="13" viewbox="0 0 17 16" fill="none" xmlns="http://www.w3.org/2000/svg"><path d="M10.7077 10.313L9.42138 9.02034L12.4023 6.05306L-6.79749e-05 6.04121L0.00175553 4.218L12.3704 4.22985L9.43961 1.28536L10.7314 -2.249e-07L15.8765 5.16881L10.7077 10.313Z" fill="#E2401B"/></svg></div></div><span class="styles_Header__subNavLinkSubText__T9G0C">Learn more about our upcoming events.</span></div></div></a></li><li class="styles_DesktopHeader__subNavItem__KgS5R"><a class="styles_Header__subNavLink__HCqh7" target="_self" rel="" href="/resources"><div class="styles_Header__subNavLinkTextAndIcon__YD6kl"><img class="styles_Header__subNavLinkIcon__ORxzC" src="https://images.ctfassets.net/paqvtpyf8rwu/5tRzRjoeG3Ny4TwWaF1WAv/41485d30c174cd9753e00d0cb34d3354/resources.svg" alt="On-demand resources"/><div><div class="styles_Header__subNavLinkGroup__FAyTJ"><p class="styles_Header__subNavLinkText__P9brF">On-demand resources</p><div class="styles_Header__subNavLinkArrow___dJm0"><svg width="17" height="13" viewbox="0 0 17 16" fill="none" xmlns="http://www.w3.org/2000/svg"><path d="M10.7077 10.313L9.42138 9.02034L12.4023 6.05306L-6.79749e-05 6.04121L0.00175553 4.218L12.3704 4.22985L9.43961 1.28536L10.7314 -2.249e-07L15.8765 5.16881L10.7077 10.313Z" fill="#E2401B"/></svg></div></div><span class="styles_Header__subNavLinkSubText__T9G0C">Tech talks, streamcasts, reports, and more.</span></div></div></a></li><li class="styles_DesktopHeader__subNavItem__KgS5R"><a class="styles_Header__subNavLink__HCqh7" target="_self" rel="" href="https://redpanda.com/slack"><div class="styles_Header__subNavLinkTextAndIcon__YD6kl"><img class="styles_Header__subNavLinkIcon__ORxzC" src="https://images.ctfassets.net/paqvtpyf8rwu/PMcVPLOUI853NUus1Ofqy/41306403501d2129761387a0291cc60d/slack.svg" alt="Community Slack"/><div><div class="styles_Header__subNavLinkGroup__FAyTJ"><p class="styles_Header__subNavLinkText__P9brF">Community Slack</p><div class="styles_Header__subNavLinkArrow___dJm0"><svg width="17" height="13" viewbox="0 0 17 16" fill="none" xmlns="http://www.w3.org/2000/svg"><path d="M10.7077 10.313L9.42138 9.02034L12.4023 6.05306L-6.79749e-05 6.04121L0.00175553 4.218L12.3704 4.22985L9.43961 1.28536L10.7314 -2.249e-07L15.8765 5.16881L10.7077 10.313Z" fill="#E2401B"/></svg></div></div><span class="styles_Header__subNavLinkSubText__T9G0C">Ask our dev community your burning questions.</span></div></div></a></li></ul></div></div></li><li class="styles_DesktopHeader__navItem__8yJw5 xl:mr-12" aria-label="Toggle COMPANY subnav dropdown"><div><button class="styles_DesktopHeader__navButton__gVr05">COMPANY<span><svg width="12" height="7" viewbox="0 0 12 7" fill="none" xmlns="http://www.w3.org/2000/svg" class=""><path d="M1 1L6 6L11 1" stroke="#667085" stroke-width="1.67" stroke-linecap="round" stroke-linejoin="round"/></svg></span></button><div class="styles_DesktopHeader__subNav__LuUHn"><ul class="styles_DesktopHeader__subNavList__xd4nt"><li class="styles_DesktopHeader__subNavItem__KgS5R"><a class="styles_Header__subNavLink__HCqh7" target="_self" rel="" href="/about-us/"><div class="styles_Header__subNavLinkTextAndIcon__YD6kl"><img class="styles_Header__subNavLinkIcon__ORxzC" src="https://images.ctfassets.net/paqvtpyf8rwu/5y3RsyLnrJb66KaVzyZ91d/d095abcc315fc62cd052936996a2eae3/about-icon.svg" alt="About us"/><div><div class="styles_Header__subNavLinkGroup__FAyTJ"><p class="styles_Header__subNavLinkText__P9brF">About us</p><div class="styles_Header__subNavLinkArrow___dJm0"><svg width="17" height="13" viewbox="0 0 17 16" fill="none" xmlns="http://www.w3.org/2000/svg"><path d="M10.7077 10.313L9.42138 9.02034L12.4023 6.05306L-6.79749e-05 6.04121L0.00175553 4.218L12.3704 4.22985L9.43961 1.28536L10.7314 -2.249e-07L15.8765 5.16881L10.7077 10.313Z" fill="#E2401B"/></svg></div></div><span class="styles_Header__subNavLinkSubText__T9G0C">Meet our team and investors.</span></div></div></a></li><li class="styles_DesktopHeader__subNavItem__KgS5R"><a class="styles_Header__subNavLink__HCqh7" target="_self" rel="" href="/careers"><div class="styles_Header__subNavLinkTextAndIcon__YD6kl"><img class="styles_Header__subNavLinkIcon__ORxzC" src="https://images.ctfassets.net/paqvtpyf8rwu/7wxiMVSCkCzeIlskXUh02h/5c66960f5ae6a43634e68872d64ca243/careers.svg" alt="Careers"/><div><div class="styles_Header__subNavLinkGroup__FAyTJ"><p class="styles_Header__subNavLinkText__P9brF">Careers</p><div class="styles_Header__subNavLinkArrow___dJm0"><svg width="17" height="13" viewbox="0 0 17 16" fill="none" xmlns="http://www.w3.org/2000/svg"><path d="M10.7077 10.313L9.42138 9.02034L12.4023 6.05306L-6.79749e-05 6.04121L0.00175553 4.218L12.3704 4.22985L9.43961 1.28536L10.7314 -2.249e-07L15.8765 5.16881L10.7077 10.313Z" fill="#E2401B"/></svg></div></div><span class="styles_Header__subNavLinkSubText__T9G0C">Explore Redpanda opportunities and culture.</span></div></div></a></li></ul></div></div></li><li class="styles_DesktopHeader__navItem__8yJw5 mr-0" aria-label="Toggle CONTACT subnav dropdown"><div><button class="styles_DesktopHeader__navButton__gVr05">CONTACT<span><svg width="12" height="7" viewbox="0 0 12 7" fill="none" xmlns="http://www.w3.org/2000/svg" class=""><path d="M1 1L6 6L11 1" stroke="#667085" stroke-width="1.67" stroke-linecap="round" stroke-linejoin="round"/></svg></span></button><div class="styles_DesktopHeader__subNav__LuUHn"><ul class="styles_DesktopHeader__subNavList__xd4nt"><li class="styles_DesktopHeader__subNavItem__KgS5R"><a class="styles_Header__subNavLink__HCqh7" target="_self" rel="" href="/contact"><div class="styles_Header__subNavLinkTextAndIcon__YD6kl"><img class="styles_Header__subNavLinkIcon__ORxzC" src="https://images.ctfassets.net/paqvtpyf8rwu/2kOTXYFsqojwB0lKISpmsA/9983226adbaeb5fa01bf31fd0f18d93a/inquiries.svg" alt="Request demo"/><div><div class="styles_Header__subNavLinkGroup__FAyTJ"><p class="styles_Header__subNavLinkText__P9brF">Request demo</p><div class="styles_Header__subNavLinkArrow___dJm0"><svg width="17" height="13" viewbox="0 0 17 16" fill="none" xmlns="http://www.w3.org/2000/svg"><path d="M10.7077 10.313L9.42138 9.02034L12.4023 6.05306L-6.79749e-05 6.04121L0.00175553 4.218L12.3704 4.22985L9.43961 1.28536L10.7314 -2.249e-07L15.8765 5.16881L10.7077 10.313Z" fill="#E2401B"/></svg></div></div><span class="styles_Header__subNavLinkSubText__T9G0C">Request private tech talks, pricing, product resources.</span></div></div></a></li><li class="styles_DesktopHeader__subNavItem__KgS5R"><a class="styles_Header__subNavLink__HCqh7" target="_blank" rel="noopener noreferrer" href="/slack"><div class="styles_Header__subNavLinkTextAndIcon__YD6kl"><img class="styles_Header__subNavLinkIcon__ORxzC" src="https://images.ctfassets.net/paqvtpyf8rwu/PMcVPLOUI853NUus1Ofqy/41306403501d2129761387a0291cc60d/slack.svg" alt="Join our Slack"/><div><div class="styles_Header__subNavLinkGroup__FAyTJ"><p class="styles_Header__subNavLinkText__P9brF">Join our Slack</p><div class="styles_Header__subNavLinkArrow___dJm0"><svg width="17" height="13" viewbox="0 0 17 16" fill="none" xmlns="http://www.w3.org/2000/svg"><path d="M10.7077 10.313L9.42138 9.02034L12.4023 6.05306L-6.79749e-05 6.04121L0.00175553 4.218L12.3704 4.22985L9.43961 1.28536L10.7314 -2.249e-07L15.8765 5.16881L10.7077 10.313Z" fill="#E2401B"/></svg></div></div><span class="styles_Header__subNavLinkSubText__T9G0C">Get answers from our community.</span></div></div></a></li><li class="styles_DesktopHeader__subNavItem__KgS5R"><a class="styles_Header__subNavLink__HCqh7" target="_blank" rel="noopener noreferrer" href="https://support.redpanda.com/"><div class="styles_Header__subNavLinkTextAndIcon__YD6kl"><img class="styles_Header__subNavLinkIcon__ORxzC" src="https://images.ctfassets.net/paqvtpyf8rwu/A9rREbqZidrb8bRNQXRk5/1f19c2ebac528c6e7e970481f28c64f3/support-requests.svg" alt="Support request"/><div><div class="styles_Header__subNavLinkGroup__FAyTJ"><p class="styles_Header__subNavLinkText__P9brF">Support request</p><div class="styles_Header__subNavLinkArrow___dJm0"><svg width="17" height="13" viewbox="0 0 17 16" fill="none" xmlns="http://www.w3.org/2000/svg"><path d="M10.7077 10.313L9.42138 9.02034L12.4023 6.05306L-6.79749e-05 6.04121L0.00175553 4.218L12.3704 4.22985L9.43961 1.28536L10.7314 -2.249e-07L15.8765 5.16881L10.7077 10.313Z" fill="#E2401B"/></svg></div></div><span class="styles_Header__subNavLinkSubText__T9G0C">Submit technical support ticket.</span></div></div></a></li></ul></div></div></li><li class="styles_DesktopHeader__navItem__8yJw5 mr-0" aria-label="Toggle Start for free subnav dropdown"><a href="/try-redpanda" tabindex="0" role="link" aria-label="Start for free" class="styles_DesktopHeader__button__fD4_O styles_Button__znVlr styles_Button__textCenter__mmDEU styles_Button__Small__Chq2W styles_Button__WhiteTextRedBg__Qn_Zp styles_Button--centered__vjrFc styles_Button--uppercase__lkHlN">Start for free </a></li><li class="styles_DesktopHeader__navItem__8yJw5 mr-0" aria-label="Toggle subnav dropdown"><a class="styles_DesktopHeader__socialLink__4vjjL styles_DesktopHeader__navButton__gVr05" target="_blank" rel="noopener noreferrer" href="https://redpanda.com/slack"><img class="styles_DesktopHeader__socialIcon__KnRth" src="https://images.ctfassets.net/paqvtpyf8rwu/69HyyrugepthrFF25coB86/bbc52835d1aa2c5714b30f62a957133a/slack__1_.svg" alt="Slack (homepage header)" width="30" height="30"/> </a></li><li class="styles_DesktopHeader__navItem__8yJw5 mr-0" aria-label="Toggle 4.3K subnav dropdown"><a class="styles_DesktopHeader__socialLink__4vjjL styles_DesktopHeader__navButton__gVr05" target="_blank" rel="noopener noreferrer" href="https://github.com/redpanda-data/redpanda/"><img class="styles_DesktopHeader__socialIcon__KnRth" src="https://images.ctfassets.net/paqvtpyf8rwu/6ZoZUb18LuxNlX5qyhD1vI/08a798d68104fe6a7f9b14d7251a45a3/github__1_.svg" alt="Github" width="30" height="30"/>4.3K</a></li></ul></nav></header><header role="contentinfo" class="styles_MobileHeader__QkweI"><div class="styles_MobileHeader__wrapper__S5jaI"><a aria-label="Red Panda Logo" href="/"><img src="https://images.ctfassets.net/paqvtpyf8rwu/5wODDUzLP7teOGDS7M5m39/466f646e44c2ccc4e78236b17d7d68ac/Logo.svg" alt="" title="Redpanda Logo" width="155" height="36"/></a><button class="styles_MobileHeader__mobileNavToggle__Jlbag" role="button" tabindex="0" aria-label="Toggle mobile navigation"><svg class="styles_MobileHeader__mobileNavToggleSvg__z4rWs" xmlns="http://www.w3.org/2000/svg" viewbox="0 0 100 100"><g fill="none" fill-rule="evenodd" stroke="#344054" stroke-linecap="square" stroke-width="10"><line x1="9" y1="22" x2="91" y2="22" id="top"/><line x1="9" y1="50" x2="91" y2="50" id="middle"/><line x1="9" y1="78" x2="91" y2="78" id="bottom"/></g></svg></button></div><nav class="styles_MobileHeader__mobileNavListWrapper__rdJoG"/></header></div><main class="styles_Main__Lod2n"><div class="styles_BlogPostTemplate__xziin"><section aria-labelledby="Blog header section." class="styles_SectionContainer__peieG" id="undefined"><div class="styles_SectionContainer__innerContainer__fsrG8 styles_SectionContainer__container--relative__rLVsP styles_SectionContainer__yPaddingOff__5AMpG"><div class="styles_BlogPostTemplate__inlineNavigation__mjkO4"><a class="styles_BlogPostTemplate__inlineNavigationItem__O_5gT" href="/"><img src="/images/home-icon.svg" alt="Red Panda logo"/></a><img src="/images/blog-nav-arrow.svg" alt="Chevron right icon"/><a class="styles_BlogPostTemplate__inlineNavigationItem__O_5gT" href="/blog">Blog</a></div><div class="styles_BlogPostTemplate__header__xETQr"><div class="styles_BlogPostTemplate__headerCategoriesAndTimeRead__Jkjmz"><a class="styles_BlogPostTemplate__headerCategory__gJTTK" href="/blog/categories/engineering"><span class="styles_BlogPostTemplate__headerCategories__pwugz">Engineering</span></a></div><h1 class="styles_RenderBreakLineText__nTFTN styles_BlogPostTemplate__headerTitle__tPZrK"/><p class="styles_BlogPostTemplate__headerDescription__HcXl_">Learn how out of memory events created challenges for our team, and how we solved them.</p><div class="styles_BlogPostTemplate__headerAuthorsAndDate__6EY9N"><span>By</span><span><a class="styles_BlogPostTemplate__headerAuthor__xNOCU" href="/blog/authors/rafal-korepta">Rafał Korepta</a></span><span>on</span><span>July 7, 2022</span></div><div class="styles_BlogPostTemplate__headerImageWrapper__xPKAF"><img src="https://images.ctfassets.net/paqvtpyf8rwu/6Q50YM3ltxfSg3uMJ25GeP/d4410c7db43a02ffcd25ba7e40831a64/redpanda-OOM-h.png" alt="Solving challenges caused by Out Of Memory (OOM) Killer in Linux"/></div><div class="styles_BlogPostSocialShare__8KTAp"><div class="styles_BlogPostSocialShare__shareButtons__Zu0HF"><a href="https://github.com/redpanda-data/redpanda/" target="_blank" rel="noopener noreferrer" class="styles_BlogPostSocialShare__shareButton__3UdFX"><img src="/images/share-buttons-icons/github.svg" alt="Social icon github" title="Share on github"/></a><a href="https://redpandacommunity.slack.com/join/shared_invite/zt-ng2ze1uv-l5VMWSGQHB9gp47~kNnYGA#/shared-invite/email" target="_blank" rel="noopener noreferrer" class="styles_BlogPostSocialShare__shareButton__3UdFX"><img src="/images/share-buttons-icons/slack.svg" alt="Social icon slack" title="Share on slack"/></a><a href="http://twitter.com/share?url=https%3A%2F%2Fredpanda.com%2Fblog%2Fsolve-out-of-memory-killer-events&amp;text=Solving challenges caused by Out Of Memory (OOM) Killer in Linux by @redpandadata #redpanda #apachekafka" target="_blank" rel="noopener noreferrer" class="styles_BlogPostSocialShare__shareButton__3UdFX"><img src="/images/share-buttons-icons/twitter.svg" alt="Social icon twitter" title="Share on twitter"/></a><a href="https://www.facebook.com/sharer/sharer.php?u=https%3A%2F%2Fredpanda.com%2Fblog%2Fsolve-out-of-memory-killer-events" target="_blank" rel="noopener noreferrer" class="styles_BlogPostSocialShare__shareButton__3UdFX"><img src="/images/share-buttons-icons/facebook.svg" alt="Social icon facebook" title="Share on facebook"/></a><a href="https://www.linkedin.com/shareArticle?mini=true&amp;url=https%3A%2F%2Fredpanda.com%2Fblog%2Fsolve-out-of-memory-killer-events" target="_blank" rel="noopener noreferrer" class="styles_BlogPostSocialShare__shareButton__3UdFX"><img src="/images/share-buttons-icons/linkedin.svg" alt="Social icon linkedin" title="Share on linkedin"/></a></div><div class="styles_BlogPostSocialShare__shareEmailAndCopy__yRire"><a href="mailto:?subject=Solving challenges caused by Out Of Memory (OOM) Killer in Linux&amp;body=https%3A%2F%2Fredpanda.com%2Fblog%2Fsolve-out-of-memory-killer-events" class="styles_BlogPostSocialShare__shareButton__3UdFX"><img src="/images/share-buttons-icons/email.svg" alt="email icon" title="Share by email"/></a><button class="styles_BlogPostSocialShare__shareCopy__OseKa styles_BlogPostSocialShare__shareButton__3UdFX" role="button" aria-label="Copy link to clipboard"><img src="/images/share-buttons-icons/copy.svg" alt="copy icon" title="Copy link"/><p>Copy link</p></button></div></div></div></div></section><section aria-labelledby="Blog post content section." class="styles_SectionContainer__peieG styles_BlogPostTemplate__content__wrapper__6DRW_" id="undefined"><div class="styles_SectionContainer__innerContainer__fsrG8 styles_SectionContainer__container--relative__rLVsP styles_SectionContainer__yPaddingOff__5AMpG"><div class="styles_BlogPostTemplate__content__with_sidebar__pzJ5V"><div class="col-span-2"><div class="styles_Marked__pncG4 styles_Marked__container__T_BSl"><h2 id="introduction" level="2" node="[object Object]">Introduction</h2> <p>Out of memory (OOM) events are common in the Linux environment when there are programs that allocate a lot of memory. Redpanda is one such program, as it uses the <a href="http://seastar.io/" target="_blank" rel="nofollow noopener noreferrer" node="[object Object]"> Seastar</a> library, which tries to utilize whole hardware to its limits.</p> <p>There is special kernel functionality, called Out Of Memory Killer (OOM Killer), that helps keep Linux machines operational by killing the biggest process with the least priority. OOM Killer can recognize and respect processes that have constraints in Linux <code class="styles_Marked__inlineCode__2XDzC">cgroups</code>.</p> <p>Unless you specify input parameters, Redpanda reads hardware-available memory and sets aside at least 1.5 GiB for the operating system (OS) and divides the rest equally for each machine core in order to maximize efficiency of the Seastar memory allocator. If Redpanda is running alongside other programs, the Linux OS might run out of memory.</p> <p>If you’ve also experienced problems with OOM Killer, keep reading to learn how we resolved our issues with it so you can do the same.</p> <h2 id="how-oom-killer-began-interrupting-our-sidecar" level="2" node="[object Object]">How OOM Killer began interrupting our sidecar</h2> <p>When we began experiencing problems with OOM Killer, Redpanda Cloud used (and still does) Kubernetes (K8s), and relied on <code class="styles_Marked__inlineCode__2XDzC">cgroups</code> and Linux namespaces to constrain the workloads. If Redpanda wasn’t told what memory parameters it should pick, then the underlying Seastar library would allocate 1.5 GiB for OS, and the rest from the <code class="styles_Marked__inlineCode__2XDzC">cgroup</code> would be divided among the number of available CPU cores.</p> <p>Such a setup didn’t make sense for a containerized environment where Redpanda was isolated from any other process. Hypothetical users of the Redpanda operator shouldn’t have to worry about how to set up the Redpanda advanced memory parameters but, depending on your desired capacity, adequate K8s nodes must be available for Redpanda, and correct limits and requests need to be set. The first sizing for the Redpanda pod in K8s reserved 0.5 GiB of the memory to the other pods running in a dedicated Redpanda node.</p> <p>To automate and ease the K8s deployment of Redpanda, we created an operator. In order to constrain Redpanda and leverage <code class="styles_Marked__inlineCode__2XDzC">cgroup</code> capability, we provided a resource configuration option in the cluster custom resource. This configuration was mapped directly to the Redpanda configuration so that Redpanda could use all memory available to the container.</p> <p>In our first Redpanda operator implementation, the K8s deployment resource was configured to not overwrite the container entry point. The default entry point leveraged <a href="https://github.com/redpanda-data/supervisord" target="_blank" rel="noopener noreferrer" node="[object Object]"> supervisord</a> to schedule Redpanda processes, telemetry reporting, and WebAssembly (Wasm) coprocessors. That simplification played a role in local environment deployments (e.g. <code class="styles_Marked__inlineCode__2XDzC">docker run</code>).</p> <p>When Redpanda warmed up its cache, OOM Killer saw that memory inside the Redpanda <code class="styles_Marked__inlineCode__2XDzC">cgroup</code> was exhausted, and it killed the biggest Redpanda process. Users would see that the broker was unavailable until container runtime restarted the Redpanda process. The Redpanda operator could automate the same function as supervisord by scheduling only one process inside one container, and the container runtime would do the heavy lifting and isolate each process. Debugging further problems was made easier by the fact that OOM Killer recognized individual processes and only those were affected.</p> <p>The first solution we tried to resolve the OOM Killer events involved the K8s deployment, where every process was running in its own dedicated container. By investigating this potential solution, we saw that <code class="styles_Marked__inlineCode__2XDzC">rpk debug info</code>, <a href="https://docs.redpanda.com/docs/reference/rpk-commands/#rpk-debug" node="[object Object]">which sends telemetry data</a>, was executed every 10 minutes. The problem was that Redpanda had a higher-than-usual load, and our sidecar used more memory than was set in <code class="styles_Marked__inlineCode__2XDzC">cgroup</code>. Then the OOM Killer started to kill this sidecar container.</p> <p>Next, the Cloud team optimized the managed solution, so we eliminated all sidecars from the deployment. The telemetry was moved outside the Redpanda pod and Wasm coprocessors were disabled until GA. With only one Redpanda process running in the pod, memory <code class="styles_Marked__inlineCode__2XDzC">cgroup</code> constraints were mapped to Redpanda memory. In long-running clusters, memory allocation grew to the point where, from the OS perspective, all available memory was consumed by Redpanda. The processes were again killed by OOM Killer. At this point, we were looking for a bug in Redpanda, but it turns out that K8s pod implementation is backed by <a href="https://github.com/kubernetes/kubernetes/tree/master/build/pause" target="_blank" rel="nofollow noopener noreferrer" node="[object Object]"> a pause container</a>.</p> <h2 id="solving-the-oom-killer-challenge" level="2" node="[object Object]">Solving the OOM Killer challenge</h2> <p>To create a container sandbox and be able to restart individual containers in a multi-container pod setup, pause processes play a crucial role to orchestrate other processes. Looking at the source code, this process might seem to not be that big in terms of memory, but it needs one page from the operating system just to work. This one page plays a key role when OOM Killer scans all <code class="styles_Marked__inlineCode__2XDzC">cgroups</code>, and finds that the Redpanda container overflows its memory usage.</p> <p>Once the OOM Killer report proved that the pause container was listed along Redpanda process, we implemented memory reservation to solve this issue. With a single container, we couldn’t allocate whole memory to the Redpanda process. <a href="https://github.com/redpanda-data/redpanda/pull/3773" target="_blank" rel="noopener noreferrer" node="[object Object]"> The Redpanda operator extends</a> cluster custom resource definition to include Redpanda resource configuration. Now, <code class="styles_Marked__inlineCode__2XDzC">cgroup</code> memory is not tight with Redpanda memory maximum allocation. Depending on the K8s worker node size and the traffic in particular, node clients can assign less memory to Redpanda in comparison to the container.</p> <p>The next improvement we made to resolve our issues with OOM Killer was to <a href="https://github.com/redpanda-data/redpanda/pull/3831" target="_blank" rel="noopener noreferrer" node="[object Object]"> add 10% default memory reservation</a> to the OS. This was done in order to prevent memory pressure in overprovisioned K8s worker nodes. If Redpanda operator users would not set Redpanda memory, then — in big enough clusters where all memory limit was distributed among all pods — clients could observe memory pressure events. With spikes in traffic and Kafka clients' usage, the SRE team might observe that default kubelet memory host reservation is not enough for the operating system. This 10% memory reservation mitigation was implemented to help clients that were using the Redpanda operator already. An operator upgrade would recalculate necessary memory reservation. This solution, instead, gives room for a pause container and other kernel data structures that are necessary for the K8S node to work correctly.</p> <h2 id="optimizing-resource-consumption-in-bigger-machines" level="2" node="[object Object]">Optimizing resource consumption in bigger machines</h2> <p>In the bigger clusters (e.g. 16 cores and 64 GiB), Redpanda needs to give more room to the auxiliary services. Each core will be occupied by the Redpanda shard. That shard doesn’t overload the metrics system or logging aggregator but, when it’s multiplied by the number of cores, it can significantly change the resource requirements (for example, Prometheus for metrics or FluentBit for logging). While OOM Killer was looking at the biggest processes with the lowest priority inside each <code class="styles_Marked__inlineCode__2XDzC">cgroup</code>, Redpanda was picked to be terminated. K8s node-exporter started to report node memory pressure events. For our biggest deployments we adjusted memory to leave more room for <code class="styles_Marked__inlineCode__2XDzC">logging collector</code>, <code class="styles_Marked__inlineCode__2XDzC">kubelet</code>, <code class="styles_Marked__inlineCode__2XDzC">node-exporter</code>, and <code class="styles_Marked__inlineCode__2XDzC">kube-proxy</code>.</p> <p>Ironically, what's interesting is that, to prevent OOM kills of Redpanda, we actually reduced the amount of memory Redpanda used. Firstly by reducing the amount of memory allocated to the <code class="styles_Marked__inlineCode__2XDzC">cgroup</code>, and then by reducing the amount of memory Seastar can use within that <code class="styles_Marked__inlineCode__2XDzC">cgroup</code>.</p> <h2 id="how-to-adjust-default-memory-allocation" level="2" node="[object Object]">How to adjust default memory allocation</h2> <p>Despite encountering these challenges with the OOM Killer, we were able to effectively troubleshoot these memory usage issues. We are now more mindful about resource constraints in a containerized environment. All improvements were done to our observability stack and Redpanda operator to ease the debugging experience of losing Redpanda nodes.</p> <p>For any user of the Redpanda operator, the most important thing is to understand that, by default, <strong>the operator will assign 10% of the provided K8s resource requests.</strong></p> <p>If users want to change the 10% threshold in the cluster custom resource section, they must calculate requests, limits, and Redpanda options to match the desired configuration:</p> </span><span/><span class="token token key" style="color:#c792ea">kind</span><span class="token token" style="color:#89ddff">:</span><span> Cluster </span><span/><span class="token token key" style="color:#c792ea">spec</span><span class="token token" style="color:#89ddff">:</span><span> </span><span> </span><span class="token token key" style="color:#c792ea">resources</span><span class="token token" style="color:#89ddff">:</span><span> </span><span> </span><span class="token token key" style="color:#c792ea">requests</span><span class="token token" style="color:#89ddff">:</span><span> </span><span> </span><span class="token token key" style="color:#c792ea">cpu</span><span class="token token" style="color:#89ddff">:</span><span> </span><span class="token token" style="color:#fd9170">2</span><span> </span><span> </span><span class="token token key" style="color:#c792ea">memory</span><span class="token token" style="color:#89ddff">:</span><span> 2.23Gi </span><span> </span><span class="token token key" style="color:#c792ea">redpanda</span><span class="token token" style="color:#89ddff">:</span><span> </span><span> </span><span class="token token key" style="color:#c792ea">cpu</span><span class="token token" style="color:#89ddff">:</span><span> </span><span class="token token" style="color:#fd9170">2</span><span> </span><span> </span><span class="token token key" style="color:#c792ea">memory</span><span class="token token" style="color:#89ddff">:</span><span> 2Gi </span><span> </span><span class="token token key" style="color:#c792ea">limits</span><span class="token token" style="color:#89ddff">:</span><span> </span><span> </span><span class="token token key" style="color:#c792ea">cpu</span><span class="token token" style="color:#89ddff">:</span><span> </span><span class="token token" style="color:#fd9170">2</span><span> </span><span> </span><span class="token token key" style="color:#c792ea">memory</span><span class="token token" style="color:#89ddff">:</span><span> 2.23Gi</span></code></div></pre> <p>If they do not need to change the 10% memory cushion, the Redpanda section can be omitted.</p>