Nifi OutOfMemoryError : Java heap space with a spe... - Cloudera Community

link管理

链接快照平台

输入网页链接，自动生成快照
标签化管理网页链接

相关文章推荐

会搭讪的茄子 · 在 Amazon EMR 上使用 LDAP ...· 4 天前 ·

聪明伶俐的白开水 · 如何使用Python ...· 2 月前 ·

傲视众生的抽屉 · 电影版《想见你》上映，原班人马续写“爱情范本 ...· 2 月前 ·

傲视众生的盒饭 · 神马书屋_神马书屋.产品中心第506章· 2 月前 ·

欢乐的砖头 · 莎士比亚十四行诗 Sonnet 18 | ...· 4 月前 ·

怕老婆的皮带 · v1.1.6版本是否支持jdk17? · ...· 4 月前 ·

I am facing a memory issue with a specific processor ConvertJSONToSQL

Not sure if there is a way to configure processor specific memory, as when this error comes i found other processor and services running fine in nifi.

Is there a way to address this apart from looking extending heapspace in general to whole nifi , currently its 8GB.

Below is the stack trace of error:

2018-02-09 05:42:53,197 ERROR [Timer-Driven Process Thread-1] o.a.n.p.standard.ConvertJSONToSQL ConvertJSONToSQL[id=6f1f8d3a-20b3-3fc5-884e-73f1e7443b58] ConvertJSONToSQL[id=6f1f8d3a-20b3-3fc5-884e-73f1e7443b58] failed to process session due to java.lang.OutOfMemoryError: Java heap space: {} java.lang.OutOfMemoryError: Java heap space at java.util.Arrays.copyOf(Arrays.java:3236) at java.io.ByteArrayOutputStream.grow(ByteArrayOutputStream.java:118) at java.io.ByteArrayOutputStream.ensureCapacity(ByteArrayOutputStream.java:93) at java.io.ByteArrayOutputStream.write(ByteArrayOutputStream.java:153) at java.io.DataOutputStream.writeLong(DataOutputStream.java:224) at org.apache.nifi.repository.schema.SchemaRecordWriter.writeFieldValue(SchemaRecordWriter.java:117) at org.apache.nifi.repository.schema.SchemaRecordWriter.writeFieldRepetitionAndValue(SchemaRecordWriter.java:69) at org.apache.nifi.repository.schema.SchemaRecordWriter.writeRecordFields(SchemaRecordWriter.java:56) at org.apache.nifi.repository.schema.SchemaRecordWriter.writeRecordFields(SchemaRecordWriter.java:47) at org.apache.nifi.repository.schema.SchemaRecordWriter.writeFieldValue(SchemaRecordWriter.java:147) at org.apache.nifi.repository.schema.SchemaRecordWriter.writeFieldRepetitionAndValue(SchemaRecordWriter.java:96) at org.apache.nifi.repository.schema.SchemaRecordWriter.writeRecordFields(SchemaRecordWriter.java:56) at org.apache.nifi.repository.schema.SchemaRecordWriter.writeRecordFields(SchemaRecordWriter.java:47) at org.apache.nifi.repository.schema.SchemaRecordWriter.writeFieldValue(SchemaRecordWriter.java:143) at org.apache.nifi.repository.schema.SchemaRecordWriter.writeFieldRepetitionAndValue(SchemaRecordWriter.java:69)

Thanks,

Tarun Kumar

You should modify bootstrap.conf file. Increase moemory size of java.arg.2 ve java.arg.3 like:

java.arg.2=-Xms512m
java.arg.3=-Xmx512m
java.arg.2=-Xms1024m
java.arg.3=-Xmx1024m

@Tarun Kumar @shreyanshu pare

Increasing the Heap size will solve your interim problem. However, I will highly recommend using the best practices for setting up high performing NiFi workflow, mentioned here .

Depending on your operations, you find bumped up heap size insufficient in future. So a couple of reasons, well documented by @Matt Clarke , as to why you may see a very high heap usage.

Every FlowFile that exists consists of two parts, FlowFile content and FlowFile Attributes. While the FlowFile's content lives on disk in the content repository, NiFi holds the "majority" of the FlowFile attribute data in the configured JVM heap memory space. I say "majority" because NiFi does swapping of Attributes to disk on any queue that contains over 20,000 FlowFiles (default, but can be changed in the nifi.properties).

So some common reason for running out of heap memory include:

1. High volume dataflow with lots of FlowFiles active any any given time across your dataflow. (Increase configured nifi heap size in bootstrap.conf to resolve)

2. Creating a large number of Attributes on every FlowFile. More Attributes equals more heap usage per FlowFile. (Increase configured nifi heap size in bootstrap.conf to resolve and/or reduce the configured swap threshold)

3. Writing large values to FlowFile Attributes. Extracting large amounts of content and writing it to an attribute on a FlowFile will result in high heap usage. Try to avoid creating large attributes when possible. (Increase configured nifi heap size in bootstrap.conf to resolve and/or reduce the configured swap threshold)

4. Using the MergeContent processor to merge a very large number of FlowFiles. NiFi can not merge FlowFiles that are swapped, so all these FlowFile's attributes must be in heap when the merge occurs. If merging a very large number of FlowFiles is needed, try using two MergeContent processors in series with one another. Have first merge a max of 10,000 FlowFiles and the second then merge those 20,000 FlowFile files in to even larger bundles. (Increase configured nifi heap size in bootstrap.conf also help)

5. Using the SplitText processor to split one File in to a very large number of FlowFiles. Swapping of a large connection queue will not occur until after the queue has exceeded swapping threshold. The SplitTEXT processor will create all the split FiLowFiles before committing them to the success relationship. Most commonly seen when SpitText is used to split a large incoming FlowFile by every line. It is possible to run out of heap memory before all the splits can be created. Try using two SplitText processors in series. Have the first split the incoming FlowFiles in to large chunks and the second split them down even further. (Increase configured nifi heap size in bootstrap.conf also help)

Hope that helps!

Terms & Conditions

Unsubscribe / Do Not Sell My Personal Information

Supported Browsers Policy

Apache Hadoop and associated open source project names are trademarks of the Apache Software Foundation. For a complete list of trademarks, click here.