添加链接
link管理
链接快照平台
  • 输入网页链接,自动生成快照
  • 标签化管理网页链接

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement . We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

I am wondering whether or not you guys would be open to merging a pull request where the GcStats also collects how much memory is rooted, which in effect, would let you know how much the result object consumes, not just the total allocated bytes.

This would involve changing the GcStats ReadInitial() and ReadFinal() to include output of GC.GetTotalMemory(true) if an optional setting on [MemoryDiagnoser] is set.

[MemoryDiagnoser(TotalMemory = true)]

I have been recently been looking at optimizing memory consumption, and have had to revert to a simple unit test framework that gets the above stat before and after.

But would like to integrate it with my benchmarks that get generated out of BenchmarkDotNet as well.

Names are definitely up for discussion.

@michal-ciechan hello Michał

Could you provide an example with an explanation which could take an advantage of this extra statistics?

I just try to understand the benefits.

Hey @adamsitnik

A question I needed to find out/answer in the last couple of weeks was, how much memory (that ends up taken by the final object, and not the total bytes allocated during creation) do the following take:

Dictionary<int, object>
ImmutableDictionary<int,object>
List<object> <--- This I could roughly workout as its 4*2^x*64bits (approx) that satisfied the count, of-course can TrimExcess() as and when needed, or set capacity to lower, once finished but leaving some room to grow.
SortedList<ValueTuple<int,object>>

Say you have 50k IDs, and you want to map it to something else,

How much memory will a Dictionary<int,object> take vs creating a large object[] / List<object> , and at what point does it roughly become feasible to stop having a List<object> and switch to something like Dictionary<int,object> .

E.g. if you have 50k unique ID's in the range of 0 - 50k, of course nothing will beat Object[50,000] / List<object>(50,000) .

But what if you have 50k ID's in the following ranges

0-75k
0-100k
0-150k
0-250k
0-500k

What is the point where it is better to use a Dictionary vs a large Array / List with gaps as both offer O(1) lookups. And then, how do they compare to other collection classes out there (even ones which do not offer O(1) lookups as O(log n) can be acceptable for certain use cases.

I have previously used methods like the following to get this data and then manually output to a spreadsheet and chart it:

public T GetSize<T>(Func<T> func, out long memConsumed)
	GC.Collect();
	GC.WaitForPendingFinalizers();
	var memBefore = GC.GetTotalMemory(true);
	var res = func();
	GC.Collect();
	GC.WaitForPendingFinalizers();
	var memAfter = GC.GetTotalMemory(true);
	memConsumed = memAfter - memBefore;
        // Makes sure the final result does not get GC'ed
	return res;

For the sake of above method, I was more than happy to run 2 GC.Collect()'s before and after creating the object, although it might have been an overkill, but never had the time to test whether GC.GetTotalMemory(true) will also wait and collect objects with finalizers.

Currently from what I have seen, BenchmarkDotNet GcStats only collects AppDomain.CurrentDomain.MonitoringTotalAllocatedMemorySize which gets an indication of memory pressure during a benchmark, but not necessarily the final object's memory consumption (excluding the collected/collectable memory).

The reason I would have to add it to the MemoryDiagnoser, is because it would have to guarantee to run right before the benchmark, and right after it, before anything else gets allocated on the heap.

If you don't think it would be useful, you are more than welcome to close this issue!

GC.GetTotalMemory returns the GC heap size

It's very tricky. Consider following examples:

byte[] ReturnsResult()
    byte[] array = null;
    for (int i = 0; i < 1000; i++)
        array = new byte[8];
    return array;

In this case we have allocated an array of 8 bytes 1000 times, but the heap has grew only by the size of one array.

void DoesNotReturnTheResult()
    byte[] array = null;
    for (int i = 0; i < 1000; i++)
        array = new byte[8];
    GC.KeepAlive(array);

In this case we have allocated an array of 8 bytes 1000 times, but the heap did not grew because the result was not returned from the method, hence not stored anywhere so the GC has cleaned it up and the diff after - before will be 0.

With allocated bytes it's clear and obvious. With heap size it's not clear and you need to take the side effects under consideration.

I am not saying that adding this feature is a bad idea, but explaining what it really does is.

@michal-ciechan if you can provide a PR which does not change current behavior, explains in the docs what is measured by the new statistic and give it a good name with some unit tests then please feel free to create the PR and I will be very happy to merge it.

@adamsitnik Thanks, I will work on it over next couple of weeks (although I am away a couple of times in next 3 weeks so probably a little longer than I would have hoped).

Regarding the first example, I would only expect the counter to return 8 bytes as that is what is left on the Heap after GC.Collect and it is precisely what I am trying to measure, how much is left on the heap of the resulting object.

Of course, if I want a set of objects I have to wrap them in an array or class so that they don't get GC'ed.

Basically, I am more interested in a measure long-term heap memory consumption of an object, rather than all allocations on the heap during creating the object.

E.g. You create a Dictionary, and you add 50k items to it, I am sure it will re-create the hash table array internally a number of times to accomodate more and more objects, but I am only interested in the size of the final object that remains on the heap, and not the interim allocations.

In your second example, this counter would be useless, as you mentioned, the object doesn't get returned, therefore get's GC'ed and result is 0. This counter is used to determine the amount of memory the resulting object takes up on the heap (after GC).

Not everything has to go in a benchmarking library; this library specifically or any other more generally. There are debugging tools that can be used to determine accurately an object’s heap consumption. For me, the MemoryDiagnoser is mostly about helping identify excessive allocation patterns, which has a direct and immediate effect on GC performance. Object size on the heap has only a very indirect relationship to performance, and sounds like one of many useful metrics that can be obtained using other tools. There is also no need in the entire BDN infrastructure for running the benchmark code repeatedly and caring for iteration counts and outliers - the object size would not change between runs. So basically, not something that really belongs in a benchmarking library.

@goldshtn Yes, you have a valid point, absolutely no need to run GC.Collect() so many times just to find this out. Therefore would be more than willing to add code to only run it once during warmup.

I currently have custom unit tests to find this out, but it means I usually want to map it back to speed benchmarks, therefore need to manually join the results from BDN to the results from other tools, and re-create the graphs.