Quick guess. The allocation is in bytes, but the indexing is in char.

Encoding issue?

the exception indicates that you somehow overflow the destChars, hence the size is probably wrong. so the only issue i can think is there might be a problem with the linq Sum, if its lazy it might give wrong calculation, but I'm not sure.

GetNativeTempBuffer allocates less than Size bytes?

Assuming that query.FieldsToFetch is a series of IList<char>, then: you are assembling the total length of all characters to put them into a single buffer area to hash. The problem is that char is two bytes wide. You are counting the number of chars, but context.GetNativeTempBuffer() is expecting the number of bytes, which is twice the number of chars. This means you overrun your buffer exactly halfway through loading the string into the buffer.

This would definitely take a while to identify without your electric fence. Nice catch!

This ought to be a trivial matter for a static code analyzer to find. Isnt there any good ones available for unsafe c# code?

Nice catch. I would love to see you more going into the BlittableJson stuff. How do you deal with updates to a large document and will be there a BlittableJsonWriter?

I have played around with the Blittable classes a bit. The managed heap is basically empty if I deserialize a 200MB json file. That is great. Even when I access the strings as LazyStrings it is still quite fast. But what is the story when you have deseralized the json into a BlittableJsonReaderObject and you want to keep it? You cannot add or change items in it since it is read only. Is this class meant as intermediary only to later materialize a normal CLR object or is it possible to mutate the BlittableJsonReader..... objects. If you still need to copy things into normal objects you loose the gained speed while reading very quickly. My current test looks like

                using (var jsonContext = new JsonOperationContext(20 * 1024 * 1024, 30 * 1024 * 1024))
                {
                    var json = jsonContext.ReadForDisk(inFile, "state");
                    var list = (BlittableJsonReaderArray)json["LargeList"];
                    List<string> data = new List<string>();
                    
                    var strSearch = "List Element 9999999";
                    for (int i=0;i<list.Length;i++)
                    {
                        var tmp = (LazyStringValue) list[i];
                        if(strSearch == tmp )
                        {
                            Console.WriteLine($"Found at {i}");
                        }
                        
                    }
                    json.Dispose();
                }

which uses this definition

    [ProtoContract]
    [DataContract]
    public class Data
    {
        [ProtoMember(1)]
        [DataMember(Order=1)]
        public string Name
        {
            get;
            set;
        }

        [ProtoMember(2)]
        [DataMember(Order = 2)]
        public string Description
        {
            get;
            set;
        }

        [ProtoMember(3)]
        [DataMember(Order = 3)]
        public List<string> LargeList
        {
            get;
            set;
        }

        internal DynamicJsonValue ToJson()
        {
            return new DynamicJsonValue
            {
                ["Name"] = Name,
                ["Description"] = Description,
                ["LargeList"] = new DynamicJsonArray(LargeList),
            };
        }
    }

Ahh ok. So this whole Blittable thing is only there to make queries over many small or some large documents cheap for the managed heap to prevent long GC pauses if the next Gen2 or Gen1 from the finalizer due to low memory kicks in? I was hoping to get something faster than Protocol Buffers or JSON.NET which also allows mutation of the read objects. Of course it is possible to come up with hybrid objects which store the modifications in normal objects but that is not something for the generic case. No matter how hard I try I seem to hit a wall at ca. 50-80 MB/s with managed code. The only thing left would be to deserialize at different locations from the stream in parallel. But that would impose severe limitations on the object design.

I currently have a large (up to 200 MB) Xml document serialized with DataContracts on disk which needs to be deserialized. I am searching for something significantly faster than plain DataContracts. If I redesign that stuff I want to use the fastest library out there. Blittable Json seems like a good idea if I can easily round trip the data with modifications. I care less about serialization but mostly about deserialization performance. So far it looks like JSON.NET would be significantly faster without requiring many changes to the current object model. While reading Blittable JSON is very fast I would loose the gained speed when I need to convert the Blittable JSON into the original object model back again. Since Blittable JSON is read only I cannot toss the original object model out of the window since the whole thing needs to be modifyable as well.

Hard to tell since ther are quite some types serialized into that container. It will be mostly read only but for some state machine states I am not so sure if they will not change anymore after they have been deserialized back. I mainly edit other peoples code where I am seldom 100% sure what it exactly does ;-).