I’m writing code that needs to read and write (overlapped) from a custom USB device. To test that this works I have written a C# Win32 .Net Framework console app which successfully uses FileStream to talk to the USB device. Here’s a snippet from the working code

var hidHandle = CreateFile(devpath, FileAccess.ReadWrite, FileShare.ReadWrite, IntPtr.Zero, FileMode.Open, FileOptions.Asynchronous, IntPtr.Zero);
IntPtr lpData;
if (HidD_GetPreparsedData(hidHandle, out lpData))
    HidCaps devcaps;
    HidP_GetCaps(lpData, out devcaps);
    InputReportByteLength = devcaps.InputReportByteLength; // 65
    OutputReportByteLength = devcaps.OutputReportByteLength; // 65
    fs = new FileStream(hidHandle, FileAccess.ReadWrite, InputReportByteLength, true);
I can then use fs to read and write to the USB device.
When I copy the code over into Unity the call to the FileStream constructor fails and I get this exception thrown:
Invalid handle

at System.IO.FileStream…ctor (IntPtr handle, FileAccess access, Boolean ownsHandle, Int32 bufferSize, Boolean isAsync, Boolean noBuffering) [0x0007d] in /Users/builduser/buildslave/mono/build/mcs/class/corlib/System.IO/FileStream.cs:97

at System.IO.FileStream…ctor (IntPtr handle, FileAccess access, Boolean ownsHandle, Int32 bufferSize, Boolean isAsync) [0x00000] in :0

at System.IO.FileStream…ctor (Microsoft.Win32.SafeHandles.SafeFileHandle handle, FileAccess access, Int32 bufferSize, Boolean isAsync) [0x00000] in :0

at (wrapper remoting-invoke-with-check) System.IO.FileStream:.ctor (Microsoft.Win32.SafeHandles.SafeFileHandle,System.IO.FileAccess,int,bool)

at Zanzibar.HIDConnector.HIDDevice.USBThreadWorker () [0x0006a] in C:\Users\	imregan\Source\Zanzibar\Software\Playground\UnityHID\HidTest\Assets\HIDDevice.cs:81
The file handle is not invalid, it has already been used successfully in the code fragment to obtain the InputReportByteLength.
I do not understand why code that works under .Net 2.0 targetting Win32 is failing under Unity. My guess is that it is something to do with overlapping ‘file’ access. Note that the FileOptions.Asynchronous flag passed into this external call is the FILE_FLAG_OVERLAPPED flag in CreateFile
[DllImport("kernel32.dll")]
protected static extern SafeFileHandle CreateFile(String pFileName, FileAccess dwDesiredAccess, FileShare dwShareMode, IntPtr pSecurityAttributes, FileMode dwCreationDisposition, FileOptions dwFlagsAndAttributes, IntPtr hTemplateFile);
Does anyone know how to use FileStream to talk to an overlapped HID device ‘file’ in Unity? Or how to diagnose why this works in a console app and not under Unity?
              
Short Answer, you can’t.  If you try to open a device string, it will say “invalid directory” because it validates it as a directory instead of a device.  You do the same with a handle, it checks the file type and it returns “unknown” so it spits out invalid handle.
Have to do what I did is either replace the version of mono or use harmony to patch a hack.  Either way it sucks as it works fine on even older version of .net on windows.