Problem reading data from TidIOHandler

I have the following code

var
  Buf: TidBytes;
  fMem: TMemoryStream;
begin
  fMem:= TMemoryStream.Create;
  try
        IOHandler.ReadBytes(Buf, 1, False);
        Size:= Buf[0];
        IOHandler.ReadBytes(Buf, Size, False);

        fMem.Clear;
        BytesToRaw(Buf, fMem, Size);
        fMem.Seek(0,soFromBeginning); <==== I get a EAccessViolation.
  finally
    fMem.Free;
  end;
end;

I seem to always end up with problems using streams - so sorry if this is a
stupid question but it has me really stumped.

Thanks in advance