Changes to be able to build system

[reactos.git] / reactos / doc / notes

*** This file contains messages I've culled off the net as well
as previous discussions all of which have useful info on fixes
that need to be added to ReactOS. messages are between five
dashes on a line by themselves.  If you implement the fix
reffered to in a message, feel free to delete it from the file.
Rex ***
-----
Subject: [ros-kernel] Inside the Boot Process
Date: Mon, 22 Mar 1999 22:05:47 +0100
From: Emanuele Aliberti <ea@iol.it>

For those working on the boot loader: in WinNt Magazine november 1998
issue (http://www.winntmag.com/) there is a detailed description, by
Mark Russinovich, of the rôle the MBR, NTLDR, boot.ini, ntdetect.com...
play in the boot process ("Inside the Boot Process, Part 1").
-----
Yes with DPCs, KeDrainDpcQueue should go to HIGH_LEVEL because 
it needs to synchronize with KeInsertDpcQueue. Also the idle thread 
should run at DISPATCH_LEVEL and regularly drain the dpc queue, that 
way if an irq happens and the dpc can't be executed immediately it 
will be executed as soon as the processor is idle rather than 
waiting for the next timer tick
-----
About the console driver, I think it might be quite useful to have a simple 
way for apps to print to the screen for debugging. But when the kernel is more 
stable, console handling should be moved to user level because console printing 
needs to know about windows and so on which can only be done at user level.
-----
Subject: Re: IMSAMP-how to avoid rebooting?
Date: 9 Nov 1998 00:40:32 -0000
From: Charles Bryant <n51190709.ch@chch.demon.co.uk>
Newsgroups: comp.os.ms-windows.programmer.nt.kernel-mode
References: 1, 2 , 3 , 4

In article <un264wzle.fsf@xxx.yyy.zzz>, David C. <qqqq@xxx.yyy.zzz> wrote:
>The reason it won't unload when something is bound to it is the same
>reason you can't unload any other driver that has an open client.  If
>you install any driver, and have a user program (or another driver) open
>a handle to it, and then give the "net stop" command to unload it,
>you'll find that the unload will be delayed until the user program
>closes its handle.

When developing a driver I found this to be a considerable nuisance.
Frequently a bug would leave an IRP stuck in the driver and I
couldn't unload and reload a fixed version. While reading NTDDK.H I
found a suspicious constant and discovered that the Flags field in
the device (the one which you OR in DO_BUFFERED_IO or DO_DIRECT_IO)
has a bit called DO_UNLOAD_PENDING. By experiment I confirmed that
this bit is set when you do 'net stop', so a driver can check it
periodically (e.g. from a timer DPC every ten seconds) and cancel all
queued IRPs if it is found to be set.

Since this is not documented anywhere that I can find, it might be
unwise to rely on it for production code, but it is very useful for
debugging. Maybe someone with internals knowledge can comment on the
reliability of it.
-----
Subject: Re: Kernel bugs
Date: Fri, 23 Oct 1998 12:08:36 -0700
From: rex <rex@lvcablemodem.com>
To: Jason Filby <jasonfilby@yahoo.com>
References: 1

Jason Filby wrote:

> Hi,
>
> Ok -- here's most of what I get when I press a key:
>
> Page fault detected at address 1fd4 with eip c042f794
> Recursive page fault detected
> Exception 14(2)
> CS:EIP 20:c042f794
>
> Rex -- do you know of anyway to find out which function in what file
> is causing the exception? I know that for problems in the kernel, you
> just look in the ntoskrnl\kernel.sym file and find the EIP value which
> matches the one given in the exception debug text. But what about
> modules? How can we track exceptions that occur in functions in modules?
>

I know this is a little belated, but I thought I'd take astab at answering
this anyway.  add an option to the
makefile for the module to generate a listing file with
symbol information.  Then, on a boot test, note the
address that the module is loaded at, and subtract
this from the EIP value.  add any offset used in the
module link specification (I dont think there currently
is one), and look for the last symbol with a lower
address offset.

Brian, I have an idea on how to make this exception
dump information a little more useful.  We should
have the load information for the load modules
in memory somewhere.  Perhaps the exception
dump could check offending addresses to see if
they lie in the kernel or in a module, and if they
lie in a module the proper offset could be subtracted
and this number could be displayed seperately.  If
I get a chance today, I'll make this change and send
it to ya.

Rex.
-----
Subject: [ros-kernel] Pet peeve of the week
Resent-Date: Sun, 25 Oct 1998 11:57:40 -0600
Resent-From: ros-kernel@sid-dis.com
Date: Sun, 25 Oct 1998 09:53:48 -0800
From: rex <rex@lvcablemodem.com>
Reply-To: <ros-kernel@sid-dis.com>
To: ReactOS Kernel Forum <ros-kernel@sid-dis.com>

Hi all,

I guess it's about time to start another mailstorm
on the list. :)

I have a suggestion for a change to the kernel.
It not a very big change, and I hope everyone
will agree that it makes sense.

There is a structure used in many places in the
kernel called LARGE_INTEGER.  the is also
a version called ULARGE_INTEGER, but it
is not used at all as far as I can tell.  this structure
is equivalent to a long long int.  You can literally
cast a pointer to a LARGE_INTEGER to a
long long int and all manipulation will work
seemlessly. My suggestion is that we replace the
use of this structure with long long ints.  Even
microsoft, in their infinite wisdom, has made this
suggestion in the DDK documentation.  If you're
wondering where, look at the RTL functions
that manipulate LARGE_INTEGER structs.

Replacing LI's with long long ints will work
because they are binary compatable.  All software
compiled to use LI's will manipulate long long ints
correctly and vice versa. There is one problem
with this suggestion: the LARGE_INTEGER type
is a structure containing 2 members.  Any code
that accesses the structure by members will break.
I think the kernel side impact is minimal, and is
worth the change.  However, the structure is used
in several of the Win32 API functions, and needs
to remain there.  I think we build a conditionally
compiled version of the LARGE_INTEGER type.
In kernel mode code (the kernel proper and drivers)
the LARGE INTEGER will be the following:

typedef long long int LARGE_INTEGER,
    *PLARGE_INTEGER;
typedef unsigned long long int ULARGE_INTEGER,
    *PULARGE_INTEGER;

and in user mode code it will expand out to the
current definition (which by the way, is not
strictly correct, but can't be because it uses a
MS compiler extension).

Brian, I would be willing to make the conversion
to those kernel modules that needed it, and of
course to the IDE driver if we want to go forward
with the change.

Lastly, I'll mention what made me consider this.
I was fixing the timer routines, and two of the
three problems turned out to be related to LI
conversion problems.

Rex.
-----
Subject:  Re: [ros-kernel] Pet peeve of the week
Date: Thu, 29 Oct 1998 19:10:37 +0100
From: Boudewijn <ariadne@xs4all.nl>
To: rex@lvcablemodem.com
References: 1

Hai Rex

    I think it is a good idea to wrap a makro around the member access
to large integers.
I haven't tested this, but do you think this is a good sugestion ?

#ifdef COMPILER_LARGE_INTEGERS
#define GET_LARGE_INTEGER_HIGH_PART(LargeInteger) ( ( LargeInteger >>
32) )
#define GET_LARGE_INTEGER_LOW_PART(LargeInteger) ( (LargeInteger &
0xFFFFFFFF) )
#define SET_LARGE_INTEGER_HIGH_PART(LargeInteger,Signed_Long) (
LargeInteger |= ( ((LARGE_INTEGER)Signed_Long) << 32 ) )
#define SET_LARGE_INTEGER_LOW_PART(LargeInteger,Unsigned_Long) (
LargeInteger |= Unsigned_Long  )
#else
#define GET_LARGE_INTEGER_HIGH_PART(LargeInteger) (  (
LargeInteger.HighPart) )
#define GET_LARGE_INTEGER_LOW_PART(LargeInteger) (
(LargeInteger.LowPart) )
#define SET_LARGE_INTEGER_HIGH_PART(LargeInteger,Signed_Long) (
LargeInteger.HighPart=  Signed_Long  )
#define SET_LARGE_INTEGER_LOW_PART(LargeInteger,Unsigned_Long) (
LargeInteger.LowPart = Unsigned_Long  )
#endif

Boudewijn
-----
Subject: Re: Question on "Sending buffers on the stack to asynchronous DeviceIoControl with buffered I/O"
Date: Mon, 16 Nov 1998 11:24:57 -0800
From: "-Paul" <paulsan@microsoftSPAM.com>
Organization: Microsoft Corp.
Newsgroups: microsoft.public.win32.programmer.kernel, comp.os.ms-windows.programmer.nt.kernel-mode
References: 1

Radu, I post the following information occassionally for questions such as
yours.  I hope it helps.

-Paul

Here is an explanation of buffers and DeviceIoControl.

First, here are the parameters,

BOOL DeviceIoControl(
    HANDLE hDevice, // handle to device of interest
    DWORD dwIoControlCode, // control code of operation to perform
    LPVOID lpInBuffer, // pointer to buffer to supply input data
    DWORD nInBufferSize, // size of input buffer
    LPVOID lpOutBuffer, // pointer to buffer to receive output data
    DWORD nOutBufferSize, // size of output buffer
    LPDWORD lpBytesReturned, // pointer to variable to receive output byte
count
    LPOVERLAPPED lpOverlapped  // pointer to overlapped structure for
asynchronous operation
   );

METHOD_BUFFERED

user-mode perspective

lpInBuffer - optional, contains data that is written to the driver
lpOutBuffer - optional, contains data that is read from the driver after
the call has completed

lpInBuffer and lpOutBuffer can be two buffers or a single shared buffer.
If a shared buffer, lpInBuffer is overwritten by lpOutBuffer.


I/O Manager perspective

examines nInBufferSize and nOutBufferSize.  Allocates memory from non-paged
pool and puts the address of this pool in Irp->AssociatedIrp.SystemBuffer.
The size of this buffer is equal to the size of the larger of the two
bufferes.  This buffer is accessible at any IRQL.

copies nInBufferSize to irpSp->Parameters.DeviceIoControl.InputBufferLength
copies nOutBufferSize to
irpSp->Parameters.DeviceIoControl.OutputBufferLength
copies contents of lpInBuffer to SystemBuffer allocated above
calls your driver



Device Driver perspective

you have one buffer, Irp->AssociatedIrp.SystemBuffer.  You read input data
from this buffer and you write output data to the same buffer, overwriting
the input data.

Before calling IoCompleteRequest, you must
- set IoStatus.Status to an approriate NtStatus
- if IoStatus.Status == STATUS_SUCCESS
 set IoStatus.Information to the
 number of bytes you want copied
 from the SystemBuffer back into
 lpOutBuffer.


I/O Manager Completion Routine perspective

looks at IoStatus block, if IoStatus.Status = STATUS_SUCCESS, copies the
number of bytes specified by IoStatus.Information from
Irp->AssociatedIrp.SystemBuffer into lpOutBuffer
completes the request






METHOD_IN_DIRECT

user-mode perspective

lpInBuffer - optional, contains data that is written to the driver.  This
buffer is used in the exact same fashion as METHOD_BUFFERED.  To avoid
confusion, mentally rename this buffer to lpControlBuffer.  This is
typically a small, optional buffer that might contain a control structure
with useful information for the device driver.  This buffer is smal and is
double buffered.

lpOutBuffer - NOT OPTIONAL, This LARGE buffer contains data that is read by
the driver.  To avoid confusion, mentally rename this buffer to
lpDataTransferBuffer.  This is physically the same buffer that the device
driver will read from.  There is no double buffering.  Technically, this
buffer is still optional, but since you are using this buffering method,
what would be the point???

I/O Manager perspective

If lpInBuffer exists, allocates memory from non-paged pool and puts the
address of this pool in Irp->AssociatedIrp.SystemBuffer.  This buffer is
accessible at any IRQL.

copies nInBufferSize to irpSp->Parameters.DeviceIoControl.InputBufferLength
copies nOutBufferSize to
irpSp->Parameters.DeviceIoControl.OutputBufferLength
copies contents of lpInBuffer to SystemBuffer allocated above
So far this is completely identical to METHOD_BUFFERED.  Most likely
lpInBuffer (mentally renamed to lpControlBuffer) is very small in size.

For lpOutBuffer (mentally renamed to lpDataTransferBuffer), an MDL is
allocated.  lpOutBuffer is probed and locked into memory.  Then, the user
buffer virtual addresses are checked to be sure they are readable in the
caller's access mode.

The MDL is address is stored in Irp->MdlAddress.
Your driver is called.


Device Driver perspective

The device driver can read the copy of lpOutBuffer via
Irp->AssociatedIrp.SystemBuffer.  Anything written by the device driver to
this buffer is lost.  The I/O Manager does not copy any data back to the
user-mode buffers as it did in the completion routine for METHOD_BUFFERED.
Art Baker's book is wrong in this respect (page 168, "data going from the
driver back to the caller is passed through an intermediate system-space
buffer" and page 177, "When the IOCTL IRP is completed, the contents of the
system buffer will be copied back into the callers original output buffer".

The device driver accesses the Win32 buffer directly via Irp->MdlAddress.
The driver uses whatever Mdl API's to read the buffer.  Usually, this
buffer is to be written to some mass storage media or some similar
operation.  Since this is a large data transfer, assume a completion
routine is required.

mark the Irp pending
queue it
return status pending




Device Driver Completion Routine perspective

standard completion routine operations
set IoStatus.Status to an approriate NtStatus
IoStatus.Information is not needed
completete the request




I/O Manager Completion Routine perspective

standard I/O Manager completion routine operations
unmap the pages
deallocate the Mdl
complete the request





METHOD_OUT_DIRECT

user-mode perspective

lpInBuffer - optional, contains data that is written to the driver.  This
buffer is used in the exact same fashion as METHOD_BUFFERED.  To avoid
confusion, mentally rename this buffer to lpControlBuffer.  This is
typically a small, optional buffer that might contain a control structure
with useful information for the device driver.  This buffer is smal and is
double buffered.

lpOutBuffer - NOT OPTIONAL, This LARGE buffer contains data that is written
by the driver and read by the wer-mode application when the request is
completed.  To avoid confusion, mentally rename this buffer to
lpDataTransferBuffer.  This is physically the same buffer that the device
driver will write to.  There is no double buffering.  Technically, this
buffer is still optional, but since you are using this buffering method,
what would be the point???

I/O Manager perspective

If lpInBuffer exists, allocates memory from non-paged pool and puts the
address of this pool in Irp->AssociatedIrp.SystemBuffer.  This buffer is
accessible at any IRQL.

copies nInBufferSize to irpSp->Parameters.DeviceIoControl.InputBufferLength
copies nOutBufferSize to
irpSp->Parameters.DeviceIoControl.OutputBufferLength
copies contents of lpInBuffer to SystemBuffer allocated above
So far this is completely identical to METHOD_BUFFERED.  Most likely
lpInBuffer (mentally renamed to lpControlBuffer) is very small in size.

For lpOutBuffer (mentally renamed to lpDataTransferBuffer), an MDL is
allocated.  lpOutBuffer is probed and locked into memory.  Then the user
buffer's addresses are checked to make sure the caller could write to them
in the caller's access mode.

The MDL is address is stored in Irp->MdlAddress.
Your driver is called.


Device Driver perspective

The device driver can read the copy of lpOutBuffer via
Irp->AssociatedIrp.SystemBuffer.  Anything written by the device driver to
this buffer is lost.

The device driver accesses the Win32 buffer directly via Irp->MdlAddress.
The driver uses whatever Mdl API's to write data to the buffer.  Usually,
this buffer is to be read from some mass storage media or some similar
operation.  Since this is a large data transfer, assume a completion
routine is required.

mark the Irp pending
queue it
return status pending




Device Driver Completion Routine perspective

standard completion routine operations
set IoStatus.Status to an approriate NtStatus
IoStatus.Information is not needed
completete the request




I/O Manager Completion Routine perspective

standard I/O Manager completion routine operations
unmap the pages
deallocate the Mdl
complete the request




METHOD_NEITHER

I/O Manager perspective

Irp->UserBuffer = lpOutputBuffer;
IrpSp->Parameters.DeviceIoControl.Type3InputBuffer = lpInputBuffer;

No comments here.  Don't use METHOD_DIRECT unless you know what you are
doing.  Simple rule.

If your IOCtl involves no data transfer buffers, then METHOD_NEITHER is the
fastest path through the I/O Manager that involves an Irp.




Final Comment

Don't touch Irp->UserBuffer.  This is a bookmark for the I/O Manager.  Two
major problems can occur.  1 - page fault at high IRQL, or 2 - you write
something to Irp->UserBuffer and the I/O Manager overwrites you in its
completion routine.  File systems access Irp->UserBuffer, but FSD writers
know all of the above and know when it is safe to touch Irp->UserBuffer.



Radu Woinaroski wrote in message <364F8F6E.2434B010@scitec.com.au>...
>Hello,
>
>I have a kernel-mode device driver that accepts a number of IoControl
>commands that use buffered data transfer (METHOD_BUFFERED).
>
>A user mode API provides a higher level access then the DeviceIoControl
>function.
>
>The function is implemented like that
>
>BOOL
Something( 
> HANDLE hDevice , 
> int param1, 
> int param2, 
> DWORD * pReturn, 
> LPOVERLAPPED pOverlapped)
>{
> // here a data buffer on the stack sent to asynchronous DeviceIoControl
>call
> int aDataIn[2];
> aDataIn[0] = param1;
> aDataIn[1] = param2;
>
> return DeviceIoControl( 
> hDevice,
> DO_SOMETHING_IO, 
> aDataIn,
> sizeof(int)*2,
> pReturn,
> sizeof(DWORD),
> pOverlapped);
>}
>
>The aDataIn buffer will not exist after DeviceIoControl returns (and
>when the I/O operation terminates). I know that for buffered IO the
>input data buffer is copyed by de IOManager to a nonpaged-pool area
>before passing the request to driver dispatch routine (DeviceControl).
>At the point of calling the dispatch routine (DeviceControl) the driver
>runs in the context of the calling thread so DeviceIoControl hasn't
>returned yet (?? or so I think) so aDataI
n will still be valid at the
>time IOManager copyes it to its buffer. So, this apears to work ok (at
>least in my opinion).
>
>Does I/O Manager use the Input buffer from the call to the Win32
>DeviceIoControl any where else after the first copy ?
>
>Is there any reason why this approach (passing a buffer on the stack to
>a asynchronous DeviceIoControl that uses buffered I/O) wouldn't work ?
>
>Allocating buffers from heap and deleting them on IO completion while
>managing asynchronous IO seems too much work ;-) .
>
>Thanks in advance for your opinions
>Radu W.
>
>--
>Radu Woinaroski
>Scitec
>Sydney, Australia
>Radu.Woinaroski@scitec.com.au
-----
Subject: Re: PCI ISR problem
Date: Fri, 20 Nov 1998 18:04:48 GMT
From: jeh@cmkrnl.com (Jamie Hanrahan)
Organization: Kernel Mode Systems, San Diego, CA
Newsgroups: comp.os.ms-windows.programmer.nt.kernel-mode
References: 1

On Thu, 19 Nov 1998 15:46:13 -0600, Eric Gardiner
<eric.gardiner@natinst.com> wrote:

>I'm having problems with NT4 not hooking the interrupt line indicated by
>a PCI device.  Here's what I'm doing:
>
>1)  Enumerating the PCI buses on the system (using HalGetBusData) until
>I find my device.
>2)  Once my device is found, I read the "Interrupt Line Register" in the
>device's PCI config space to determine what interrupt level to pass to
>HalGetInterruptVector.

Whups!  No.  Call HalAssignSlotResources and look at the returned
CM_RESOURCE_LIST to find the vector, level, port addresses, etc., for
your device.  (Then pass the returned CM_RESOURCE_LIST to ExFreePool.)


See Knowledge Base article Q152044. 

        --- Jamie Hanrahan, Kernel Mode Systems, San Diego CA (jeh@cmkrnl.com)
Drivers, internals, networks, applications, and training for VMS and Windows NT
NT kernel driver FAQ, links, and other information:  http://www.cmkrnl.com/

Please post replies, followups, questions, etc., in news, not via e-mail. 
-----
Subject: Re: IRP canceling
Date: Mon, 23 Nov 1998 09:05:47 -0500
From: Walter Oney <waltoney@oneysoft.com>
Organization: Walter Oney Software
Newsgroups: comp.os.ms-windows.programmer.nt.kernel-mode
References: 1

Seol,Keun Seok wrote:
> But, if the IRP was the CurrentIrp of the Device Object,
> the Driver's Start I/O routine will try to process the IRP.
> In the DDK help, the Start I/O routine MUST check the current IRP's
> Cancel bit.
> If set, Start I/O routine must just return.
> 
> But I think that the IRP already completed should not be accessed. 

You're absolutely right. I recommend the following code in a standard
StartIo routine to avoid the problem you point out:

VOID StartIo(PDEVICE_OBJECT DeviceObject, PIRP Irp)
  {
  KIRQL oldirql;
  IoAcquireCancelSpinLock(&oldirql);
  if (Irp != DeviceObject->CurrentIrp || Irp->Cancel)
    {
    IoReleaseCancelSpinLock(oldirql);
    return;
    }
  else
    {
    IoSetCancelRoutine(Irp, NULL);
    IoReleaseCancelSpinLock(oldirql);
    }
  . . .
  }

This dovetails with a standard cancel routine:

VOID CancelRoutine(PDEVICE_OBJECT DeviceObject, PIRP Irp)
  {
  if (DeviceObject->CurrentIrp == Irp)
    {
    IoReleaseCancelSpinLock(Irp->CancelIrql);
    IoStartNextPacket(DeviceObject, TRUE);
    }
  else
    {
    KeRemoveEntryDeviceQueue(&DeviceObject->DeviceQueue,
      &Irp->Tail.Overlay.DeviceQueueEntry);
    IoReleaseCancelSpinLock(Irp->CancelIrql);
    }
  Irp->IoStatus.Status = STATUS_CANCELLED;
  Irp->IoStatus.Information = 0;
  IoCompleteRequest(Irp, IO_NO_INCREMENT);
  }

You need to remember that the C language specification requires that
evaluation of boolean operators short circuit when the result is known.
So, if StartIo discovers that the Irp it got as an argument is not the
same as CurrentIrp, it will not attempt to evaulate Irp->Cancel.

Now, as to why this works: StartIo gets called either by IoStartPacket
or IoStartNextPacket. Each of them will grab the cancel spin lock and
set CurrentIrp, then release the spin lock and call StartIo. If someone
should sneak in on another CPU and cancel this very same IRP, your
cancel routine will immediately release the spin lock and call
IoStartNextPacket. One of two things will then happen. IoStartNextPacket
may succeed in getting the cancel spin lock, whereupon it will nullify
the CurrentIrp pointer. If another IRP is on the queue, it will remove
it from the queue, set CurrentIrp to point to this *new* IRP, release
the spin lock, and call StartIo. [You now have two instances of StartIo
running on two different CPUs for two different IRPs, but it's not a
problem because they won't be able to interfere with each other.]
Meanwhile, your original instance of StartIo gets the cancel spin lock
and sees that CurrentIrp is not equal to the IRP pointer it got as an
argument, so it gives up.

The second way this could play out is that StartIo gets the cancel lock
before IoStartNextPacket does. In this case, CurrentIrp is still
pointing to the IRP that's in the process of being cancelled and that
StartIo got as an argument. But this IRP hasn't been completed yet (the
CPU that's running your cancel routine is spinning inside
IoStartNextPacket and therefore hasn't gotten to calling
IoCompleteRequest yet), so no-one will have been able to call IoFreeIrp
to make your pointer invalid.

People may tell you that you should be using your own queues for IRPs so
you can avoid bottlenecking the system on the global cancel spin lock.
That's true enough, but doing it correctly with Plug and Play and Power
management things in the way is gigantically complicated. There's a
sample in the NT 5 beta-2 DDK called CANCEL that shows how to manage
your own queue if you don't worry about PNP and POWER. I hear tell of an
upcoming MSJ article by a Microsoft developer that may solve the
complete problem.
-----
Subject: ANNOUNCE: ALINK v1.5
Date: 16 Nov 1998 16:36:05 GMT
From: anthony_w@my-dejanews.com
Organization: Deja News - The Leader in Internet Discussion
Newsgroups: comp.os.ms-windows.programmer.win32, comp.lang.asm.x86, comp.os.msdos.programmer

ALINK is a freeware linker, creating MSDOS COM and EXE files and Win32 PE EXE
and DLL files from OMF format OBJ and LIB files, win32-COFF format OBJ files,
and win32 RES files.

NEW for version 1.5:

Win32 COFF object file support.

Download it now from my home page.

Anthony
--
anthony_w@geocities.com
http://www.geocities.com/SiliconValley/Network/4311/index.html

-----------== Posted via Deja News, The Discussion Network ==----------
http://www.dejanews.com/       Search, Read, Discuss, or Start Your Own    
-----