As a newbie of file system driver development, most of the suggestion I have heard is "go to read the sample in DDK" or "search ntfsd".

When I read the sample in DDK, I always have to think "why I need to call A first before B", "why I can't call C". I have no concept, so it is useless to read the code. I also read book but all I get is lots of concept only. Some are not suitable and needed for beginner, but as a beginner, I don't know what concept is needed. So I read all of them (and I still get question mark in my head on how to transform the concept to work).

Many common and simple questions like "encryption of a text file" involve lots of skill which newbie feels difficult but easy for you guys. (e.g. get file name, allocate and swap buffer, context) What I suggest is taking these common topics as tutorial explaining with both code and concept in between. A series of blog post is enough already, there is no need to make it too formal. After reading these tutorials, at least the readers get something to try and work on. And they can say "I can do XXX now" instead of feeling helpless.

I hope you guy can take it as a consideration. It must be good to spread the knowledge of driver development. I am eager to read one :)

(I have google a lot and I am sure that that is lack of such style of tutorial for driver development)

Franz Wong

There is - OSR classes.

Not free, but neither is developing a tutorial.


mm

I know there are classes held by OSR. Even though I can afford the course fee, most of them held in North America.

   MSDN WDK docs + samples + some concept books like Solomon/Russinovich + some developer's books like Rajeev Nagar, Walter Oney and the K/UMDF book.

I am learning to write an encryption minifilter. I have the passion to search for knowledge, but I don't know whether other people will be scared if they are acknowledged to read from dozen of source. After I have succeed in something simple, then I can read more to handle more difficult task.

I have searched and found a thread which may be suitable for me. (http://www.osronline.com/showThread.cfm?link=119140) I appreciate this kind of work. It is a good try. For example, he stated the reason why only decrypt for paging io and provide the code. Instead of a single line comment "only decrypt in paging io". However, it seems that he was also a beginner. The solution proposed is not full and it is buggy. The repliers can give me the concept but not the implementation. I still cannot solve my problem. Then I think "why those repliers or other experienced subscriber cannot provide a full solution with explanation". That makes me starting this thread.

Both concept and code are very important on learning programming.

You also chose one of the most difficult drivers to write for Windows.  Even
Microsoft found it to be necessary or maybe desirable to have a special hook
in the file system driver to support compression and encryption.  When doing
it in the file system driver, the compression and encryption code can be
called when it is appropriate for it to be done and in a context where it
will be easy for them to do it.  When you are doing it in a file system
filter where you have to contend with other filters, the file system driver,
cache manager, and the memory manager it can become much more of a
nightmare.

Why would someone who really knows this stuff and has invested thousands of
hours in learning, developing, and testing designs and code for this type of
task want to give it away?  It would mean they are taking food from their
family's mouths instead of using all that effort to provide income for their
needs.  As Tony of OSR has indicated, there are some edge cases where only a
file system based solution can work.

Even with the minifilter model, there are still major issues to be solved
especially when you get beyond FAT since the source code to the other file
systems is not available.  Even with source code access, you would still
have to contend with patches, updates, and possible redesign or enhancements
of the underlying file systems.  Also with source code access, you still
have a lot of work to do to understand all the edge cases and how the file
system driver handles them.

Your arguments sound as if you think Windows should be like Linux or
FreeBSD.  You work for hours, days, weeks, months, and years with no
compensation other than the feel-good kind.  It doesn't help those who need
money to live.  For the rich who don't need the money, it could provide
something they feel is needed in their life but that is not most of us.

You have turned the discussion to "whether open source is communism". That is another topic and I don't think it is suitable to be discussed here. I would only say, adjusting the learning curve can create a larger community which also benefit experienced developer. Non free software in Windows can be developed by programmer who learns from free source. Author can also be paid by Microsoft or OSR in order to let more people writing Windows driver or attending the classes for more knowledge. Sharing knowledge to others is also a way to consolidate the knowledge we know.

From your words, I agree that an encryption minifilter can be very sophisticated. But there must be some easier cases existing. Just like the sample in DDK. I would not think the code in the sample is suitable for all cases. But for a beginner, at least, I have a cutting point into the problem.

I think you were the first to mention communism.  I said most of us work to
provide for ourselves and our families.  Those who have the time and
motivation can write code and distribute it as they wish unless they are
bound by employee agreements or other legally binding restrictions.  I do
not agree with your premise about 'adjusting the learning curve'.  There are
just so many jobs for file systems and file system filter driver developers.
Most of us who have worked in this arena invested a lot of time and money
obtaining the knowledge.  Only Microsoft has the motivation to provide the
samples to the community so that there will be fewer crashes.  Have you
noticed that other than the swapbuffers sample, there are no compression or
encryption samples in the WDK?  Even that sample doesn't really do anything
other than provide an example of how to treat the buffers containing the
data.  Adding headers or trailers and handling the many requests that can
determine the size of the file are ignored.

There is only one book that has been written on file systems.  I remember
seeing it available used for $300 USD.  The IFS Kit used to cost $1000.  The
last IFS Kit was reduced to about $110, IIRC.  Mark Russinovich used to
release the sources to his filespy utility that required a license for it to
be used in a commercial product.  Many encryption filters were developed
from that code without paying for a license.  The source was removed many
years ago and when Sysinternals/Wininternal was bought by Microsoft, all the
remaining source code was eliminated from the website.  Now whether
Microsoft is an icon of the developer industry can be debated, but most of
us obtain our living from working on code that is run on Windows.  Some
companies do pay developers to write drivers for open source projects
because their customers require it.  Some, mine included, write for rather
small environments but as the laws in the USA require, those expenditures
are balanced against profit.  Either the customers require support in a
particular OS or some contract specified such a deliverable.

@Franz Wong

Please keep my comments out of this debate and kindly try to understand the point,
I did one similar driver few years back, was a new guy to file system, I got all types of comments (Many supporting, Many sarcastic, some bad and so on), read the US edition of FS book as Indian economic edition was not available, read many others books and articles, searched this forum, spend many nights (3 months, daily abt 15+ hours with no holidays) with my great teammates and with the great FILESPY. I remember analyzing save operation of a .doc and .xls file in filespy and than comparing each and every IRP parameters with similar operation done with our filter driver and than correction for every single mistake (And hundreds of such probs relating to size and data corruption etc). with that I remember the pressure to release it with vista i.e. sept 2006. Thanks MS to delay Vista till Jan 07, you saved 10 jobs ;-).

So after all this effort I learned to write a on the fly encryption filter driver which works for all file formats and provides some XYZ features.The code was under agreement but hypothetically even if it was not under agreement should I post all that code on net for a tutorial? If I do than here in my country some company instead of hiring me, will hire a 2 years guy with no knowledge on FS and ask him to copy and modify the code to suits their need. So I support David with this aspect of his post.

Few years back the situation was almost same, with no tutorial as such but *enough* information available on the subject though not at single place; one just need to have patience and add the bits and pieces logically. we completed our project we did not debated with anyone as that was not our motive. Our motive was to create the product and we did it.
Agree but there are cases when you have to move ahead without any of these (Assembly helps ;-)), I did few such tasks(like to delete a loaded DLL, runnig exe from disk plus few others), leave source code aside I can not even ask for concepts as the requirement is unusual but than sometimes you have to do that. So I worked on each and every suggestions proposed on forum and at last it worked. Thanks WinDbg (Did I mentioned thanks to Mr. Scott Noone ;-))

So yes what you said abt importance of code on concepts are true but more important is survival and for that you need to have some USP and you can't reveal that.

best of luck and

Thanks for patience,
Aditya

To David,

We always work for ourselves and family. I am tired to hear something like teaching you simple sentence freely, this will lead to destruction of thousand English teaching jobs.

I found a document fully discussing the sfilter sample. (http://www.ccgcn.com/bbs/attachment.php?aid=305). The author states the reason why he wrote such a document. The reason is the same as I said. Although it is written in Chinese, you can get a skim on how other people are doing. Another software TrueCrypt also opens its source to the public. I guess this Chinese author and developers from TrueCrypt are wealthy enough and gain the motivation to do technical sharing. Think it by heart, there are always reasons and excuses to convince ourselves not to do something.

I have seen the so-called "Professionalism" here. And I understand why Microsoft needs to give MVP Award to encourage developers. It is because people are always afraid of losing job, especially in this economic downturn. When I start this topic, I didn't think I would get this kind of feedback in a technology SHARING forum. My replies turn to be more aggressive. I think I stop discussing on this topic. Sorry for disturbance.

The problem is that your assumptions are wrong and your analogies very poor. There are only very little people in this world who have the knowledge and experience to write a file system encryption filter. As has been pointed out this is one of the hardest drivers to write. Before even thinking about starting this undertaking you surely must know your way around the kernel, understand a lot of concepts and get experience in the file system stack otherwise any such effort would be futile.

At best such a document would give a false impression that this is an easy thing to do which would be dangerous inside the hands of somebody without knowledge. Whatever the document will describe is something that will work only "most of the time" without handling the edge cases in which lies most of the work. This would surely damage the reputation of the author.

Go and get some experience in the file system stack. As soon as you will have enough knowledge and experience that you can write an active filter (expect several years) you will be able to understand that "file system encryption filters for dummies"  does not serve much of a purpose.

//Daniel