Enumerating a directory, FindFirst()/FindNext()?

Is there a way to enumerate the files in a directory?  The only method I see
to get the files in a directory is Directory.GetFiles().  I don't want to get
a list of all files in the directory but instead enumerate the files in a
directory.  The reason is that there are hundreds of thousands of files in
the directory I'm processing and the Directory.GetFiles() method is taking
quite a bit of time to build the list.  Instead I would rather have the
unmanaged functionality of FindFirst()/FindNext().

Do I need to go through interop to get this functionality?
--
Thanks,
Nick

nicknospamdu@community.nospam
remove "nospam" change community. to msn.com

Enumerating a directory, FindFirst()/FindNext()?

On Tue, 31 Mar 2009 12:07:01 -0700, nickdu <nicknospamdu@community.nospam>


Pretty much, yes.  You could "divide and conquer" the Directory.GetFiles()
approach by careful crafting of search patterns to use, so that each call
to GetFiles() didn't retrieve so many files at once.  But it's probably
easier to just use the unmanaged API, if that's really the behavior you
want.

Pete

Hi Nick,Actually the Directory.

Hi Nick,

Actually the Directory.GetFiles() method calls the Win32 FindFirstFile &
FindNextFile functions to generate the result string array.

However, this is not always what we want - I don't want the thread being
blocked for 10 seconds to get a huge string array while all I want to do is
process the files one by one. I totally understand the pain so I have made
a DirectoryEnumerator class to solve the problem.

The basic idea is to implement the IEnumerable<string> interface in the
DirectoryEnumerator class, which provides an IEnumerator<string> to enable
using foreach loop to get the filenames one at a time. Something looks like
this:

foreach (string file in new DirectoryEnumerator(@"C:\Windows\*.log",
Mode.File))
{
// process the file
}

The enumerator will find the next file only when the MoveNext mothod of the
IEnumerator interface is called. The shortage of this implementation is you
have forward only access, no going back, no access by index. But if you
want random or by-index access, you can just go back to the GetFiles method.

Here is my proof of concept implementation of the DirectoryEnumerator
class, you can make improvements based on it to meet your requirements.

using System;
using System.IO;
using System.Text;
using System.Collections;
using System.Collections.Generic;
using System.Security.Permissions;
using System.Runtime.InteropServices;
using System.Runtime.ConstrainedExecution;
using Microsoft.Win32.SafeHandles;
using System.ComponentModel;

public class DirectoryEnumerator : IEnumerable<string>
{
#region The Enumerator

public struct Enumerator : IEnumerator<string>
{
#region Private members

private SafeFindHandle hFindFile;
private string current;
private string pattern;
private Mode mode;

#endregion

#region .ctor

internal Enumerator(string pattern, Mode mode)
{
this.pattern = pattern;
this.current = null;
this.hFindFile = null;
this.mode = mode;
}

#endregion

#region IEnumerator<string> Members

public string Current
{
get { return current; }
}

#endregion

#region IDisposable Members

public void Dispose()
{
if (null != hFindFile)
{
hFindFile.Close();
}
}

#endregion

#region IEnumerator Members

object IEnumerator.Current
{
get { return this.Current; }
}

public bool MoveNext()
{
if (null == hFindFile)
{
return FindFirst();
}
else
{
return FindNext();
}
}

public void Reset()
{
if (null != hFindFile)
{
hFindFile.Close();
hFindFile = null;
}
}

#endregion

#region Find Methods

private bool FindFirst()
{
Win32Native.WIN32_FIND_DATA fd = new
Win32Native.WIN32_FIND_DATA();

hFindFile = Win32Native.FindFirstFile(pattern, fd);

if (hFindFile.IsInvalid)
{
int code = Marshal.GetLastWin32Error();

if (code != Win32Native.ERROR_FILE_NOT_FOUND)
{
throw new Win32Exception(code);
}
else
{
return false;
}
}

if (!AttributesMatchMode(fd.dwFileAttributes))
{
return FindNext();
}

current = fd.cFileName;
return true;
}

private bool FindNext()
{
Win32Native.WIN32_FIND_DATA fd = new
Win32Native.WIN32_FIND_DATA();

while (Win32Native.FindNextFile(hFindFile, fd))
{
if (!AttributesMatchMode(fd.dwFileAttributes))
{
continue;
}

current = fd.cFileName;
return true;
}

int code = Marshal.GetLastWin32Error();

if (code != Win32Native.ERROR_NO_MORE_FILES)
{
throw new Win32Exception(code);
}
else
{
return false;
}
}

private bool AttributesMatchMode(int fileAttributes)
{
bool isDir = (fileAttributes &
Win32Native.FILE_ATTRIBUTE_DIRECTORY) ==
Win32Native.FILE_ATTRIBUTE_DIRECTORY;

return ((isDir && (mode & Mode.Directory) == Mode.Directory) ||
(!isDir && (mode & Mode.File) == Mode.File));
}

#endregion
}

#endregion

#region FileEnumeratorMode

[Flags]
public enum Mode
{
Directory = 1,
File = 2
}

#endregion

#region Private members

private string pattern;
private Mode mode;

#endregion

#region .ctor

public DirectoryEnumerator(string pattern) : this(pattern,
Mode.Directory | Mode.File)
{
}

public DirectoryEnumerator(string pattern, Mode mode)
{
this.pattern = pattern;
this.mode = mode;
}

#endregion

#region IEnumerable<string> Members

IEnumerator<string> IEnumerable<string>.GetEnumerator()
{
return new Enumerator(pattern, mode);
}

#endregion

#region IEnumerable Members

IEnumerator IEnumerable.GetEnumerator()
{
return ((IEnumerable<string>)this).GetEnumerator();
}

#endregion
}

internal sealed class SafeFindHandle : SafeHandleZeroOrMinusOneIsInvalid
{
[SecurityPermission(SecurityAction.LinkDemand, UnmanagedCode = true)]
internal SafeFindHandle()
: base(true)
{
}

protected override bool ReleaseHandle()
{
// Close the search handle.
return Win32Native.FindClose(base.handle);
}
}

internal static class Win32Native
{
[Serializable, StructLayout(LayoutKind.Sequential, CharSet =
CharSet.Auto), BestFitMapping(false)]
internal class WIN32_FIND_DATA
{
internal int dwFileAttributes;
internal int ftCreationTime_dwLowDateTime;
internal int ftCreationTime_dwHighDateTime;
internal int ftLastAccessTime_dwLowDateTime;
internal int ftLastAccessTime_dwHighDateTime;
internal int ftLastWriteTime_dwLowDateTime;
internal int ftLastWriteTime_dwHighDateTime;
internal int nFileSizeHigh;
internal int nFileSizeLow;
internal int dwReserved0;
internal int dwReserved1;
[MarshalAs(UnmanagedType.ByValTStr, SizeConst = 260)]
internal string cFileName;
[MarshalAs(UnmanagedType.ByValTStr, SizeConst = 14)]
internal string cAlternateFileName;
}

[DllImport("kernel32.dll", CharSet = CharSet.Auto, SetLastError = true)]
internal static extern SafeFindHandle FindFirstFile(string fileName,
[In, Out] WIN32_FIND_DATA data);

[DllImport("kernel32.dll", CharSet = CharSet.Auto, SetLastError = true)]
internal static extern bool FindNextFile(SafeFindHandle hndFindFile,
[In, Out, MarshalAs(UnmanagedType.LPStruct)] WIN32_FIND_DATA
lpFindFileData);

[ReliabilityContract(Consistency.WillNotCorruptState, Cer.Success),
DllImport("kernel32.dll")]
internal static extern bool FindClose(IntPtr handle);

internal const int ERROR_NO_MORE_FILES = 18;
internal const int ERROR_FILE_NOT_FOUND = 2;
internal const int FILE_ATTRIBUTE_DIRECTORY = 0x00000010;
}

If you have any further questions regarding this issue, please feel free to
post here.

Regards,

Jie Wang (jiewan@online.microsoft.com, remove 'online.')

Microsoft Online Community Support

Delighting our customers is our #1 priority. We welcome your comments and
suggestions about how we can improve the support we provide to you. Please
feel free to let my manager know what you think of the level of service
provided. You can send feedback directly to my manager at:
msdnmg@microsoft.com.

==================================================
Get notification to my posts through email? Please refer to
http://msdn.microsoft.com/en-us/subscriptions/aa948868.aspx#notifications.

Note: MSDN Managed Newsgroup support offering is for non-urgent issues
where an initial response from the community or a Microsoft Support
Engineer within 2 business days is acceptable. Please note that each follow
up response may take approximately 2 business days as the support
professional working with you may need further investigation to reach the
most efficient resolution. The offering is not appropriate for situations
that require urgent, real-time or phone-based interactions. Issues of this
nature are best handled working with a dedicated Microsoft Support Engineer
by contacting Microsoft Customer Support Services (CSS) at
http://msdn.microsoft.com/en-us/subscriptions/aa948874.aspx
==================================================
This posting is provided "AS IS" with no warranties, and confers no rights.

This is exactly what I'm looking for. Thanks. I'll give your class a try.

This is exactly what I am looking for.  Thanks.  I will give your class a try.
--
Thanks,
Nick

nicknospamdu@community.nospam
remove "nospam" change community. to msn.com

Enumerating a directory, FindFirst()/FindNext()?

d I =A0
t =A0
a
es =A0
he
s() =A0
=A0
=A0
=A0

On a side note, this looks like something worthy of a feature request
on MS Connect. Who knows, it might get into .NET 5.0 that way :)

Jie,I also have been considering implementing this type of functionality.

Jie,

I also have been considering implementing this type of functionality.

Thanks,
Mike Ober.

Hi Nick,Any comments or questions on the sample code?

Hi Nick,

Any comments or questions on the sample code?

Regards,

Jie Wang (jiewan@online.microsoft.com, remove 'online.')

Microsoft Online Community Support

Delighting our customers is our #1 priority. We welcome your comments and
suggestions about how we can improve the support we provide to you. Please
feel free to let my manager know what you think of the level of service
provided. You can send feedback directly to my manager at:
msdnmg@microsoft.com.

==================================================
Get notification to my posts through email? Please refer to
http://msdn.microsoft.com/en-us/subscriptions/aa948868.aspx#notifications.

Note: MSDN Managed Newsgroup support offering is for non-urgent issues
where an initial response from the community or a Microsoft Support
Engineer within 2 business days is acceptable. Please note that each follow
up response may take approximately 2 business days as the support
professional working with you may need further investigation to reach the
most efficient resolution. The offering is not appropriate for situations
that require urgent, real-time or phone-based interactions. Issues of this
nature are best handled working with a dedicated Microsoft Support Engineer
by contacting Microsoft Customer Support Services (CSS) at
http://msdn.microsoft.com/en-us/subscriptions/aa948874.aspx
==================================================
This posting is provided "AS IS" with no warranties, and confers no rights.

Hi Mike,Yes this is essential to processing a large number of files within a

Hi Mike,

Yes this is essential to processing a large number of files within a
directory.

Hope the code sample helps.

Thanks,

Jie Wang (jiewan@online.microsoft.com, remove 'online.')

Microsoft Online Community Support

Delighting our customers is our #1 priority. We welcome your comments and
suggestions about how we can improve the support we provide to you. Please
feel free to let my manager know what you think of the level of service
provided. You can send feedback directly to my manager at:
msdnmg@microsoft.com.

==================================================
Get notification to my posts through email? Please refer to
http://msdn.microsoft.com/en-us/subscriptions/aa948868.aspx#notifications.

Note: MSDN Managed Newsgroup support offering is for non-urgent issues
where an initial response from the community or a Microsoft Support
Engineer within 2 business days is acceptable. Please note that each follow
up response may take approximately 2 business days as the support
professional working with you may need further investigation to reach the
most efficient resolution. The offering is not appropriate for situations
that require urgent, real-time or phone-based interactions. Issues of this
nature are best handled working with a dedicated Microsoft Support Engineer
by contacting Microsoft Customer Support Services (CSS) at
http://msdn.microsoft.com/en-us/subscriptions/aa948874.aspx
==================================================
This posting is provided "AS IS" with no warranties, and confers no rights.

Reading Folders

How would i go with using your class if i just wanted a count of all Folders on a drive ?



Is the pattern for a Directory *.dir ?