Is it possible to use windows file system as a database system? |
ZhiguoYoun posted on Wednesday, May 30, 2007 1:08 PM
|
Is it possible to use windows file system as a database system?
I am working on an image crawler now.
One the most challenging problems I am facing is to judge whether a
certain URL has already been crawled.
I cannot keep all the visited URL strings in memory, and currently I
am using MySQL to manage them.
But as the number of URL increases, access to the db become slow.
Very likely I am not getting the most out of MySQL.
Does anyone know how to tune it?
An alternative method I have thought of is to use the file system as a
database system.
After crawling URL _current_url, I create a file _current_url.txt and
write some information in it.
So when _new_url comes, I only need to check the existence of file
_new_url.txt to decide whether it is already visited.
Is this method sound enough?
How does the system decide whether a file exist?
Any help will be appreciated.
Thanks in advance. |
 |
|