fs.readdirSync 有多快?我可以加快速度吗?
<p>我有一个函数,可以使用 fs.readdirSync 递归地获取目录中的所有文件。
它与我作为测试运行的小目录配合得很好,但现在我在一个超过 100GB 的目录上运行它,需要很长时间才能完成。关于如何加快速度或者是否有更好的方法,有什么想法吗?我最终将不得不在一些包含 TB 数据的目录上运行它。</p>
<pre class="brush:php;toolbar:false;">// Recursive function to get files
function getFiles(dir, files = []) {
// Get an array of all files and directories in the passed directory using fs.readdirSync
const fileList = fs.readdirSync(dir);
// Create the full path of the file/directory by concatenating the passed directory and file/directory name
for (const file of fileList) {
const name = `${dir}/${file}`;
// Check if the current file/directory is a directory using fs.statSync
if (fs.statSync(name).isDirectory()) {
// If it is a directory, recursively call the getFiles function with the directory path and the files array
getFiles(name, files);
} else {
// If it is a file, push the full path to the files array
files.push(name);
}
}
return files;
}</pre></p>
不幸的是,
异步速度较慢。所以我们需要优化你的代码。您可以使用{withFileTypes:true}选项来完成此操作,速度提高 2 倍。我还尝试过节点 v20 的
{recursive:true}选项,但它甚至比您的解决方案还要慢。它不适用于withFileTypes。也许具有高读取速度的更好 SSD 会有所帮助。虽然我猜文件条目是从文件系统索引读取的,但不确定硬件如何影响它。
import fs from 'fs'; const DIR = '/bytex'; function getFiles(dir, files = []) { // Get an array of all files and directories in the passed directory using fs.readdirSync const fileList = fs.readdirSync(dir); // Create the full path of the file/directory by concatenating the passed directory and file/directory name for (const file of fileList) { const name = `${dir}/${file}`; // Check if the current file/directory is a directory using fs.statSync if (fs.statSync(name).isDirectory()) { // If it is a directory, recursively call the getFiles function with the directory path and the files array getFiles(name, files); } else { // If it is a file, push the full path to the files array files.push(name); } } return files; } function getFiles2(dir, files = []) { const fileList = fs.readdirSync(dir, { withFileTypes: true }); fileList.forEach(file => file.isDirectory() ? getFiles2(`${dir}/${file.name}`, files) : files.push(`${dir}/${file.name}`)); return files; } let start = performance.now(); let files = getFiles(DIR); console.log(performance.now() - start); console.log(files.length); start = performance.now(); files = getFiles2(DIR); console.log(performance.now() - start); console.log(files.length);输出: