How to speed up directory listing on high-inode partitions?












2















I have almost 7,000 files in a single directory which makes loading that particular directory a tedious and time-consuming one. Is there any way to list and perform operations on files in that directory faster rather than distributing the files into multiple sub-folders(which makes it even more difficult)?



I'm using ext4. But also received the same from efs filesystems.






inode







share|improve this question
















bumped to the homepage by Community♦ 1 hour ago


This question has answers that may be good or bad; the system has marked it active so that they can be reviewed.











  • 1





    It's better to you make filter, you have some way, find, globbing and so on , But at first you explain exactly your task.

    – PersianGulf
    Mar 7 '15 at 20:22






  • 1





    Could you clarify what you mean by "loading"? Simply ls, or displaying the contents in a file manager (which one?), or something else? On my system listing 7,000 files with ls on an ext4 filesystem is pretty much instantaneous, as is displaying the directory in Thunar.

    – Stephen Kitt
    Mar 7 '15 at 20:56











  • @SnazzySanoj please answer comments, by amending your question.

    – ctrl-alt-delor
    Mar 7 '15 at 22:30






  • 1





    A lot of systems alias ls to something that makes it lstat every file. --color and -F do this. If you don't need ls to decorate the output, remove these aliases. Listing a 7000 file directory shouldn't be slow.

    – Mark Plotnick
    Mar 7 '15 at 23:48
















2















I have almost 7,000 files in a single directory which makes loading that particular directory a tedious and time-consuming one. Is there any way to list and perform operations on files in that directory faster rather than distributing the files into multiple sub-folders(which makes it even more difficult)?



I'm using ext4. But also received the same from efs filesystems.






inode







share|improve this question
















bumped to the homepage by Community♦ 1 hour ago


This question has answers that may be good or bad; the system has marked it active so that they can be reviewed.











  • 1





    It's better to you make filter, you have some way, find, globbing and so on , But at first you explain exactly your task.

    – PersianGulf
    Mar 7 '15 at 20:22






  • 1





    Could you clarify what you mean by "loading"? Simply ls, or displaying the contents in a file manager (which one?), or something else? On my system listing 7,000 files with ls on an ext4 filesystem is pretty much instantaneous, as is displaying the directory in Thunar.

    – Stephen Kitt
    Mar 7 '15 at 20:56











  • @SnazzySanoj please answer comments, by amending your question.

    – ctrl-alt-delor
    Mar 7 '15 at 22:30






  • 1





    A lot of systems alias ls to something that makes it lstat every file. --color and -F do this. If you don't need ls to decorate the output, remove these aliases. Listing a 7000 file directory shouldn't be slow.

    – Mark Plotnick
    Mar 7 '15 at 23:48














2












2








2


1






I have almost 7,000 files in a single directory which makes loading that particular directory a tedious and time-consuming one. Is there any way to list and perform operations on files in that directory faster rather than distributing the files into multiple sub-folders(which makes it even more difficult)?



I'm using ext4. But also received the same from efs filesystems.






inode







share|improve this question
















I have almost 7,000 files in a single directory which makes loading that particular directory a tedious and time-consuming one. Is there any way to list and perform operations on files in that directory faster rather than distributing the files into multiple sub-folders(which makes it even more difficult)?



I'm using ext4. But also received the same from efs filesystems.






inode




inode






share|improve this question















share|improve this question













share|improve this question




share|improve this question








edited Mar 7 '15 at 22:30









ctrl-alt-delor

12k42360




12k42360










asked Mar 7 '15 at 20:18









Snazzy SanojSnazzy Sanoj

1195




1195





bumped to the homepage by Community♦ 1 hour ago


This question has answers that may be good or bad; the system has marked it active so that they can be reviewed.







bumped to the homepage by Community♦ 1 hour ago


This question has answers that may be good or bad; the system has marked it active so that they can be reviewed.










  • 1





    It's better to you make filter, you have some way, find, globbing and so on , But at first you explain exactly your task.

    – PersianGulf
    Mar 7 '15 at 20:22






  • 1





    Could you clarify what you mean by "loading"? Simply ls, or displaying the contents in a file manager (which one?), or something else? On my system listing 7,000 files with ls on an ext4 filesystem is pretty much instantaneous, as is displaying the directory in Thunar.

    – Stephen Kitt
    Mar 7 '15 at 20:56











  • @SnazzySanoj please answer comments, by amending your question.

    – ctrl-alt-delor
    Mar 7 '15 at 22:30






  • 1





    A lot of systems alias ls to something that makes it lstat every file. --color and -F do this. If you don't need ls to decorate the output, remove these aliases. Listing a 7000 file directory shouldn't be slow.

    – Mark Plotnick
    Mar 7 '15 at 23:48














  • 1





    It's better to you make filter, you have some way, find, globbing and so on , But at first you explain exactly your task.

    – PersianGulf
    Mar 7 '15 at 20:22






  • 1





    Could you clarify what you mean by "loading"? Simply ls, or displaying the contents in a file manager (which one?), or something else? On my system listing 7,000 files with ls on an ext4 filesystem is pretty much instantaneous, as is displaying the directory in Thunar.

    – Stephen Kitt
    Mar 7 '15 at 20:56











  • @SnazzySanoj please answer comments, by amending your question.

    – ctrl-alt-delor
    Mar 7 '15 at 22:30






  • 1





    A lot of systems alias ls to something that makes it lstat every file. --color and -F do this. If you don't need ls to decorate the output, remove these aliases. Listing a 7000 file directory shouldn't be slow.

    – Mark Plotnick
    Mar 7 '15 at 23:48








1




1





It's better to you make filter, you have some way, find, globbing and so on , But at first you explain exactly your task.

– PersianGulf
Mar 7 '15 at 20:22





It's better to you make filter, you have some way, find, globbing and so on , But at first you explain exactly your task.

– PersianGulf
Mar 7 '15 at 20:22




1




1





Could you clarify what you mean by "loading"? Simply ls, or displaying the contents in a file manager (which one?), or something else? On my system listing 7,000 files with ls on an ext4 filesystem is pretty much instantaneous, as is displaying the directory in Thunar.

– Stephen Kitt
Mar 7 '15 at 20:56





Could you clarify what you mean by "loading"? Simply ls, or displaying the contents in a file manager (which one?), or something else? On my system listing 7,000 files with ls on an ext4 filesystem is pretty much instantaneous, as is displaying the directory in Thunar.

– Stephen Kitt
Mar 7 '15 at 20:56













@SnazzySanoj please answer comments, by amending your question.

– ctrl-alt-delor
Mar 7 '15 at 22:30





@SnazzySanoj please answer comments, by amending your question.

– ctrl-alt-delor
Mar 7 '15 at 22:30




1




1





A lot of systems alias ls to something that makes it lstat every file. --color and -F do this. If you don't need ls to decorate the output, remove these aliases. Listing a 7000 file directory shouldn't be slow.

– Mark Plotnick
Mar 7 '15 at 23:48





A lot of systems alias ls to something that makes it lstat every file. --color and -F do this. If you don't need ls to decorate the output, remove these aliases. Listing a 7000 file directory shouldn't be slow.

– Mark Plotnick
Mar 7 '15 at 23:48










1 Answer
1






active

oldest

votes


















0














A large directory can be problematic for a number of reasons other than simply listing of files. For one thing, the time it takes to open a file in that directory will increase because the directory has to be read until the file is found. On many filesystems including ext*, directory entries not organized nor optimized for retrieval efficiency.



Answering your specific question, I think that you'll find using ls takes awhile due to the sorting involved. A solution is to run ls unsorted (if there is such an option in your distribution of ls). Specifically, I might ls unsorted to a file and the sort it. I then have the file to refer to without having to do another ls for a bit.



Another similar method is to use the find command and listout the directory contents into a file (which will be unsorted) and then go from there.



Thus my suggestion would be based on accessibility, effeciency and easily finding files, to use multiple sub-directories.






share|improve this answer


























  • "A directory is not organized to speed up retrievals of specific entries": that completely depends on the filesystem in use, and many commonly-used filesystem types do optimize directory indices for just this ability.

    – Celada
    Mar 8 '15 at 5:09











  • OK, I qualified my response. Could you provide the commonly used filesystems that do what you state for the record?

    – mdpc
    Mar 8 '15 at 5:42






  • 1





    XFS automatically, ext4 (probably ext3 as well?) with the "dir_index" option enabled (which is part of the default options) (see mkfs.ext4 manpage), and I didn't bother to look up references but probably most of the other modern filesystem types as well: Btrfs, ZFS, JFS, all support efficient indexing for large directories.

    – Celada
    Mar 8 '15 at 7:19











Your Answer








StackExchange.ready(function() {
var channelOptions = {
tags: "".split(" "),
id: "106"
};
initTagRenderer("".split(" "), "".split(" "), channelOptions);

StackExchange.using("externalEditor", function() {
// Have to fire editor after snippets, if snippets enabled
if (StackExchange.settings.snippets.snippetsEnabled) {
StackExchange.using("snippets", function() {
createEditor();
});
}
else {
createEditor();
}
});

function createEditor() {
StackExchange.prepareEditor({
heartbeatType: 'answer',
autoActivateHeartbeat: false,
convertImagesToLinks: false,
noModals: true,
showLowRepImageUploadWarning: true,
reputationToPostImages: null,
bindNavPrevention: true,
postfix: "",
imageUploader: {
brandingHtml: "Powered by u003ca class="icon-imgur-white" href="https://imgur.com/"u003eu003c/au003e",
contentPolicyHtml: "User contributions licensed under u003ca href="https://creativecommons.org/licenses/by-sa/3.0/"u003ecc by-sa 3.0 with attribution requiredu003c/au003e u003ca href="https://stackoverflow.com/legal/content-policy"u003e(content policy)u003c/au003e",
allowUrls: true
},
onDemand: true,
discardSelector: ".discard-answer"
,immediatelyShowMarkdownHelp:true
});


}
});














draft saved

draft discarded


















StackExchange.ready(
function () {
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2funix.stackexchange.com%2fquestions%2f188793%2fhow-to-speed-up-directory-listing-on-high-inode-partitions%23new-answer', 'question_page');
}
);

Post as a guest















Required, but never shown

























1 Answer
1






active

oldest

votes








1 Answer
1






active

oldest

votes









active

oldest

votes






active

oldest

votes









0














A large directory can be problematic for a number of reasons other than simply listing of files. For one thing, the time it takes to open a file in that directory will increase because the directory has to be read until the file is found. On many filesystems including ext*, directory entries not organized nor optimized for retrieval efficiency.



Answering your specific question, I think that you'll find using ls takes awhile due to the sorting involved. A solution is to run ls unsorted (if there is such an option in your distribution of ls). Specifically, I might ls unsorted to a file and the sort it. I then have the file to refer to without having to do another ls for a bit.



Another similar method is to use the find command and listout the directory contents into a file (which will be unsorted) and then go from there.



Thus my suggestion would be based on accessibility, effeciency and easily finding files, to use multiple sub-directories.






share|improve this answer


























  • "A directory is not organized to speed up retrievals of specific entries": that completely depends on the filesystem in use, and many commonly-used filesystem types do optimize directory indices for just this ability.

    – Celada
    Mar 8 '15 at 5:09











  • OK, I qualified my response. Could you provide the commonly used filesystems that do what you state for the record?

    – mdpc
    Mar 8 '15 at 5:42






  • 1





    XFS automatically, ext4 (probably ext3 as well?) with the "dir_index" option enabled (which is part of the default options) (see mkfs.ext4 manpage), and I didn't bother to look up references but probably most of the other modern filesystem types as well: Btrfs, ZFS, JFS, all support efficient indexing for large directories.

    – Celada
    Mar 8 '15 at 7:19
















0














A large directory can be problematic for a number of reasons other than simply listing of files. For one thing, the time it takes to open a file in that directory will increase because the directory has to be read until the file is found. On many filesystems including ext*, directory entries not organized nor optimized for retrieval efficiency.



Answering your specific question, I think that you'll find using ls takes awhile due to the sorting involved. A solution is to run ls unsorted (if there is such an option in your distribution of ls). Specifically, I might ls unsorted to a file and the sort it. I then have the file to refer to without having to do another ls for a bit.



Another similar method is to use the find command and listout the directory contents into a file (which will be unsorted) and then go from there.



Thus my suggestion would be based on accessibility, effeciency and easily finding files, to use multiple sub-directories.






share|improve this answer


























  • "A directory is not organized to speed up retrievals of specific entries": that completely depends on the filesystem in use, and many commonly-used filesystem types do optimize directory indices for just this ability.

    – Celada
    Mar 8 '15 at 5:09











  • OK, I qualified my response. Could you provide the commonly used filesystems that do what you state for the record?

    – mdpc
    Mar 8 '15 at 5:42






  • 1





    XFS automatically, ext4 (probably ext3 as well?) with the "dir_index" option enabled (which is part of the default options) (see mkfs.ext4 manpage), and I didn't bother to look up references but probably most of the other modern filesystem types as well: Btrfs, ZFS, JFS, all support efficient indexing for large directories.

    – Celada
    Mar 8 '15 at 7:19














0












0








0







A large directory can be problematic for a number of reasons other than simply listing of files. For one thing, the time it takes to open a file in that directory will increase because the directory has to be read until the file is found. On many filesystems including ext*, directory entries not organized nor optimized for retrieval efficiency.



Answering your specific question, I think that you'll find using ls takes awhile due to the sorting involved. A solution is to run ls unsorted (if there is such an option in your distribution of ls). Specifically, I might ls unsorted to a file and the sort it. I then have the file to refer to without having to do another ls for a bit.



Another similar method is to use the find command and listout the directory contents into a file (which will be unsorted) and then go from there.



Thus my suggestion would be based on accessibility, effeciency and easily finding files, to use multiple sub-directories.






share|improve this answer















A large directory can be problematic for a number of reasons other than simply listing of files. For one thing, the time it takes to open a file in that directory will increase because the directory has to be read until the file is found. On many filesystems including ext*, directory entries not organized nor optimized for retrieval efficiency.



Answering your specific question, I think that you'll find using ls takes awhile due to the sorting involved. A solution is to run ls unsorted (if there is such an option in your distribution of ls). Specifically, I might ls unsorted to a file and the sort it. I then have the file to refer to without having to do another ls for a bit.



Another similar method is to use the find command and listout the directory contents into a file (which will be unsorted) and then go from there.



Thus my suggestion would be based on accessibility, effeciency and easily finding files, to use multiple sub-directories.







share|improve this answer














share|improve this answer



share|improve this answer








edited Mar 8 '15 at 5:41

























answered Mar 8 '15 at 3:12









mdpcmdpc

5,05621838




5,05621838













  • "A directory is not organized to speed up retrievals of specific entries": that completely depends on the filesystem in use, and many commonly-used filesystem types do optimize directory indices for just this ability.

    – Celada
    Mar 8 '15 at 5:09











  • OK, I qualified my response. Could you provide the commonly used filesystems that do what you state for the record?

    – mdpc
    Mar 8 '15 at 5:42






  • 1





    XFS automatically, ext4 (probably ext3 as well?) with the "dir_index" option enabled (which is part of the default options) (see mkfs.ext4 manpage), and I didn't bother to look up references but probably most of the other modern filesystem types as well: Btrfs, ZFS, JFS, all support efficient indexing for large directories.

    – Celada
    Mar 8 '15 at 7:19



















  • "A directory is not organized to speed up retrievals of specific entries": that completely depends on the filesystem in use, and many commonly-used filesystem types do optimize directory indices for just this ability.

    – Celada
    Mar 8 '15 at 5:09











  • OK, I qualified my response. Could you provide the commonly used filesystems that do what you state for the record?

    – mdpc
    Mar 8 '15 at 5:42






  • 1





    XFS automatically, ext4 (probably ext3 as well?) with the "dir_index" option enabled (which is part of the default options) (see mkfs.ext4 manpage), and I didn't bother to look up references but probably most of the other modern filesystem types as well: Btrfs, ZFS, JFS, all support efficient indexing for large directories.

    – Celada
    Mar 8 '15 at 7:19

















"A directory is not organized to speed up retrievals of specific entries": that completely depends on the filesystem in use, and many commonly-used filesystem types do optimize directory indices for just this ability.

– Celada
Mar 8 '15 at 5:09





"A directory is not organized to speed up retrievals of specific entries": that completely depends on the filesystem in use, and many commonly-used filesystem types do optimize directory indices for just this ability.

– Celada
Mar 8 '15 at 5:09













OK, I qualified my response. Could you provide the commonly used filesystems that do what you state for the record?

– mdpc
Mar 8 '15 at 5:42





OK, I qualified my response. Could you provide the commonly used filesystems that do what you state for the record?

– mdpc
Mar 8 '15 at 5:42




1




1





XFS automatically, ext4 (probably ext3 as well?) with the "dir_index" option enabled (which is part of the default options) (see mkfs.ext4 manpage), and I didn't bother to look up references but probably most of the other modern filesystem types as well: Btrfs, ZFS, JFS, all support efficient indexing for large directories.

– Celada
Mar 8 '15 at 7:19





XFS automatically, ext4 (probably ext3 as well?) with the "dir_index" option enabled (which is part of the default options) (see mkfs.ext4 manpage), and I didn't bother to look up references but probably most of the other modern filesystem types as well: Btrfs, ZFS, JFS, all support efficient indexing for large directories.

– Celada
Mar 8 '15 at 7:19


















draft saved

draft discarded




















































Thanks for contributing an answer to Unix & Linux Stack Exchange!


  • Please be sure to answer the question. Provide details and share your research!

But avoid …



  • Asking for help, clarification, or responding to other answers.

  • Making statements based on opinion; back them up with references or personal experience.


To learn more, see our tips on writing great answers.




draft saved


draft discarded














StackExchange.ready(
function () {
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2funix.stackexchange.com%2fquestions%2f188793%2fhow-to-speed-up-directory-listing-on-high-inode-partitions%23new-answer', 'question_page');
}
);

Post as a guest















Required, but never shown





















































Required, but never shown














Required, but never shown












Required, but never shown







Required, but never shown

































Required, but never shown














Required, but never shown












Required, but never shown







Required, but never shown







-

Popular posts from this blog

宮崎県

Image
みやざきけん 宮崎県 高千穂峡 宮崎県旗 宮崎県徽章 国 日本 地方 九州地方 団体コード 45000-6 ISO 3166-2:JP JP-45 面積 7,735.32 km 2 （境界未定部分あり） 総人口 1,079,727 人 （推計人口、2018年10月1日） 人口密度 140 人/km 2 隣接都道府県 大分県、熊本県、鹿児島県 県の木 フェニックス（1966年9月3日制定） ヤマザクラ（2003年2月6日制定） オビスギ（飫肥杉）（2003年2月6日制定） 県の花 ハマユウ（1964年12月22日制定） 県の鳥 コシジロヤマドリ（1964年12月22日制定） 他のシンボル 県の歌：宮崎県民歌（1964年制定） マスコット：みやざき犬（2011年11月11日制定） 宮崎県庁 知事 河野俊嗣 法人番号 4000020450006 所在地 〒 880-8501 宮崎県宮崎市橘通東2丁目10番1号 北緯31度54分39.6秒東経131度25分25.9秒 座標: 北緯31度54分39.6秒 東経131度25分25.9秒 外部リンク 宮崎県 ■ ― 市 / ■ ― 町 / ■ ― 村 地理院地図 Google Bing GeoHack MapFan Mapion Yahoo! NAVITIME ゼンリン ウィキポータル 日本の都道府県/宮崎県   表示   ウィキプロジェクト 宮崎県 （みやざきけん）は、九州地方南東部に位置する日本の県である。県庁所在地は宮崎市。 県木である「フェニックス」に代表される南国情緒豊かな気候から、1960年代には日南地区を中心とした新婚旅行のメッカだった。現在も春季のプロ野球などのキャンプ地として知られる。 目次 1 地理・地域 1.1 地形 1.2 自然公園 1.3 気候 1.4 自治体 1.5 合併済みの市町村 2 歴史 2.1 神話 2...
Read more

濃尾地震

Image
濃尾地震 『岐阜市街大地震之図』 歌川国利画 本震 発生日 1891年（明治24年）10月28日 発生時刻 6時38分50秒（JST） 震央 日本 岐阜県本巣郡西根尾村（現・本巣市） 北緯35度35分 東経136度20分（ 地図 ） 規模    マグニチュード（M）8.0 最大震度    震度7： 注1 福井県今立郡鯖江町、愛知県葉栗郡大田島村、東春日井郡勝川村 地震の種類 直下型地震 被害 死傷者数 死者7,273人、負傷者17,175人 注1：当時の震度階級では「激烈」 プロジェクト:地球科学 プロジェクト:災害 テンプレートを表示 濃尾地震 （のうびじしん）は、1891年（明治24年）10月28日に濃尾地方で発生した、日本史上最大の内陸地殻内地震。「 美濃・尾張地震 （みの・おわりじしん）」とも呼ばれている。辛卯の年に発生したことから 辛卯震災 と呼んでいる報告書もある。 目次 1 概要 1.1 震源断層 2 被害 3 各地の震度 4 前兆現象 5 報道 6 学術的な意義 7 地震防災 8 脚注 9 関連項目 10 外部リンク 概要 濃尾地震発生当時の根尾谷断層 濃尾地震を引き起こした根尾谷断層 写真中央を斜めに走る段差が根尾谷断層 濃尾地震は、1891年10月28日6時38分50秒に発生した。震源は、岐阜県本巣郡西根尾村（現・本巣市）、北緯35度35分、東経136度20分付近。河角廣 (1951) は岐阜市付近（北緯35.6度、東経136.6度）に震央を仮定し規模 M K = 7.0 を与え [1] 、マグニチュードは M = 8.4 に換算されているが、明治・大正期の地震については0.5程度大きく見積もられているとされる [2] 。また、震央距離と震度との関係など当時のデータから後にM8.0 [3] とも推定される。アメリカ地質調査所 (USGS) でも8.0としている [4] 。「根尾谷断層帯」が活動した典型的な内陸地殻内地震（いわゆる直下型地震）であり、これは記録が残っている日本の内陸...
Read more

シテ島

Image
シテ島 Île de la Cité シテ島（トゥルネル橋からの眺望） 所在地 フランス･パリ 座標 北緯48度51分17秒 東経2度20分45秒 座標: 北緯48度51分17秒 東経2度20分45秒 面積 0.225 km² 最高標高 48.854722 m  プロジェクト 地形 テンプレートを表示 シテ島 （シテとう、仏 : Île de la Cité）は、サン＝ルイ島と並ぶ、パリの中心部を流れるセーヌ川の中州である。パリ市内で最も古い歴史をもつパリ1区及び4区に属し、“パリ発祥の地”とも称される。ユリウス・カエサルの『ガリア戦記』では、すでに紀元前1世紀にパリシイ族 (Parisii) が住んでいたことが述べられている。 目次 1 概要 2 歴史ギャラリー 3 名所 4 シテ島の交通 4.1 橋 4.2 鉄道 5 ギャラリー 6 関連項目 7 脚注 8 外部リンク 概要 東半分が4区に、西半分は1区に属する。 ノートルダム大聖堂とパリ警視庁とパリ市立病院（ オテル・デュー・ド・パリ （フランス語版） 、直訳では「パリの神の館」）がある。病院の名前は、旅の病人を宿泊させ、病になれば看病した教会、巡礼教会からの発展の名残である。近代外科医学の開拓者で、床屋医者の出身でもある外科医アンブロワーズ・パレが勤務していた病院としてもその名が知られている。位置関係は大聖堂が警視庁庁舎と向き合い、警視庁の隣に市立病院。 また、この島はとても狭いので、ここの住民たちは夜も一定の時間を過ぎるとトイレの水は流さない等、互いに気配りしつつ一緒に住んでいくことの不便さを味わってきた。ここから「シテ島に住む者」を表す古仏語：citeain （現代仏語：シトワイヤン citoyen 、現代英語：シティズン citizen）の意味が転じ、「 市民 」という言葉が成立したという（仏語：cité は英語：city に相当する）。 中世以来からの狭く入り組んだ路地は、19世紀に入りセーヌ川左岸と共に貧民窟化していた。そうした狭く不衛生で入り組んだ空間は、2月革命に見るようなバリケード戦といった民衆騒...
Read more