Managing many thousand buffers in emacs

Question

I am on a project with over ten thousand files. I would like to be able to navigate around these files easily.

iswitchb mode bogs down quite badly with this many files; icicles is better in speed, but I like the UI of iswitchb.

Suggestions?

Stefan requested a profiler run on the iswitchb-mode execution:

- call-interactively                                             2740  79%
  - iswitchb-buffer                                              2628  76%
    - iswitchb                                                   2628  76%
      - iswitchb-read-buffer                                     2628  76%
        - completing-read                                        2599  75%
          - completing-read-default                              2596  75%
            - read-from-minibuffer                               2596  75%
              - iswitchb-post-command                            1034  30%
                - iswitchb-exhibit                               1034  30%
                  - iswitchb-set-common-completion                947  27%
                    - iswitchb-find-common-substri                925  26%
                      - mapcar                                    925  26%
                        - iswitchb-word-matching-s                922  26%
                          + iswitchb-case                          10   0%
                            string-match                            1   0%
                  + iswitchb-set-matches                           63   1%
                  + iswitchb-completions                           24   0%
              + redisplay_internal (C function)                   869  25%
              - timer-event-handler                               499  14%
                - byte-code                                       499  14%
                  - apply                                         499  14%
                    - jit-lock-context-fontify                    491  14%
                      + buffer-list                                15   0%
                    + show-paren-function                           6   0%
                    + display-time-event-handler                    2   0%
              + call-interactively                                 34   0%
              + icicle-unhighlight-lighter                          3   0%
              + iswitchb-pre-command                                1   0%
        + iswitchb-make-buflist                                    15   0%

I'm guessing, judging by your reputation (and the way you've asked the question), that this isn't some stupid request that could be done far more easily and efficiently. Why do you need to navigate so many files at once? Are these all source code files? I dont't have a solution for you, just curious. — Reinstate Monica -- notmaynard, Nov 05 '13 at 23:59
@iamnotmaynard: All are source code. I'm tracing code execution paths that wander around. If you assume each file has an average of 500 LoC, that's a multimillion LoC project to explore and maintain. — Paul Nathan, Nov 06 '13 at 00:04
If I may, what is the language of the project? Some languages might have tools outside editors to do that kind of stuff. `clang` can offer some source code analysis of this kind for C/C++. Ropes can do for Python. Having this many buffers open would be a performance hit for any editor, code navigation usually relies on some sort of a project model instead of literally searching through files. — , Nov 06 '13 at 00:31
@wxvvw: mix of c# and objectiveC - tired of flipping between the two IDEs and wishing my beloved emacs would work for me. :) — Paul Nathan, Nov 06 '13 at 00:41
Other answers at http://stackoverflow.com/questions/19283368/how-can-i-open-quickly-a-file-in-emacs. — Stefan, Nov 06 '13 at 01:05
@Stefan: opening files isn't a problem. :-p it's the gigantic amount of files that's causing issues. — Paul Nathan, Nov 06 '13 at 01:11
Have you considered using frames to help organize files into groups and then switching between frames? Frame-bufs by Al Parker was just recently updated to the current version of Emacs: https://github.com/alpaker/Frame-Bufs You can set up a regexp for display-buffer sending certain file groups to certain frames. Here is an example: http://stackoverflow.com/questions/18346785/how-to-intercept-a-file-before-it-opens-and-decide-which-frame — lawlist, Nov 06 '13 at 01:51
Here is an example where I have blended tabbar with frame-bufs, using regexp and `display-buffer` -- my display-buffer function automatically associates select files with the specific frames as the files are being opened. You do *not* need to use tabbar for this to work. I have added a couple of functions like associate-current-buffer, disassociate-current-buffer, and lawlist-frame-bufs-reset. I have also modified frame-bufs so that it works independently of buff-menu.el, so changes in Emacs versions are less likely to break its functionality. https://github.com/lawlist/tabbar-lawlist — lawlist, Nov 06 '13 at 02:12
@PaulNathan: I understand why you have thousands of files, but not why you have thousands of buffers. Emacs code is usually not optimized for thousands of buffers, so maybe your problem is a simple performance bug that can be fixed easily. Please try `M-x profiler-start RET RET`, then use iswitchb for a while, triggering repeatedly the slow behavior, then `M-x profiler-report` (don't forget to hit RET on the "+" there to see the inside of the profile trees). — Stefan, Nov 06 '13 at 13:29
@Stefan: I recognize I might be breaking the designed limitations of emacs here. I'll do some profiling today. — Paul Nathan, Nov 06 '13 at 18:32
The profiler info indicates that the problem is probably that Iswitchb takes a long time to generate a very long line (the 25% spent in redisplay is most likely due to the long line). `icomplete` in contrast stops after outputting the equivalent of `icomplete-prospects-height` lines, so not only it spends less time processing a long list, but the redisplay then also spends much less time diplaying it. — Stefan, Nov 11 '13 at 01:55

score 3 · Answer 1 · edited Mar 20 '15 at 15:47

If you want to use Icicles, but you want the interactive behavior to more closely resemble iswitchb, then there are several customizations you can try, depending on which parts of the iswitchb interaction you like. See this section of the Icicles doc: Icicles - Ido and IswitchB.

Wrt using zillions of files: remember that you can also define projects using Icicles -- sets of related files that you use together. That, like etags and other approaches, is one way to pre-filter the possible file-name candidates.

score 3 · Answer 2 · answered Nov 06 '13 at 19:13

Having just tried iswitchb in a sessions with 10000 buffers, I see indeed that's it's unbearably slow.

I suggest you try M-x icomplete-mode: in Emacs-24.3, the default C-x b completion already accepts substring matches, just like iswitchb, so the only missing feature is the one offered by icomplete-mode, but icomplete-mode is still perfectly fast in my session with 10000 buffers. Basically, since Emacs-24.3, iswitchb is made obsolete by icomplete-mode.

score 0 · Answer 3 · answered Nov 06 '13 at 02:31

Have you tried this?

iswitchb-max-to-show is a variable defined in `iswitchb.el'.
Its value is nil

Documentation:
If non-nil, limit the number of names shown in the minibuffer.
If this value is N, and N is greater than the number of matching
buffers, the first N/2 and the last N/2 matching buffers are
shown.  This can greatly speed up iswitchb if you have a
multitude of buffers open.

You can customize this variable.

score 0 · Answer 4 · answered Nov 06 '13 at 06:57

0

You can give Helm a try. It can be used for switching buffers and much more: http://amitp.blogspot.hu/2012/10/emacs-helm-for-finding-files.html

answered Nov 06 '13 at 06:57

Tom

7,515
7
38
54

score 0 · Answer 5 · answered Nov 06 '13 at 17:35

Can your zillions of files can be organized into useful sets -- even perhaps overlapping sets?

Bookmarks can help a lot with such organization. With Bookmark+, you can bookmark:

Individual files.
Sets of files.
Individual Dired buffers, recording their markings, "omit" settings, and inserted subdirectories.
Sparse trees of Dired buffers, based on marked subdirectories, recursively.
Emacs desktops.
Bookmark files, i.e., sets of bookmarks.
Bookmark-list displays, with their settings (markings, omissions etc.).

And you can tag bookmarks, delicious-style. Tags are names (arbitrary strings) you pick to categorize bookmarks or files (i.e., autofile bookmarks). You can assign any number of tags to a bookmark. Tags can even have associated (arbitrary) Lisp values.

You can bookmark the same file (or file set) multiple times, and give those different bookmarks different tags.

IOW, there are various ways to define different "views" of files and sets of files.

No, things are pretty random-access, no groupings feasible at this point. — Paul Nathan, Nov 06 '13 at 18:32

Managing many thousand buffers in emacs

5 Answers5