How to import external package symbols with excluding some

Question

In a new package I want to :use (inherit from) package packa and packb.

packa and packb have an intersection of exported symbols.

How is it possible to inherit all from packa but only those of packb that are not intersecting with packa?

UPDATE:

I've tried :use packa and selectively :import-from symbols from packb that I need and don't collide with packa. However that's quite combersome.

I've experimented a bit with do-external-symbols and intern/import, but that doesn't seem to work, or at least I don't know how it could work.

score 4 · Accepted Answer · answered Jan 07 '23 at 20:48

4

Shadowing is the way to avoid conflicts in this case.

If you want to use symbols from packa without a prefix, use :shadowing-import-from #:packa #:sym1 #:sym2 ... in your package definition. Or, use packb if you prefer those symbols without a prefix.

If you prefer to use prefixes for all of the conflicting symbols from both packages, use :shadow #:sym1 #:sym2 ... instead.

answered Jan 07 '23 at 20:48

Xach

11,774
37
38

CLHS MAKE-PACKAGE says: "In situations where the packages to be used contain symbols which would conflict, it is necessary to first create the package with :use '(), then to use shadow or shadowing-import to address the conflicts, and then after that to use use-package once the conflicts have been addressed." – Rainer Joswig Jan 08 '23 at 12:14
1

Yes, if you are in a situation where you must use MAKE-PACKAGE, that's the way to go. (This isn't very common, is it?) – Xach Jan 08 '23 at 14:35
one maybe wants to compute the conflicting symbols, instead of writing them down. why not use make-package then? – Rainer Joswig Jan 08 '23 at 15:41
I'm using `:shadow`. Works fine in my situation. Though this `conduit-packages` library with ':extends/including' / 'excluding' is semantically nicer. – Manfred Jan 08 '23 at 17:39

score 3 · Answer 2 · answered Jan 07 '23 at 15:37

I assume that by 'an intersection of exported symbols' what you mean is that the two packages have an intersection of exported symbol names, not the symbols themselves (see below). So, for instance, I assume the packages are defined something like this:

(defpackage :p1
  (:use)
  (:export #:s1 #:s2))

(defpackage :p2
  (:use)
  (:export #:s2 #:s3))

Which means that both P1 and P2 export a symbol named "S2" but (eq 'p1:s2 'p2:s2) is false.

In this case you can't use both P1 and P2. You can massage things by explicitly importing (or shadowing-importing) symbols but generally that's quite undesirable, not to mention messy.

A good approach in this case is to define a conduit package, which is a package which simply acts as a conduit between you and one or more implementation packages, reexporting symbols as needed. The simplest approach is to use a pre-canned way of creating conduit packages such as Tim Bradshaw's 'conduit-packages' system, available in Quicklisp. (Some tentacle of ASDF also has a similar system I think but I am not familiar with that one.) This provides an extended variant of defpackage which does what you need.

Using this system, and with the above package definitions, you might say this to create a conduit:

(define-conduit-package :p3
  (:use)
  (:extends :p1)
  (:extends/excluding :p2 #:s2))

This new package P3 now reexports the symbols from P1 and P2, except for P2:S2:

 (use-package :p3)
t

> (symbol-package 's1)
#<The P1 package, 0/16 internal, 2/16 external>

> (symbol-package 's2)
#<The P1 package, 0/16 internal, 2/16 external>

> (symbol-package 's3)
#<The P2 package, 0/16 internal, 2/16 external>

You can obviously provide more than one symbol name to exclude, and you can pick and choose: assume you now have

(defpackage :p1
  (:use)
  (:export #:s1 #:s2 #:s3)

(defpackage :p2
  (:use)
  (:export #:s2 #:s3 #:s4))

(define-conduit-package :p3
  (:use)
  (:extends/excluding :p1 #:s3)
  (:extends/excluding :p2 #:s2))

Then, this time without using P3 to make it easier to see:

> '(p3:s1 p3:s2 p3:s3 p3:s4)
(p1:s1 p1:s2 p2:s3 p2:s4)

You can also define conduits which extend packages only including certain symbol names, for instance:

(define-conduit-package :p4
  (:use)
  (:extends/including :p1 #:s1)
  (:extends/excluding :p2 #:s1))

will tell it to reexport only the symbol named "S1" from P1, and not to reexport any symbol with this name from P2.

Finally you can of course define conduits with functions. For instance:

(defun make-conduit-package-excluding (n for-package/s excluding-from-package/s)
  ;; extend FOR-PACKAGE/S, excluding exports from EXCLUDING-FROM-PACKAGE/S
  (let ((conduit (make-package n :use '()))
        (excluders (if (listp excluding-from-package/s)
                       (mapcar #'find-package excluding-from-package/s)
                     (list (find-package excluding-from-package/s)))))
    (dolist (p (if (listp for-package/s)
                    (mapcar #'find-package for-package/s)
                  (list (find-package for-package/s)))
               conduit)
      (do-external-symbols (s p)
        (let ((sn (symbol-name s)))
          (unless (some (lambda (excluder)
                        (multiple-value-bind (ss status) (find-symbol sn excluder)
                          (and (eq status ':external)
                               (not (eq ss s)))))
                      excluders)
          (import s conduit)
          (export s conduit)))))))

Now if I say (using the most recent definitions of P1 and P2 above:

> (make-conduit-package-excluding "P5" "P1" "P2")
#<The P5 package, 0/16 internal, 1/16 external>

> (use-package "P5")
t

> (use-package "P2")
t

Everything is again OK because I told the function that P5 should not reexport any symbols from P1 which would clash with P2's exports.

A note on 'intersection of exported symbols'. If you have two packages which export some of the same symbols rather than different symbols with the same names, then there is no issue using both. For example:

(defpackage :cl-re-1
  (:use :cl)
  (:export #:defpackage))


(defpackage :cl-re-2
  (:use :cl)
  (:export #:defpackage))

Then

> 'cl-re-1:defpackage
defpackage

> 'cl-re-2:defpackage
defpackage

> (use-package :cl-re-1)
t

> (use-package :cl-re-2)
t

It's neither undesirable nor messy to use shadowing to control how symbols are found. It's standard. — Xach, Jan 07 '23 at 20:45
@Xach: standards can also be ugly, undesirable, fragile, hacks. Better standards can be created if people will only try. — ignis volens, Jan 07 '23 at 23:59
Thanks for the writeup. The library ('conduit-packages') looks great. Semantically I like this better than the `:shadow` approach. — Manfred, Jan 08 '23 at 17:35

score 1 · Answer 3 · answered Jan 08 '23 at 12:25

If the problem is because you don't want to manually write down all the overlapping functions - you could let a function write it down for you.

Let's say in a setting of:

(defpackage :p1
  (:use :cl)
  (:export #:s1 #:s2 #:s3))

(in-package :p1)

(defconstant s1 1)
(defconstant s2 2)
(defconstant s3 3)

(in-package :cl-user)

(defpackage :p2
  (:use :cl)
  (:export #:s2 #:s3 #:s4))

(in-package :p2)

(defconstant s2 "b")
(defconstant s3 "c")
(defconstant s4 "d")

(in-package :cl-user)
(defpackage :p3
  (:use :cl :p1 :p2)
  (:shadowing-import-from :p1 #:s2 #:s3)
  (:export #:s1 #:s2 #:s3 #:4))

(in-package :p3)

;; package local nicknames
;; https://gist.github.com/phoe/2b63f33a2a4727a437403eceb7a6b4a3

(in-package :cl-user)

(defpackage :p4
  (:use #:cl #:p1 #:p2)
  (:shadowing-import-from #:p2 #:s2 #:s3)
  (:export #:s1 #:s2 #:s3 #:s4))

(defpackage #:p5
  (:use #:cl-user)
  (:export #:s2 #:s3 #:s4))

I prefer to use uninterened keywords ("#:") to not to "pollute" the KEYWORD package's content.

One could then define the helper functions:

(defun make-uninterned-keyword (name)
  "String to Uninterned Keyword"
  (read-from-string (format nil "#:~a" name)))

(defun make-keyword (name)
  "String to Keyword"
  (values (intern (string-upcase name) "KEYWORD")))

(defun get-symbols (package)
  "Return all symbols of a package 
   (`package` should be a string)"
  (let (symbols)
     (do-external-symbols (s (find-package (string-upcase package)))
       (push s symbols))
    (nreverse (mapcar #'symbol-name symbols))))
;; (lambda (s) (read-from-string (format nil "#:~a" (symbol-name s))))

(defun %overlapping-symbols (package-1 package-2)
  "Determine symbols overlapping between the packages"
  (let ((symbols-1 (get-symbols package-1))
        (symbols-2 (get-symbols package-2)))
    (intersection symbols-1 symbols-2 :test #'string=)))

(defun overlapping-symbols (package &rest packages)
  "Determine symbols overlapping from first package with the rest of the packages"
  (remove-duplicates (loop for pkg in packages
                           nconcing (%overlapping-symbols package pkg))
                     :test #'string=))

(defun generate-shadowing-import-form (package &rest packages)
  "Construct code for shadowing-import-from"
  (let* ((overlapping-symbols (apply #'overlapping-symbols package packages))
         (overlapping-keywords (mapcar #'make-uninterned-keyword overlapping-symbols)))
    `(:shadowing-import-from ,(make-uninterned-keyword package) ,@overlapping-keywords)))

(defun shadowing-import-string (package &rest packages)
  "Construct string for :shadowing-import-from directive"
  (string-downcase (format nil "~s" (apply #'generate-shadowing-import-form package packages))))

So by running:

(shadowing-import-string "P2" "P1" "P5")
;; => "(:shadowing-import-from #:p2 #:s2 #:s3 #:s4)"

You get the piece of code you want to copy paste into your DEFPACKAGE definition. Any symbol in the :p2 package which overlaps with the :p1 and/or the :p5 package is then listed.

How to import external package symbols with excluding some

3 Answers3