Hi guys.

Users have taken to the streets and are crying for a push-button way to:

• Declare Lisp and Java (Maven Central) dependencies side-by-side.
• Easily locate and load those Java dependencies into the REPL.
• Expose some main function from Lisp and bundle a single image / fatjar.
• Do all this from the comfort of an ABCL repl session.

Much of what’s required for this seemingly already exists. This thread
elaborates prior art in the field, and lays out suggestions for future
development.

Prior Art

abcl-asdf

    (asdf:defsystem #:abcl-telegram-bot
      :description "Create telegram bots with ABCL"
      :author "Alejandro Zamora Fonseca <ale2014.zamora@gmai.com>"
      :license  "MIT"
      :version "0.0.1"
      :serial t
      :depends-on (:abcl-memory-compiler :alexandria)
      :components ((:mvn "org.telegram/telegrambots-longpolling/8.3.0")
                   (:mvn "org.telegram/telegrambots-client/8.3.0")
                   (:file "package")
                   (:file "abcl-telegram-bot")))

abcl-asdf extends ASDF to comprehend (:mvn ...) entries under :components. Upon
system load, it downloads these Java dependencies to ~/.m2/, the location of
which can’t be configured, even though the mvn CLI tool allows it:

mvn dependency:get \
    -DgroupId=org.apache.commons \
    -DartifactId=commons-text \
    -Dversion=1.13.1 \
    -Dmaven.repo.local=vendored/java/

Pros:

• This provides a first-class way to specify Java dependencies alongside Lisp ones.
• Downloading and classpath management is handled automatically.

Issues:

• Counter-intuitive is that these Java dependencies aren’t defined within
  :depends-on.
• If you haven’t already manually loaded abcl-contrib and abcl-asdf before
  loading this system, ASDF doesn’t know what to do with the :mvn blocks and
  throws an error.
• There is seemingly no way to tell abcl-asdf not to download or not to manage
  the classpath.

java:add-to-classpath

If you have a JAR somewhere on your machine, there is technically no need to go
through abcl-asdf; adding it manually to the classpath is sufficient:

    (java:add-to-classpath "/home/colin/code/common-lisp/vend/vendored/java/commons-io/commons-io/2.16.1/commons-io-2.16.1.jar")
    (#"toAbsolutePath" (#"current" 'org.apache.commons.io.file.PathUtils))
    ;; => #<sun.nio.fs.UnixPath /home/colin/code/common-lisp/ven.... {440C8006}>

It’s simple enough for any tool trying to build up classpath entries to
recursively parse POM-file XML and call add-to-classpath as appropriate.

One thing I’m not sure about is whether calling add-to-classpath is still
necessary when we’ve already built a fatjar with jar cfm and a MANIFEST.MF. I
suspect not but haven’t confirmed.

asdf-jar

This provides a way to package all our Lisp sources (and dependencies!) into a
single JAR. Given:

    (defpackage abcl-test
      (:use :cl :arrow-macros)
      (:export #:launch))
    
    (in-package :abcl-test)
    
    (require :java)
    
    (defun launch ()
      (->> (java:jstatic "now" "java.time.LocalDate")
           (java:jcall "toString")
           (format t "Date: ~a~%")))

then by calling:

    (require :abcl-contrib)
    (require :asdf-jar)
    
    (asdf-jar:package :abcl-test :out #p"./" :fasls t :verbose t)

we get our JAR.

Issues:

• As mentioned in this issue, despite :fasls t, subsequent asdf:load-system
  calls don’t seem to respect the bundled .abcl fasl files.
• As mentiond in this issue, loading asdf-jar is inconsistent and often fails.

Lisp-from-Java wrapping and jar cfm

In theory a universal entry-point runner could be written on the Java side to
run our program:

    import org.armedbear.lisp.Interpreter;
    
    public class Main
    {
        public static void main(String[] args) {
            Interpreter i = Interpreter.createInstance();
            i.eval("(require :asdf)");
            i.eval("(require :abcl-contrib)");
            i.eval("(require :asdf-jar)");
            // Somehow refer to the child JAR within this JAR.
            // i.eval("(asdf-jar:add-to-asdf \"/home/colin/code/common-lisp/abcl-test/abcl-test-all-0.0.1.jar\")");
            i.eval("(asdf:load-system :abcl-test)");
            i.eval("(abcl-test:launch)");
        }
    }

We can compile this with:

javac -cp /usr/share/java/abcl.jar Main.java

And given a MANIFST.MF:

Manifest-Version: 1.0
Main-Class: Main
Class-Path: /usr/share/java/abcl.jar /usr/share/java/abcl-contrib.jar /home/colin/code/common-lisp/abcl-test/abcl-test-all-0.0.1.jar

The Class-Path here is probably wrong, but all this can be bundled together into
a single JAR with:

jar cfm app.jar MANIFEST.MF Main.class \
    -C /usr/share/java/ abcl.jar \
    -C /usr/share/java/ abcl-contrib.jar \
    -C /home/colin/code/common-lisp/abcl-test/ abcl-test-all-0.0.1.jar

We now have a “fatjar”. Then calling java -jar app.jar actually runs! But it
currently always fails trying to load asdf-jar, as mentioned above. If we could
get past that, and work out the classpath issues, in theory we have all the
pieces (although not at all automated).

asdf:make

Briefly I will mention a few other solutions for inspiration.

asdf:make simply fails under ABCL. Given:

    (defsystem "abcl-test"
      :version "0.0.1"
      :depends-on (:arrow-macros)
      :components ((:module "src" :components ((:file "main"))))
      :build-operation program-op
      :build-pathname "abcl-test"
      :entry-point "abcl-test:launch")

We are told:

#<THREAD “interpreter” native {518FC5E}>: Debugger invoked on condition of type NOT-IMPLEMENTED-ERROR
Not (currently) implemented on ABCL: UIOP/IMAGE:DUMP-IMAGE dumping an executable

(ECL) asdf:make-build

ECL relies on a special implementation of asdf:make-build, otherwise normally
deprecated, for building its own binaries. This doesn’t require special entries
in .asd. Here is how vend is built:

    (asdf:make-build :vend
                     :type :program
                     :move-here #p"./"
                     :epilogue-code '(vend:main))

Dead simple. See here for a bigger example that sets linker flags for binding to
C (.so) dependencies.

(SBCL) sb-ext:save-lisp-and-die

SBCL doesn’t really differentiate between building an “image” and building an
executable; in the latter case there is simply a well-defined entrypoint, and
the blob can be run as-is from the terminal.

    (sb-ext:save-lisp-and-die #p"aero-fighter"
                              :toplevel #'aero-fighter:launch
                              :executable t
                              :compression t)

Note that you can’t call this from within Sly/Slime sessions, it typically must
be done in a fresh, standalone REPL (or build script).

(Clojure) Naive running

Assuming the user has Clojure installed, it’s enough to run any program (with
any mix of Clojure and Maven dependencies) by running commands like:

clojure -M -m some_namespace

Where there is a file in your project somewhere defining a namespace / module
that contains a -main:

    (defn -main [& args]
      (println "Hi!"))

This “just works” with no extra config, and is especially convenient if you
don’t need to “distribute” to non-devs.

(Clojure) lein uberjar

The Clojure tool lein is also able to build uberjars. With a separate
project.clj:

    (defproject my-project "0.1.0-SNAPSHOT"
      :description "A Clojure project with Java dependencies"
      :dependencies [[org.clojure/clojure "1.11.1"]
                     [com.fasterxml.jackson.core/jackson-databind "2.17.2"]] ; Example Java dependency
      :main my-project.core
      :aot [my-project.core])

Then lein uberjar produces our fatjar that can be run with java -jar.

Recent Development

I have an experimental branch on vend that handles downloading through mvn to a
project-local locations, followed by POM parsing and classpath building. For the
rest of what’s needed, technically I can auto-generate a Main.java and a
Manifest file, and run various shell commands to produce the uberjar. However
we’re not quite there yet, and it would require an expansion of features on my
end to support the required configuration for ABCL projects.

The Future

Here are a few potential paths the future could take.

Status Quo: just run ABCL

Tell ABCL users to bundle a version of ABCL as a dependency of their production
app, where their “executable” becomes a wrapper around a call to abcl on their
code.

Pro: Nothing to do.

Con: Haven’t advanced the state of the art.

Implement uiop/image:dump-image

My personal bias is to not overrely on ASDF, especially where compilers have
their own first-class solution. That first-class solution will simply always be
better supported than asdf:make, etc. It’s fine to use ASDF to load systems, but
beyond that I don’t think it’s its responsibility to handle the production of
executables.

Fix asdf-jar + rely on external tooling

Perhaps it’s just a matter of making the Java wrapper shown above more
consistent, at which point vend or other tools can use all the existing
components to cobble together a runnable fatjar.

Pro: Potentially not too much work.

Con: No push-button solution from ABCL itself.

New core functionality / new contrib

Perhaps in tandem with an expansion to abcl-asdf (or a rewrite), ABCL can
provide some ext:fatjar function that:

• Compiles all FASLs and loads them into a JAR.
• Includes all specified Java deps from a customizable location (default to ~/.m2/).
• Includes the ABCL jar itself (probably contrib too).
• Accepts a :main or :entry keyword arg which accepts an entrypoint symbol.
• Produces a Main.class that internally invokes the entrypoint (similar to what’s shown above).

So the function could look something like:

    (ext:fatjar :my-project
                :main #'my-project:launch
                :java #p"/home/me/code/my-project/java-deps/")

Thank you for taking the time to read and consider this. Please let me know your thoughts.