👋 I'm a bit down a rabbit hole so I'll explain my journey and how I got here

I was taking a look at the newest version of pip today and I noticed it was visibly slower than before. a pretty noticeable half second pause on startup (450-500ms)

I did a little bit of profiling and noticed that almost half of that time (minus the base interpreter startup) was spent importing pyparsing and it felt like it was slower than before

I poked around a little bit and noticed that the performance regressed pretty significantly between 2.4.7 and 3.0.0

here's a comparison of some of the import times there (note that these are ~best of 5 which isn't necessarily a super useful representation of speed -- but it's enough to show significance here)

base interpreter startup

(essentially import site) -- I'm using python 3.8.10 here on ubuntu 20.04

$ time python3 -c ''

real	0m0.020s
user	0m0.020s
sys	0m0.000s

2.4.7

$ time python3 -c 'import pyparsing'

real	0m0.047s
user	0m0.043s
sys	0m0.004s

3.0.0

$ time python3 -c 'import pyparsing'

real	0m0.169s
user	0m0.168s
sys	0m0.000s

and just to confirm it's still present on the latest version, this is 938f59d

$ time python3 -c 'import pyparsing'

real	0m0.169s
user	0m0.169s
sys	0m0.000s

profiling

I did a bit of profiling, it looks like it spends ~75% of the time compiling regular expressions -- most of it coming from pyparsing.core:Regex.__init__ -- I'll attach a svg of the profile below

regressions (?)

import argparse
import subprocess
import sys
import time


def main() -> int:
    parser = argparse.ArgumentParser()
    parser.add_argument('--best-of', type=int, default=5)
    parser.add_argument('--cutoff', type=float, default=.1)
    args = parser.parse_args()

    best = None

    for i in range(args.best_of):
        t0 = time.monotonic()
        subprocess.check_call((sys.executable, '-c', 'import pyparsing'))
        t1 = time.monotonic()

        duration = t1 - t0
        if best is None or duration < best:
            best = duration

    print(f'best of {args.best_of}: {best:.3f}')
    return best >= args.cutoff


if __name__ == '__main__':
    raise SystemExit(main())

the nice thing about this script is it lets me set thresholds and find each regression here

here's some thresholds as well as their commits:

• 90ms threshold, 98ms best of 5: e2fb9f2
• 120ms threshold, 163ms best of 5: aab37b6

these seem to be the two most impactful changes to startup time

ideas

optimizing this is potentially a little annoying, but I think the best thing that could be done here is to lazily compile those regular expressions -- perhaps by moving the regex object to a property? or lazily constructing the Regex instance ? or if python3.7+ is targetted it could use module-level __getattr__ to get some further wins