Here's the key idea that I want to draw from partial evaluation. You have an application, written in some language L. You also have an interpretive implementation of L, written in some lower-level language M. The application takes some input I and generates some output O. But we can think of the interpreter as a function of two inputs: I is actually also an input to the interpreter, as is the application specification.

Now since we're going to want to run the application many times on different inputs, we can treat the application as static (constant), and partially evaluate the interpreter with respect to the application. The result is this big blob of code, written in M, that implements the application, and includes a version of the interpreter specialized to the specific needs of the application. Loops are unrolled, dead code is eliminated. The resulting system has the same input and output as the original, but its performance is an order of magnitude better due to specialization stripping off the interpretive overhead.

And we can get extensibility, too. We can extend the implementation of L with some new functionality f', possibly extending L to L'. If our application then uses that functionality (depends on the new features of L'), then f' appears as part of the compiled blob of code.