Does/could this support snapshot+restore?

[tekknolagi]

KVM looks like it supports snapshot and restore so I am wondering if there are any design decisions in tinykvm that would preclude such a feature.

[fwsGonzo]

Are you talking about per-request isolation now? If so yes, that's already supported. See the VMOD or the CLI.

If you are talking about storing a VM to disk and restoring it later, or somewhere else, then there's considerations: Does your guest program use files, sockets etc.? Anything that has state that could change between snapshot/restore would obviously make it unusable.

Live migration is another matter, but perhaps not suitable for TinyKVM. Better suited for full-system virtualization.

Personally, I would just implement serialize/deserialize inside guest programs which reduces the stored state down to the absolute minimum and forces the program to be aware. This is already the case in the VMOD, as it supports live-updates (hot reload, if you will) by serializing data into a newer version of the program. See: https://github.com/varnish/libvmod-tinykvm/blob/main/docs/program.md#live-updating

[tekknolagi]

Whoa, kvmserver looks brand new. Neat.

Anything that has state that could change between snapshot/restore would obviously make it unusable.

Yeah, there would be a decent chunk of state but it could be managed in userspace given the right hook(s) from KVM. Like a callback to restore the state on restore. One can imagine, for example, a Python application:

def before_checkpoint():
    reset_sockets()
    close_files()
    tinykvm.set_wakeup_hook(after_checkpoint)

def after_checkpoint():
    connect_to_db()
    # ...

So the program would have to have some awareness for things that interface with the outside world, but I would rather not handle all of this stuff manually with loaded .so, JITed code, etc.

[fwsGonzo]

In the VMOD (which is what I maintain), this is indeed the case that you get to handle live updates in a similar fashion to checkpoint/restore. It's not the real deal, as it's intended for a transaction between an old and a newly updated program. But with everything already implemented for live updates, I think the path forward for something like that is very short. I'm guessing you want to bring something up/down fast.

Because the VMOD has tight integration with the programs it's running, it's the best candidate for such a thing. There's an expectation of callbacks already. The VMOD can run unmodified programs using FFI. KVM server (by lrowe) runs unmodified programs with no FFI, and so you can't call functions.

[tekknolagi]

For my ideal use case it doesn't have to be too fast but it would need to support a VM being brought up on machine A (say, a continuous integration box), quiesced to some image format, transferred to machine B, and then brought up again.

[tekknolagi]

I will take a look at VMOD when I get a moment -- thank you.

[fwsGonzo]

I think that perhaps merits a specialized program that embeds TinyKVM (similar to the VMOD and the CLI) but does exactly what you want. Seems doable though! There aren't any good APIs for dumping the VM to a file and restoring it yet, but that's not too much of a problem to implement. I might make an attempt some time in the autumn if I feel like it.

[lrowe]

For my ideal use case it doesn't have to be too fast but it would need to support a VM being brought up on machine A (say, a continuous integration box), quiesced to some image format, transferred to machine B, and then brought up again.

This is definitely something I would like to see for KVM Server since the exact same JIT-warmed image could be pushed to multiple servers and developers could use it for debugging.

To speed things up on first run I think we could record pages read and written by the forked vm while serving the warmup url so the appropriate sections of the image can be madvised in one go and avoid page faulting in serial on first serve.

[fwsGonzo]

Initial measurements look very promising:

Program 'test.com' is loaded (local, not cached, vm=32 ephemeral-kwm, nodes=1, storage=no, huge=0/0, ready=0.40ms)
Program 'deno' is loaded (local, not cached, vm=32 ephemeral-kwm, nodes=1, storage=no, huge=0/0, ready=0.55ms)

Versus before:

Program 'deno' is loaded (local, not cached, vm=32 ephemeral-kwm, nodes=1, storage=no, huge=0/0, ready=124.20ms, warmup=1ms)

[tekknolagi]

Out of curiosity, have you done more work here? This looks very interesting.

[lrowe]

There has been progress here and there are now snapshot and snaprun sub commands in the CLI.

There’s still some quality of life improvements to make as currently the snapshots are sparse files the size of virtual memory which are a pain with work with in docker containers. At some point I’d like to defragment them so the non sparse blocks are together at the beginning of the file.

[fwsGonzo]

It's about time for an update methinks. Currently we are experimenting with reordering VM snapshots according to the actual order pages are needed for fast sequential NVMe during loading. We are seeing 750us (0.75ms) end-to-end from terminal to completed request with page caches. And if we drop caches, it turns into 2100us (2.1ms) end-to-end. This is for a full unmodified Deno hello world service.

I think that's pretty good, and very likely some kind of record. For a small custom program it can be as low as 170us with caches and 600us cold. Starting a new service and handling a HTTP request in 170us is quite something.