A machine monitor, or machine language monitor, is a tool to examine memory, assemble and disassemble program code, and debug programs a the machine language level.
I don't think a ton of keywords are needed. Here's the list I can think off off the top of my head:
M [start] [length] - dump memory.
W or > addr data [data] ... write memory. M prints the format that > uses as input, making modifying memory simple.
A or . [addr] operator - assemble starting at the specified address.
F addr [length] data [data] - fill memory with byte or bytes specified.
C start1 start2 length - compare memory
T addr1 addr2 length - transfer (copy) memory. If addr2 is inside of addr1+length range, then copy operates backward so no data is lost
S, L, V start len filename - save, load, verify.
R - display registers in format A=xx X=xx, etc.
; - Update registers. Like >, the R command prefaces the register dump with ; to make updates simple.
@ - get disk status or issue disk command. eg: @CD /GAMES changes to GAMES directory on root of hard drive.
I used JMON as the starting point for a monitor for my homebrew 6502 projects. I included the assembler and disassembler--I usually have 32K of ROM (minus a page or 1K for memory mapped I/O), so even with a BASIC interpreter space typically isn't an issue. I also like to have a fairly robust monitor. I ended up with the following command set:
A - Assemble <address>
B - BASIC C - Copy <start> <end> <destination>
D - Disassemble <start>
F - Fill <start> <end> <value>
G - Go <address>
H - HEX Dump <start> <end> (note: dumps in Intel .hex format)
K - Checksum <start> <end>
L - Load Memory L <address> <data>...
M - Dump Memory <start>
N - Info (note: provides info such as processor type, memory size, I/O options, etc.)
O - Options (all upper case, force processor type, etc.)
P - Breakpoint P <n or ?> <address>
R - Registers
S - Search <start> <end> <data>...
= - Math = <address> +/- <address>
. - Trace
^ - Clr Screen
; - HEX Load (note: load Intel .hex format which can be generated by using H command)
? - Help
This has worked well for systems with 64K of address space. I'd have to think some about what would feel intuitive when it came to the 65816's address space.
Another logical starting point would be the WDC W65C256's embedded monitor. I believe the source code is out there. The '256 is basically a '816 core with some I/O and the built in ROM. The manual has the monitor command summary on page 82:
http://www.westerndesigncenter.com/Wdc/documentation/265monrom.pdf
Obviously to use that code wholesale would require the okay from WDC.
I like the ability to dump and load either Intel hex (in my monitor) or Moto S (in the WDC monitors) files.
Just my two cents worth, but happy to help where I can. Looks like an exciting project.
Thanks,
Jim