EDIT: of course this is what I get for not scrolling to the last page before spending the time to write this up. I'm not the first to identify the KAS/KIS method, and I'm not sure if the errors identified by KML would have resolved themselves or not. Cheers everyone.
Sorry if this is redundant with anything above. Hope this can help someone who's stuck, or help Squad better understand a way to solve this issue.
This bug ruined a big assembly for me a year or so back and did a bunch of manual sfs editing with these original instructions. It hit me again today and I actually discovered a pretty easy way that is mostly in game, but it does involve KAS/KIS if you don't use that and also uses KML. On that note, KML is my new favorite tool for bug squashing (Kraken-slaying?)
First, clarification on the DOCKEE vs DOCKER terms, mostly so I don't confuse myself:
The DOCKER is this vessel doing the docking
The DOCKEE is the vessel that is being docked with.
The DOCKEE is the vessel that when you dock, the new 'assembly' inherits the name. I believe KSP determines it by mass (bigger vessel is being docked with, smaller vessel is docking). I've seen some people report that this has to do with which vessel is being controlled during the dock, but through some experimentation I wasn't able to get the relation to swap. In practice I've had a station core with RCS pick up 'dumb' modules to reduce part count, and the name of the station remains unchanged. Really it doesn't matter for the scope of this.
Every time I've had this happen it's because I tried to dock two vessels that already had other vessels docked to them. As someone above mentioned, I think it has something to do with the game confusing itself trying to flip around who is the dockee/docker. So avoid that.
Quick description of my issue, arrows point to the DOCKEE.
[fuel module]--->[Station core]<---[shuttlecraft]
[greenhouse/lab module]--->[transfer stage]<---[greenhouse/lab module]
When trying to undock the transfer stage, the greenhouse parts would go flying off through the rest of the station. No collisions, no explosions. Looking a little closer here, the ship never undocked either; you can still right click on parts both on the transfer stage and the station.
Undock lab modules: The lab modules become separate vessels, but are still physically attached to the transfer stage.
Undock lab modules, undock transfer stage: the lab modules again become separate vessels, and the transfer stage breaks into pieces when undocked, even with all the ALT + F12 cheats on.
KML editing the stable .sfs: No errors thrown.
KML editing the .sfs after undocking: autorepair yielded various versions of "this is fixed but this isn't", which upon loading cause glorious explosions and cause an interesting failure mode similar to the apocolypse Kraken: When quickloading back to a stable save, there is no Sun/Mun and some of the textures are corrupted (station was transparent-ish?). Returning to the spacecenter shows that we are now in an eternal night.
Using KIS/KAS, flew up to the station. Used the power drill and the 'detach' function, I was able to separate the docking ports from each other and the transfer stage was completely free!
KML sanity check:
An examination of the .sfs threw a few errors with both docking ports that autorepair was unable to fix. In practice I believe this would have prevented the ports from being used again. But the autorepair tool gave me the exact part numbers so they were easy to find.
Navigate to both docking ports, 'ModuleDockingNode'.
Set dockUId to 0.
Set the state to READY
delete the 'DOCKED VESSEL' node. Yes, delete. Poof.
Save, load, and fly safe!