It can be done only by deriving from the ShapeNode class. You can override AddHandlesToCanvas() to create custom handles. The control expects the handles to be Shape-derived objects, and you should add them to Diagram.DocumentPlane.Children.

Additionally, you could set the Tag of the handle's Shape object to an instance of the AdjustmentHandle class, and the built-in hit-testing should work automatically then. Otherwise you will have to also override the HitTestHandle method.

Finally, you will need to override the UpdateDrag method to update the node when the user drags a custom handle. You can reuse the built-in logic for move and resize by replacing ist.AdjustmentHandle with one that refers to the respective index of the built-in handles, and then calling base.UpdateDrag(ist).

You can lock the built-in handles by setting the EnabledHandles property. To disable custom handles, override the AllowDrag method.

I hope that helps,
Stoyan