Regarding mixing the layouts, CompositeLayout seems designed to apply the same layout for subgraphs indeed, but there is some built-in tree recognition and you can apply separate layouts by setting the SubtreeLayout and SubgraphLayout properties:
var l = new CompositeLayout();
l.PartitionMethod = PartitionMethod.Custom;
l.CustomPartition = new List<DiagramNodeCollection> { set1, set2 };
//l.CustomLayout = new LayoutSubgraph(ArrangeSub);
l.MasterLayout = new LayeredLayout { Orientation = MindFusion.Diagramming.Layout.Orientation.Horizontal };
l.SubgraphLayout = new CircularLayout();
l.SubtreeLayout = new TreeLayout { Type = TreeLayoutType.Radial, IgnoreLinkDirection = true };
l.Arrange(diagram);
Alternatively if using a custom layout delegate, you'd need some way to discover which set the provided items belong to:
private bool ArrangeSub(Diagram diagram, DiagramItemCollection items)
{
if (set1.Contains(items[0] as DiagramNode))
new CircularLayout().Arrange(diagram, items);
else
new TreeLayout { Type=TreeLayoutType.Radial, IgnoreLinkDirection = true }.Arrange(diagram, items);
return true;
}
Both methods create the image below with these sets:
set1 = new DiagramNodeCollection();
for (int i = 0; i < 10; i++)
set1.Add(diagram.Factory.CreateShapeNode(0, 0, 8, 8));
for (int i = 0; i < 9; i++)
for (int j = i + 1; j < 10; j++)
diagram.Factory.CreateDiagramLink(set1[i], set1[j]);
set2 = new DiagramNodeCollection();
for (int i = 0; i < 20; i++)
set2.Add(diagram.Factory.CreateShapeNode(0, 0, 8, 8));
for (int i = 1; i < 20; i++)
diagram.Factory.CreateDiagramLink(set2[i / 2], set2[i]);
diagram.Factory.CreateDiagramLink(set1[0], set2[0]);
I'm not sure what you mean by "classic mesh network with X nodes (circular arrangement)" though, how it should be different form CircularLayout?