dumping/profiling tf::DataPipeline example #599
Description
I am trying to understand how DataPipeline works exactly and was hoping that dumping the graph and/or profiling would help.
I am using the example in https://taskflow.github.io/taskflow/DataParallelPipeline.html
When I dump the graph, I get the picture also shown on that page with tasks "rt-0" to "rt-3", which is not helpful to understand what is actually executed. If I generate profile data I get:
[
{}
,{"executor":"1","data":[{"worker":13,"level":0,"data":[{"span":[304,3000457],"name":"rt-2","type":"static"},{"span":[6001163,10001414],"name":"rt-3","type":"static"},{"span":[10001859,14002142],"name":"rt-3","type":"static"},{"span":[14002801,18003053],"name":"rt-3","type":"static"},{"span":[18003797,22004197],"name":"rt-3","type":"static"}]},{"worker":15,"level":0,"data":[{"span":[328,3000492],"name":"rt-3","type":"static"},{"span":[5000936,9001253],"name":"rt-2","type":"static"},{"span":[9001738,13002007],"name":"rt-2","type":"static"},{"span":[13002642,17002958],"name":"rt-2","type":"static"},{"span":[17003666,21003893],"name":"rt-2","type":"static"}]},{"worker":16,"level":0,"data":[{"span":[240,246],"name":"cond","type":"condition"},{"span":[251,7000794],"name":"rt-0","type":"static"},{"span":[7001322,11001640],"name":"rt-0","type":"static"},{"span":[11001990,15002472],"name":"rt-0","type":"static"},{"span":[15002940,19003278],"name":"rt-0","type":"static"},{"span":[19004113,20004227],"name":"rt-0","type":"static"},{"span":[20004234,23004558],"name":"rt-1","type":"static"}]},{"worker":17,"level":0,"data":[{"span":[226,23004570],"name":"pipeline","type":"module"}]},{"worker":17,"level":1,"data":[{"span":[277,3000437],"name":"rt-1","type":"static"},{"span":[4000597,8001050],"name":"rt-1","type":"static"},{"span":[8001499,12001996],"name":"rt-1","type":"static"},{"span":[12002241,16002878],"name":"rt-1","type":"static"},{"span":[16003139,20003673],"name":"rt-1","type":"static"}]}]}
]
which is an invalid format according to https://taskflow.github.io/tfprof/
You can fix this by deleting {}, at the beginning. I assume this is a bug?
Finally, that is also not very helpful to understand what is happening exactly in this example. Is there any other way to inspect how the jobs in the pipeline are scheduled?
Thank you.