OpenTitan support #1442
Description
Here is the branch with the current progress on OpenTitan support:
https://github.com/antmicro/symbiflow-arch-defs/tree/opentitan_earlgray
Currently, I'm struggling with the error in vpr_io_place.py:
Generating earlgray_nexys_video/artix7_200t-xc7a200t-virt-xc7a200t-test/top_io.place
Traceback (most recent call last):
File "/home/build/rwinkler/google-symbiflow-arch-defs/xc/common/utils/prjxray_create_ioplace.py", line 159, in <module>
main()
File "/home/build/rwinkler/google-symbiflow-arch-defs/xc/common/utils/prjxray_create_ioplace.py", line 73, in main
io_place.load_block_names_from_net_file(args.net)
File "/home/build/rwinkler/google-symbiflow-arch-defs/utils/vpr_io_place.py", line 60, in load_block_names_from_net_file
"//block[@instance='inpad[0]'] | //block[@instance='outpad[0]']"
File "src/lxml/etree.pyx", line 1581, in lxml.etree._Element.xpath
File "src/lxml/xpath.pxi", line 305, in lxml.etree.XPathElementEvaluator.__call__
File "src/lxml/xpath.pxi", line 225, in lxml.etree._XPathEvaluatorBase._handle_result
lxml.etree.XPathEvalError: Error in xpath expression
xc/xc7/tests/soc/earlgray/CMakeFiles/file_xc_xc7_tests_soc_earlgray_earlgray_nexys_video_artix7_200t-xc7a200t-virt-xc7a200t-test_top_io.place.dir/build.make:71: recipe for target 'xc/xc7/tests/soc/earlgray/earlgray_nexys_video/artix7_200t-xc7a200t-virt-xc7a200t-test/top_io.place' failed
make[3]: *** [xc/xc7/tests/soc/earlgray/earlgray_nexys_video/artix7_200t-xc7a200t-virt-xc7a200t-test/top_io.place] Error 1
make[3]: *** Deleting file 'xc/xc7/tests/soc/earlgray/earlgray_nexys_video/artix7_200t-xc7a200t-virt-xc7a200t-test/top_io.place'
CMakeFiles/Makefile2:222236: recipe for target 'xc/xc7/tests/soc/earlgray/CMakeFiles/file_xc_xc7_tests_soc_earlgray_earlgray_nexys_video_artix7_200t-xc7a200t-virt-xc7a200t-test_top_io.place.dir/all' failed
make[2]: *** [xc/xc7/tests/soc/earlgray/CMakeFiles/file_xc_xc7_tests_soc_earlgray_earlgray_nexys_video_artix7_200t-xc7a200t-virt-xc7a200t-test_top_io.place.dir/all] Error 2
CMakeFiles/Makefile2:222935: recipe for target 'xc/xc7/tests/soc/earlgray/CMakeFiles/earlgray_nexys_video_bit.dir/rule' failed
make[1]: *** [xc/xc7/tests/soc/earlgray/CMakeFiles/earlgray_nexys_video_bit.dir/rule] Error 2
Makefile:2933: recipe for target 'xc/xc7/tests/soc/earlgray/CMakeFiles/earlgray_nexys_video_bit.dir/rule' failed
make: *** [xc/xc7/tests/soc/earlgray/CMakeFiles/earlgray_nexys_video_bit.dir/rule] Error 2
However, the same XPath expression works for other designs like button or counter on nexys video. I think that the error is in the wrong .net or .eblif file itself. Maybe due to unsupported elements of the design or wrong import in previous importing scripts.
Metadata
Metadata
Assignees
Labels
No labels
Type
Projects
Milestone
Relationships
Development
No branches or pull requests
Participants
Activity
[-]OpenTitan support[/-][+][WIP] OpenTitan support[/+][-][WIP] OpenTitan support[/-][+]OpenTitan support[/+]mkurc-ant commentedon Apr 24, 2020
I've tried to lint the
top.netxml usingxmllintand it succeeded. Next, I've tried to query the XPath (part of it) and this is what i got:It seems that the
libxmlhas hard time using XPath for such a huge XML file (~330MB). The python wrapper simply misinterprets the error and throws thelxml.etree.XPathEvalErrorexception even though the XPath itself is correct.The
xmllintI used is built with:I need to double check that against the version of
libxmlthat we use in SymbiFlow.@rw1nkler For now try applying this patch to
utils/vpr_io_place.py, this one works by querying all blocks and filtering them in python:rw1nkler commentedon May 5, 2020
Utilization report
Here you can find vivado utilization report:
top_earlgrey_nexysvideo_utilization_placed.log
OpenTitan burndown list
Here is the short burndown list for OpenTitan:
.netfileIt will be good to optimize lookahead generation. Currently, it requires more than 10h to complete the computation. This slows down the working process since every change in architecture or in the tools requires lookahead recomputing.
Changes in the OpenTitan design
Here is the list of changes introduced for OpenTitan:
Due to the fact that tristate buffers are not inferred correctly when instantiated in submodules, the top module was changed. (Broken OBUFT inference in submodules YosysHQ/yosys#1737). Here you can find the changed file:
Because of bad RAM36 inference (Xilinx's RAMB36 inference YosysHQ/yosys#1748), the appropriate workaround was created. It consists of two files:
One
hmac.svfile aftersv2vconversion produced the following error in Yosys:ERROR: 2nd expression of generate for-loop is not constant!. Due to that, it was changed:Some parts of Verilog syntax are not supported by Yosys, due to this fact the following commands were used to all the files:
Soon I will update the issue with the diff file with all the introduced changes.
Current work
Currently, we are in the step of
.routefile generation. This step consumes a lot of RAM, and runners with 126Gb of memory were unable to handle it. After introducing the--congested_routing_iteration_threshold 0.08flag to the routing process it seems that the routing step does not exceed available resources.rw1nkler commentedon May 12, 2020
Currently, the OpenTitan design can be routed with the changes introduced in:
However, the routing process consumes about 90Gb of RAM.
The SymbiFlow failes on the frames step with the following error:
tcal-x commentedon May 29, 2020
I encountered the same error in libxml described above, "Error in xpath expression", at the same place in vpr_io_place.py. It was a large design -- a version of scalable_proc with N=100.
I ran into the same xml issue also in vpr_place_constraints.py at line 48:
for attr in net_root.xpath("//attribute[@name='LOC']"):
rw1nkler commentedon Jul 28, 2020
Routing issues
After reducing the design frequency to 25MHz, I was able to generate bitstream of EarlGrey design that works on hardware. I used old
symbiflow-arch-defsversion because the routing process on the current master fails due to huge RAM consumption that exceeds 128GB.The bitstream was successfully generated using the
symbiflow-arch-defsin version (f8aa52f) with conda VPR.I figured out that there are small differences between locally build VPR and the conda one, but the final routing parameters do not change dramatically - about
+2nsfor critical path delay. I guess that this is normal behavior.Here is the table with information about the router iterations:
I wanted to rebase the code on the current master but the routing parameters are much worse. Here is the routing table for the design rebased on (5267e78). The most significant change is the routing quality is visible in the critical path delay which increases by
~110ns. Additionally, the whole routing process takes ages and consumes ~300GB of RAM. I killed the process since the test was taking too much time.Now I am trying to find the commit that breaks the routing process. I'm using
git bisectfor that.Additional information
Placement problems
Rebasing on the (a1f0153) causes the problem with the design placement:
The mentioned commit contains a new Yosys version introduced in 206bd56. That is the only change in the flow since (5267e78) passes the placement step.
@acomodi told me that this should be resolved by #1612.
RAM usage
All VPR runs for EarlGrey design consumes a lot of RAM memory. Part of that is related to the size of the routing graph:
Nevertheless, a huge amount of RAM (~80GB) and time (~43min) is used for the
Clean circuitstep:This has a huge impact on the time required for the tests since the
clean circuitstep is executed in every VPR run:I believe that all four peaks came from the
clean circuitstep in VPR.rw1nkler commentedon Jul 28, 2020
FYI @litghost
rw1nkler commentedon Jul 28, 2020
The source code used for the bitstream generation is available in #1590.
Previous
symbiflow-arch-defsversions require also:and the CLK_HROW_BOT_R bits obtained in:
f4pga/prjxray#1381
Once the #1419 is accepted, all the required modifications for the EarlGrey bitstream generation will be on the master branch. The only problem now is the routing issue described above.
litghost commentedon Jul 28, 2020
I have an initial fix for the RAM explosion in verilog-to-routing/vtr-verilog-to-routing#1454 , which will hopefully be merged upstream. However, the RAM explosion usuallly accompanies an enormous runtime. Still working on that
litghost commentedon Jul 28, 2020
@rw1nkler If you run with #1612 rebased on master, does it complete?
rw1nkler commentedon Aug 11, 2020
Yes, It works after applying the mentioned changes. Also, the routing is successful.
I rebased your changes on (f0e7b42)
5 remaining items