[CI] Added System Verilog Regression Tests to GitHub Runners #2885

AlexandreSinger · 2025-02-07T16:07:16Z

The system verilog regression tests were put into the currently not-working nightly tests. Since they are so small, putting them in the GitHub runners tests instead so they do not regress.

The strong parmys test also has system verilog tests (I think); however, that test may be too large for the GitHub runners. We may consider moving that test to the SAVI workflows in the future.

AlexandreSinger · 2025-02-07T20:14:00Z

@amirarjmand93 The System Verilog regression tests do not appear to be working. There seems to be a build error with the -DYOSYS_F4PGA_PLUGINS=ON CMAKE option:

Have you seen this error before and know how to fix it?

amirarjmand93 · 2025-02-07T20:55:37Z

#2841 (comment)

I ran this test locally and the result was okay.

have you tried "-DYOSYS_F4PGA_PLUGINS=ON" ? I had changed to "-DSYNLIG_SYSTEMVERILOG=ON" in nightly_testl

AlexandreSinger · 2025-02-07T21:03:44Z

#2841 (comment)

I ran this test locally and the result was okay.

have you tried "-DYOSYS_F4PGA_PLUGINS=ON" ? I had changed to "-DSYNLIG_SYSTEMVERILOG=ON" in nightly_testl

What is the difference between -DYOSYS_F4PGA_PLUGINS=ON and -DSYNLIG_SYSTEMVERILOG=ON? Currently I am matching what is written in the NightlyTests:

But the NightlyTests are never being run; so I am moving it to our GitHub runners. Which of the two arguments should I give to this test? Both?

amirarjmand93 · 2025-02-07T21:10:08Z

Before Synlig , we had used F4PGA pluggin and -DYOSYS_F4PGA_PLUGINS=ON was the command to make vtr with system verilog.
in Synlig, i changed -DYOSYS_F4PGA_PLUGINS=ON to -DSYNLIG_SYSTEMVERILOG=ON. so just this new cmake command should work and make vtr + synlig.

AlexandreSinger · 2025-02-07T21:18:55Z

Before Synlig , we had used F4PGA pluggin and -DYOSYS_F4PGA_PLUGINS=ON was the command to make vtr with system verilog. in Synlig, i changed -DYOSYS_F4PGA_PLUGINS=ON to -DSYNLIG_SYSTEMVERILOG=ON. so just this new cmake command should work and make vtr + synlig.

The test I added is using -DSYNLIG_SYSTEMVERILOG=ON, everything else is using the default libraries and build commands for Ubuntu 24.04. Is it possible that you have something locally that is not in this test?

amirarjmand93 · 2025-02-07T21:30:22Z

git submodule init
git submodule update --recursive

I think adding this to the YML might solve the problem. I usually have tried them before making and testing a VTR with Synlig.

AlexandreSinger · 2025-02-07T21:44:51Z

git submodule init
git submodule update --recursive
I think adding this to the YML might solve the problem. I usually have tried them before making and testing a VTR with Synlig.

The --recursive is interesting. We do not do that by default in the CI, we normally just do a regular submodule update. I have pushed a revision which does the submodule update recursively. Hopefully that fixes it!

amirarjmand93 · 2025-02-07T22:36:32Z

let's add commands under run:

run: |
       git submodule sync --recursive  # Sync submodules
       git submodule update --init --recursive  # Ensure they are initialized & updated'
.
.
.

do you think it might help?

The system verilog regression tests were put into the currently not-working nightly tests. Since they are so small, putting them in the GitHub runners tests instead so they do not regress. The strong parmys test also has system verilog tests (I think); however, that test may be too large for the GitHub runners. We may consider moving that test to the SAVI workflows in the future.

AlexandreSinger · 2025-02-07T22:45:32Z

git submodule update --init --recursive # Ensure they are initialized & updated'

Lets give it a shot, but why is this necessary?

amirarjmand93 · 2025-02-08T16:05:24Z

I think this Building discrepancy might arise from some library dependency, a path conflicting with CI or a race condition between Cloning and Updating modules. but I checked and it seems no new system or Python packages are required for Synlig. (you can also check https://github.com/chipsalliance/synlig)
Also, I rebuilt on local env and no errors appeared.
make CMAKE_PARAMS="-DCMAKE_BUILD_TYPE=Release -G 'Unix Makefiles' -Werror=dev -DSYNLIG_SYSTEMVERILOG=ON" -j16

Can I manipulate this branch and run some tests?

AlexandreSinger · 2025-02-08T16:37:12Z

@amirarjmand93 Feel free to modify the branch anyway you like! I recommend just deleting the other tests so that just the SystemVerilog test is active if you will use the CI to test (just to save yourself time waiting for the other known working tests).

If you think it is a race condition in the build script (which honestly could be the problem!), you can make the build script single threaded by setting the NUM_PROC to 1 here:

amirarjmand93 · 2025-02-08T17:13:40Z

Thank you,
Yes usually I disable nightly tests but in this test, I think working tests should be removed from this list, right?

AlexandreSinger · 2025-02-08T17:24:21Z

Thank you, Yes usually I disable nightly tests but in this test, I think working tests should be removed from this list, right?

No, thats a different list just for the coverity scan test. To turn off the other jobs you should be able to just delete the jobs from the workflow:

vtr-verilog-to-routing/.github/workflows/test.yml

Lines 36 to 581 in f41ea8c

    
           jobs: 
        
             Build: 
        
               name: 'B: Building VtR' 
        
               runs-on: ubuntu-24.04 
        
               strategy: 
        
                 fail-fast: false 
        
                 matrix: 
        
                   include: 
        
                   - { build_type: 'release', verbose: '0' } 
        
                   - { build_type: 'debug',   verbose: '0' } 
        
                   - { build_type: 'debug',   verbose: '1' } 
        
               steps: 
        
               - uses: actions/setup-python@v5 
        
                 with: 
        
                   python-version: 3.12.3 
        
               - uses: actions/checkout@v4 
        
                 with: 
        
                   submodules: 'true' 
        
               - name: Get number of CPU cores 
        
                 uses: SimenB/github-actions-cpu-cores@v2 
        
                 id: cpu-cores 
        
               - name: Install dependencies 
        
                 run: ./.github/scripts/install_dependencies.sh 
        
               - uses: hendrikmuhs/ccache-action@v1.2 
        
               - name: Test 
        
                 env: 
        
                   BUILD_TYPE: ${{ matrix.build_type }} 
        
                   NUM_PROC: ${{ steps.cpu-cores.outputs.count }} 
        
                 run: | 
        
                   export PATH="/usr/lib/ccache:/usr/local/opt/ccache/libexec:$PATH" 
        
                   ./.github/scripts/build.sh VERBOSE=${{ matrix.verbose }} 
        
             Format: 
        
               runs-on: ubuntu-24.04 
        
               strategy: 
        
                 fail-fast: false 
        
                 matrix: 
        
                   include: 
        
                   - { name: 'C/C++',       script: 'check-format.sh'    } 
        
                   - { name: 'Python',      script: 'check-format-py.sh' } 
        
                   - { name: 'Python Lint', script: 'pylint_check.py'    } 
        
               name: 'F: ${{ matrix.name }}' 
        
               steps: 
        
               # TODO: This should be on the same version of Python as would be found on 
        
               #       Ubuntu 24.04 (3.12.3); however that version has some linting errors. 
        
               - uses: actions/setup-python@v5 
        
                 with: 
        
                   python-version: 3.10.10 
        
               - uses: actions/checkout@v4 
        
               - name: Install dependencies 
        
                 run: ./.github/scripts/install_dependencies.sh 
        
               - name: Test 
        
                 run: ./dev/${{ matrix.script }} 
        
             UnitTests: 
        
               name: 'U: C++ Unit Tests' 
        
               runs-on: ubuntu-24.04 
        
               steps: 
        
               - uses: actions/setup-python@v5 
        
                 with: 
        
                   python-version: 3.12.3 
        
               - uses: actions/checkout@v4 
        
                 with: 
        
                   submodules: 'true' 
        
               - name: Get number of CPU cores 
        
                 uses: SimenB/github-actions-cpu-cores@v2 
        
                 id: cpu-cores 
        
               - name: Install dependencies 
        
                 run: ./.github/scripts/install_dependencies.sh 
        
               - name: Test 
        
                 env: 
        
                   CMAKE_PARAMS: "-DVTR_ASSERT_LEVEL=3 -DWITH_BLIFEXPLORER=on" 
        
                   NUM_PROC: ${{ steps.cpu-cores.outputs.count }} 
        
                 run: ./.github/scripts/unittest.sh 
        
             Warnings: 
        
               name: 'W: Check Compilation Warnings' 
        
               runs-on: ubuntu-24.04 
        
               steps: 
        
               - uses: actions/setup-python@v5 
        
                 with: 
        
                   python-version: 3.12.3 
        
               - uses: actions/checkout@v4 
        
                 with: 
        
                   submodules: 'true' 
        
               - name: Get number of CPU cores 
        
                 uses: SimenB/github-actions-cpu-cores@v2 
        
                 id: cpu-cores 
        
               - name: Install dependencies 
        
                 run: ./.github/scripts/install_dependencies.sh 
        
               - uses: hendrikmuhs/ccache-action@v1.2 
        
               - name: Test 
        
                 env: 
        
                   #In order to get compilation warnings produced per source file, we must do a non-IPO build 
        
                   #We also turn warnings into errors for this target by doing a strict compile 
        
                   CMAKE_PARAMS: "-DVTR_ASSERT_LEVEL=3 -DWITH_BLIFEXPLORER=on -DVTR_ENABLE_STRICT_COMPILE=on -DVTR_IPO_BUILD=off" 
        
                   NUM_PROC: ${{ steps.cpu-cores.outputs.count }} 
        
                 run: | 
        
                   export PATH="/usr/lib/ccache:/usr/local/opt/ccache/libexec:$PATH" 
        
                   ./.github/scripts/build.sh 
        
             Regression: 
        
               runs-on: ubuntu-24.04 
        
               strategy: 
        
                 fail-fast: false 
        
                 matrix: 
        
                   include: [ 
        
                     { 
        
                       name: 'Basic', 
        
                       params: '-DCMAKE_COMPILE_WARNING_AS_ERROR=on -DVTR_IPO_BUILD=off -DVTR_ASSERT_LEVEL=3 -DWITH_BLIFEXPLORER=on', 
        
                       suite: 'vtr_reg_basic', 
        
                       extra_pkgs: "" 
        
                     }, 
        
                     { 
        
                       name: 'Basic with highest assertion level', 
        
                       params: '-DCMAKE_COMPILE_WARNING_AS_ERROR=on -DVTR_IPO_BUILD=off -DVTR_ASSERT_LEVEL=4 -DWITH_BLIFEXPLORER=on', 
        
                       suite: 'vtr_reg_basic', 
        
                       extra_pkgs: "" 
        
                     }, 
        
                     { 
        
                       name: 'Basic_odin', 
        
                       params: '-DCMAKE_COMPILE_WARNING_AS_ERROR=on -DVTR_IPO_BUILD=off -DVTR_ASSERT_LEVEL=3 -DWITH_BLIFEXPLORER=on -DWITH_PARMYS=OFF -DWITH_ODIN=on', 
        
                       suite: 'vtr_reg_basic_odin', 
        
                       extra_pkgs: "" 
        
                     }, 
        
                     { 
        
                       name: 'Basic with NO_GRAPHICS', 
        
                       params: '-DCMAKE_COMPILE_WARNING_AS_ERROR=on -DVTR_IPO_BUILD=off -DVTR_ASSERT_LEVEL=3 -DWITH_BLIFEXPLORER=on -DVPR_USE_EZGL=off', 
        
                       suite: 'vtr_reg_basic', 
        
                       extra_pkgs: "" 
        
                     }, 
        
                     { 
        
                       name: 'Basic with NO_SERVER', 
        
                       params: '-DVTR_ASSERT_LEVEL=3 -DWITH_BLIFEXPLORER=on -DVPR_USE_EZGL=on -DVPR_USE_SERVER=off', 
        
                       suite: 'vtr_reg_basic', 
        
                       extra_pkgs: "" 
        
                     }, 
        
                     { 
        
                       name: 'Basic with CAPNPROTO disabled', 
        
                       params: '-DCMAKE_COMPILE_WARNING_AS_ERROR=on -DVTR_IPO_BUILD=off -DVTR_ASSERT_LEVEL=3 -DWITH_BLIFEXPLORER=on -DVTR_ENABLE_CAPNPROTO=off', 
        
                       suite: 'vtr_reg_basic', 
        
                       extra_pkgs: "" 
        
                     }, 
        
                     { 
        
                       name: 'Basic with VTR_ENABLE_DEBUG_LOGGING', 
        
                       params: '-DCMAKE_COMPILE_WARNING_AS_ERROR=on -DVTR_IPO_BUILD=off -DVTR_ASSERT_LEVEL=3 -DWITH_BLIFEXPLORER=on -DVTR_ENABLE_DEBUG_LOGGING=on', 
        
                       suite: 'vtr_reg_basic', 
        
                       extra_pkgs: "" 
        
                     }, 
        
                     { 
        
                       name: 'Basic_odin with VTR_ENABLE_DEBUG_LOGGING', 
        
                       params: '-DCMAKE_COMPILE_WARNING_AS_ERROR=on -DVTR_IPO_BUILD=off -DVTR_ASSERT_LEVEL=3 -DWITH_BLIFEXPLORER=on -DVTR_ENABLE_DEBUG_LOGGING=on -DWITH_PARMYS=OFF -DWITH_ODIN=on', 
        
                       suite: 'vtr_reg_basic_odin', 
        
                       extra_pkgs: "" 
        
                     }, 
        
                     { 
        
                       name: 'Strong', 
        
                       params: '-DCMAKE_COMPILE_WARNING_AS_ERROR=on -DVTR_IPO_BUILD=off -DVTR_ASSERT_LEVEL=3 -DWITH_BLIFEXPLORER=on', 
        
                       suite: 'vtr_reg_strong', 
        
                       extra_pkgs: "libeigen3-dev" 
        
                     }, 
        
                     { 
        
                       name: 'Strong_odin', 
        
                       params: '-DCMAKE_COMPILE_WARNING_AS_ERROR=on -DVTR_ASSERT_LEVEL=3 -DWITH_BLIFEXPLORER=on -DWITH_PARMYS=OFF -DWITH_ODIN=on', 
        
                       suite: 'vtr_reg_strong_odin', 
        
                       extra_pkgs: "" 
        
                     }, 
        
                     { 
        
                       name: 'Valgrind Memory', 
        
                       params: '-DCMAKE_COMPILE_WARNING_AS_ERROR=on -DVTR_IPO_BUILD=off -DVTR_ASSERT_LEVEL=3 -DWITH_BLIFEXPLORER=on -DWITH_ODIN=on', 
        
                       suite: 'vtr_reg_valgrind_small', 
        
                       extra_pkgs: "" 
        
                     } 
        
                   ] 
        
               name: 'R: ${{ matrix.name }}' 
        
               steps: 
        
               - uses: actions/setup-python@v5 
        
                 with: 
        
                   python-version: 3.12.3 
        
               - uses: actions/checkout@v4 
        
                 with: 
        
                   submodules: 'true' 
        
               - name: Get number of CPU cores 
        
                 uses: SimenB/github-actions-cpu-cores@v2 
        
                 id: cpu-cores 
        
               - name: Install dependencies 
        
                 run: ./.github/scripts/install_dependencies.sh 
        
               - name: Install external libraries 
        
                 run: sudo apt install -y ${{ matrix.extra_pkgs }} 
        
                 if: ${{ matrix.extra_pkgs }} 
        
               - uses: hendrikmuhs/ccache-action@v1.2 
        
               - name: Test 
        
                 env: 
        
                   CMAKE_PARAMS: ${{ matrix.params }} 
        
                   NUM_PROC: ${{ steps.cpu-cores.outputs.count }} 
        
                 run: | 
        
                   export PATH="/usr/lib/ccache:/usr/local/opt/ccache/libexec:$PATH" 
        
                   ./.github/scripts/build.sh 
        
                   ./run_reg_test.py ${{ matrix.suite }} -show_failures -j${{ steps.cpu-cores.outputs.count}} 
        
               - name: Upload regression run files 
        
                 if: ${{ !cancelled() }} 
        
                 uses: actions/upload-artifact@v4 
        
                 with: 
        
                   name: ${{matrix.name}}_run_files 
        
                   path: | 
        
                     vtr_flow/**/*.out 
        
                     # vtr_flow/**/*.blif  # Removed since it was taking too much space and was hardly used. 
        
                     vtr_flow/**/*.p 
        
                     vtr_flow/**/*.net 
        
                     vtr_flow/**/*.r 
        
               - name: Upload regression results 
        
                 if: ${{ !cancelled() }} 
        
                 uses: actions/upload-artifact@v4 
        
                 with: 
        
                   name: ${{matrix.name}}_results 
        
                   path: | 
        
                     vtr_flow/**/*.log 
        
                     vtr_flow/**/parse_results*.txt 
        
             Sanitized: 
        
               runs-on: ubuntu-24.04 
        
               strategy: 
        
                 fail-fast: false 
        
                 matrix: 
        
                   include: [ 
        
                     { 
        
                       name: 'Basic', 
        
                       params: '-DVTR_ASSERT_LEVEL=3 -DVTR_ENABLE_SANITIZE=on -DVTR_IPO_BUILD=off -DWITH_BLIFEXPLORER=on -DWITH_PARMYS=OFF -DWITH_ODIN=on', 
        
                       suite: 'vtr_reg_basic_odin' 
        
                     } 
        
                     #- { name: 'Strong', suite: 'vtr_reg_strong' } # SKIP Too long to run on GitHub Actions (max 6h) 
        
                   ] 
        
               name: 'S: ${{ matrix.name }}' 
        
               steps: 
        
               - uses: actions/setup-python@v5 
        
                 with: 
        
                   python-version: 3.12.3 
        
               - uses: actions/checkout@v4 
        
                 with: 
        
                   submodules: 'true' 
        
               - name: Get number of CPU cores 
        
                 uses: SimenB/github-actions-cpu-cores@v2 
        
                 id: cpu-cores 
        
               - name: Install dependencies 
        
                 run: ./.github/scripts/install_dependencies.sh 
        
               - uses: hendrikmuhs/ccache-action@v1.2 
        
               - name: Test 
        
                 env: 
        
                   CMAKE_PARAMS: ${{ matrix.params }} 
        
                   BUILD_TYPE: debug 
        
                   LSAN_OPTIONS: 'exitcode=42' #Use a non-standard exit code to ensure LSAN errors are detected 
        
                   # In Ubuntu 20240310.1.0, the entropy of ASLR has increased (28 -> 32). LLVM 14 in this 
        
                   # image is not compatible with this increased ASLR entropy. Apparently, memory sanitizer 
        
                   # depends on LLVM and all CI tests where VTR_ENABLE_SANITIZE is enabled fail. For a temporary 
        
                   # fix, we manually reduce the entropy. This quick fix should be removed in the future 
        
                   # when github deploys a more stable Ubuntu image. 
        
                   NUM_PROC: ${{ steps.cpu-cores.outputs.count }} 
        
                 run: | 
        
                   sudo sysctl -w vm.mmap_rnd_bits=28 
        
                   export PATH="/usr/lib/ccache:/usr/local/opt/ccache/libexec:$PATH" 
        
                   ./.github/scripts/build.sh 
        
                   # We skip QoR since we are only checking for errors in sanitizer runs 
        
                   ./run_reg_test.py ${{ matrix.suite }} -show_failures -j${{ steps.cpu-cores.outputs.count }} -skip_qor 
        
             Parmys: 
        
               name: 'Parmys Basic Test' 
        
               runs-on: ubuntu-24.04 
        
               steps: 
        
                 - uses: actions/setup-python@v5 
        
                   with: 
        
                     python-version: 3.12.3 
        
                 - uses: actions/checkout@v4 
        
                   with: 
        
                     submodules: 'true' 
        
                 - name: Get number of CPU cores 
        
                   uses: SimenB/github-actions-cpu-cores@v2 
        
                   id: cpu-cores 
        
                 - name: Install dependencies 
        
                   run: ./.github/scripts/install_dependencies.sh 
        
                 - uses: hendrikmuhs/ccache-action@v1.2 
        
                 - name: Test 
        
                   env: 
        
                     CMAKE_PARAMS: '-DVTR_IPO_BUILD=off' 
        
                     BUILD_TYPE: debug 
        
                     NUM_PROC: ${{ steps.cpu-cores.outputs.count }} 
        
                   run: | 
        
                     export PATH="/usr/lib/ccache:/usr/local/opt/ccache/libexec:$PATH" 
        
                     ./.github/scripts/build.sh 
        
                     ./run_reg_test.py parmys_reg_basic -show_failures -j${{ steps.cpu-cores.outputs.count }} 
        
             SystemVerilog: 
        
               name: 'System Verilog Regression' 
        
               runs-on: ubuntu-24.04 
        
               steps: 
        
               - uses: actions/setup-python@v5 
        
                 with: 
        
                   python-version: 3.12.3 
        
               # NOTE: This is using recursive on purpose. In order to use the System 
        
               #       Verilog feature, need submodules inside of a submodule. 
        
               - uses: actions/checkout@v4 
        
                 with: 
        
                   submodules: 'recursive' 
        
               - name: Get number of CPU cores 
        
                 uses: SimenB/github-actions-cpu-cores@v2 
        
                 id: cpu-cores 
        
               - name: Install dependencies 
        
                 run: ./.github/scripts/install_dependencies.sh 
        
               - uses: hendrikmuhs/ccache-action@v1.2 
        
               - name: Test 
        
                 env: 
        
                   CMAKE_PARAMS: '-DSYNLIG_SYSTEMVERILOG=ON' 
        
                   NUM_PROC: ${{ steps.cpu-cores.outputs.count }} 
        
                 run: | 
        
                   git submodule sync --recursive  # Sync submodules 
        
                   git submodule update --init --recursive  # Ensure they are initialized & updated' 
        
                   export PATH="/usr/lib/ccache:/usr/local/opt/ccache/libexec:$PATH" 
        
                   ./.github/scripts/build.sh 
        
                   ./run_reg_test.py vtr_reg_system_verilog -show_failures -j${{ steps.cpu-cores.outputs.count}} 
        
             ODINII: 
        
               name: 'ODIN-II Basic Test' 
        
               runs-on: ubuntu-24.04 
        
               steps: 
        
               - uses: actions/setup-python@v5 
        
                 with: 
        
                   python-version: 3.12.3 
        
               - uses: actions/checkout@v4 
        
                 with: 
        
                   submodules: 'true' 
        
               - name: Get number of CPU cores 
        
                 uses: SimenB/github-actions-cpu-cores@v2 
        
                 id: cpu-cores 
        
               - name: Install dependencies 
        
                 run: ./.github/scripts/install_dependencies.sh 
        
               - uses: hendrikmuhs/ccache-action@v1.2 
        
               - name: Test 
        
                 env: 
        
                   CMAKE_PARAMS: '-DVTR_ASSERT_LEVEL=3 -DVTR_ENABLE_SANITIZE=on -DVTR_IPO_BUILD=off -DWITH_BLIFEXPLORER=on -DWITH_PARMYS=OFF -DWITH_ODIN=on' 
        
                   BUILD_TYPE: debug 
        
                   NUM_PROC: ${{ steps.cpu-cores.outputs.count }} 
        
                 run: | 
        
                   sudo sysctl -w vm.mmap_rnd_bits=28 
        
                   export PATH="/usr/lib/ccache:/usr/local/opt/ccache/libexec:$PATH" 
        
                   ./.github/scripts/build.sh 
        
                   ./run_reg_test.py odin_reg_basic -show_failures -j${{ steps.cpu-cores.outputs.count }} 
        
             VQM2BLIF: 
        
               name: 'VQM2BLIF Basic Tests' 
        
               runs-on: ubuntu-24.04 
        
               steps: 
        
               - uses: actions/setup-python@v5 
        
                 with: 
        
                   python-version: 3.12.3 
        
               - uses: actions/checkout@v4 
        
                 with: 
        
                   submodules: 'true' 
        
               - name: Get number of CPU cores 
        
                 uses: SimenB/github-actions-cpu-cores@v2 
        
                 id: cpu-cores 
        
               - name: Install dependencies 
        
                 run: ./.github/scripts/install_dependencies.sh 
        
               - uses: hendrikmuhs/ccache-action@v1.2 
        
               - name: Test 
        
                 env: 
        
                   BUILD_TYPE: release 
        
                   NUM_PROC: ${{ steps.cpu-cores.outputs.count }} 
        
                 run: | 
        
                   export PATH="/usr/lib/ccache:/usr/local/opt/ccache/libexec:$PATH" 
        
                   ./.github/scripts/build.sh 
        
                   ./utils/vqm2blif/test/scripts/test_vqm2blif.sh 
        
             Compatibility: 
        
               runs-on: ubuntu-24.04 
        
               strategy: 
        
                 fail-fast: false 
        
                 matrix: 
        
                   include: 
        
                   - { name: 'GCC 11 (Ubuntu Noble - 24.04)',    eval: 'CC=gcc-11 && CXX=g++-11',           } 
        
                   - { name: 'GCC 12 (Ubuntu Noble - 24.04)',    eval: 'CC=gcc-12 && CXX=g++-12',         } 
        
                   - { name: 'GCC 14 (Ubuntu Noble - 24.04)',    eval: 'CC=gcc-14 && CXX=g++-14',         } 
        
                   - { name: 'Clang 16 (Ubuntu Noble - 24.04)',  eval: 'CC=clang-16 && CXX=clang++-16',   } 
        
                   - { name: 'Clang 17 (Ubuntu Noble - 24.04)',  eval: 'CC=clang-17 && CXX=clang++-17',   } 
        
                   - { name: 'Clang 18 (Ubuntu Noble - 24.04)',  eval: 'CC=clang-18 && CXX=clang++-18',   } 
        
                   # Note: We do not include GCC-13 since it is the default and is already tested. 
        
               name: 'B: ${{ matrix.name }}' 
        
               steps: 
        
               - uses: actions/setup-python@v5 
        
                 with: 
        
                   python-version: 3.12.3 
        
               - uses: actions/checkout@v4 
        
                 with: 
        
                   submodules: 'true' 
        
               - name: Get number of CPU cores 
        
                 uses: SimenB/github-actions-cpu-cores@v2 
        
                 id: cpu-cores 
        
               - name: Install dependencies 
        
                 run: ./.github/scripts/install_dependencies.sh 
        
               - uses: hendrikmuhs/ccache-action@v1.2 
        
               - name: Test 
        
                 env: 
        
                   CMAKE_PARAMS: "-DVTR_ASSERT_LEVEL=3 -DWITH_BLIFEXPLORER=on -DWITH_ODIN=on" 
        
                   MATRIX_EVAL: ${{ matrix.eval }} 
        
                   BUILD_TYPE: release 
        
                   NUM_PROC: ${{ steps.cpu-cores.outputs.count }} 
        
                 run: | 
        
                   export PATH="/usr/lib/ccache:/usr/local/opt/ccache/libexec:$PATH" 
        
                   ./.github/scripts/build.sh 
        
             JammyCompatibility: 
        
               name: 'Ubuntu Jammy - 22.04 Compatibility Test' 
        
               runs-on: ubuntu-22.04 
        
               steps: 
        
               - uses: actions/setup-python@v5 
        
                 with: 
        
                   python-version: 3.10.10 
        
               - uses: actions/checkout@v4 
        
                 with: 
        
                   submodules: 'true' 
        
               - name: Get number of CPU cores 
        
                 uses: SimenB/github-actions-cpu-cores@v2 
        
                 id: cpu-cores 
        
               - name: Install Dependencies 
        
                 run: ./.github/scripts/install_jammy_dependencies.sh 
        
               - uses: hendrikmuhs/ccache-action@v1.2 
        
               - name: Test 
        
                 env: 
        
                   CMAKE_PARAMS: "-DVTR_ASSERT_LEVEL=3 -DCMAKE_COMPILE_WARNING_AS_ERROR=on -DVTR_IPO_BUILD=off" 
        
                   MATRIX_EVAL: 'CC=gcc-11 && CXX=g++-11' 
        
                   BUILD_TYPE: release 
        
                   NUM_PROC: ${{ steps.cpu-cores.outputs.count }} 
        
                 run: | 
        
                   export PATH="/usr/lib/ccache:/usr/local/opt/ccache/libexec:$PATH" 
        
                   ./.github/scripts/build.sh 
        
             Coverity: 
        
               name: 'Coverity Scan' 
        
               needs: 
        
                 - Build 
        
                 - Format 
        
                 - UnitTests 
        
                 - Warnings 
        
                 - Regression 
        
                 - Sanitized 
        
                 - Parmys 
        
                 - SystemVerilog 
        
                 - ODINII 
        
                 - VQM2BLIF 
        
                 - Compatibility 
        
               runs-on: ubuntu-24.04 
        
               steps: 
        
               - uses: actions/setup-python@v5 
        
                 with: 
        
                   python-version: 3.12.3 
        
               - uses: actions/checkout@v4 
        
                 with: 
        
                   submodules: 'true' 
        
               - name: Get number of CPU cores 
        
                 uses: SimenB/github-actions-cpu-cores@v2 
        
                 id: cpu-cores 
        
               - name: Install dependencies 
        
                 run: ./.github/scripts/install_dependencies.sh 
        
               - uses: hendrikmuhs/ccache-action@v1.2 
        
               - name: Test 
        
                 env: 
        
                   CMAKE_PARAMS: '-DVTR_ASSERT_LEVEL=3 -DWITH_BLIFEXPLORER=on' 
        
                   NUM_PROC: ${{ steps.cpu-cores.outputs.count }} 
        
                   _COVERITY_URL: 'https://scan.coverity.com/download/linux64' 
        
                   _COVERITY_MD5: 'd0d7d7df9d6609e578f85096a755fb8f' 
        
                 run: | 
        
                   export PATH="/usr/lib/ccache:/usr/local/opt/ccache/libexec:$PATH" 
        
                   ./.github/scripts/build.sh

So basically just delete everything under "Build:", "Format:", "UnitTests:", etc. Leaving behind just the "SystemVerilog:" job.

The regular tests work slightly different from the Nightly Tests.

Change install path

paths corrected

check synlig path

update CPU core

amirarjmand93 · 2025-02-09T15:18:15Z

The previous error during Build time was resolved.
Currently, we have error on the Test. <ERROR: TCL interpreter returned an error: couldn't execute "/home/runner/work/vtr-verilog-to-routing/vtr-verilog-to-routing/build/bin/synlig_install/usr/local/bin/synlig": no such file or directory
probably Synlig has not been installed!
How can I get access to working files for this test(working VTR folder)? I tried Action pathway but I couldn't find it.
Also there is no problem with Building, Test and QOR on the local server for this test(SystemVerilog):

Run the default cmake --build

reduce build proc

Synlig Hardcoded Paths resolved

update synlig path

edited to relaunch the test

vaughnbetz · 2025-02-10T20:46:39Z

The path looks a bit funny ... verilog-to-routing is in it twice. Is that the issue (Cmake issue where the path isn't set properly?):

/home/runner/work/vtr-verilog-to-routing/vtr-verilog-to-routing/build/bin/synlig_install/usr/local/bin/synlig

amirarjmand93 · 2025-02-10T22:43:09Z

I believe there is a problem related to CMake in the Synlig project.

in CMake in EXTERNAL:

-When I use the explicit build and install command, it can create an execution file on the local PC (works fine) but cannot do it in CI ( I don't know why?).
BUILD_COMMAND ${MAKE_PROGRAM} -C ${SYNLIG_SOURCE_DIR} install DESTDIR=${CMAKE_BINARY_DIR}/synlig_install

-When I don't use any command and let CMake automatically build and install, it can't create executable files for synlig in PC or CI. (due to some issue in synlig's CMake file)

change CMAKE_BINARY_DIR to CMAKE_CURRENT_SOURCE_DIR. CI use separate build directions.

add debug message

amirarjmand93 · 2025-02-11T20:03:42Z

@AlexandreSinger, is there a time limit on the GitHub runner? sometimes, I receive a termination signal (due to the Synlig compilation workload), and if possible, could you please increase it?

[100%] Built target test_vpr
-- synlig build command succeeded. See also /home/runner/work/vtr-verilog-to-routing/vtr-verilog-to-routing/libs/EXTERNAL/synlig/src/synlig-stamp/synlig-build-*.log
[100%] Performing install step for 'synlig'
make[3]: *** [libs/EXTERNAL/CMakeFiles/synlig.dir/build.make:104: /home/runner/work/vtr-verilog-to-routing/vtr-verilog-to-routing/libs/EXTERNAL/synlig/src/synlig-stamp/synlig-install] Terminated
make[2]: *** [CMakeFiles/Makefile2:1515: libs/EXTERNAL/CMakeFiles/synlig.dir/all] Terminated
make[1]: *** [Makefile:146: all] Terminated
make: *** [Makefile:90: all] Terminated
Error: Process completed with exit code 143.

AlexandreSinger · 2025-02-11T20:12:09Z

@AlexandreSinger, is there a time limit on the GitHub runner? sometimes, I receive a termination signal (due to the Synlig compilation workload), and if possible, could you please increase it?

[100%] Built target test_vpr -- synlig build command succeeded. See also /home/runner/work/vtr-verilog-to-routing/vtr-verilog-to-routing/libs/EXTERNAL/synlig/src/synlig-stamp/synlig-build-*.log [100%] Performing install step for 'synlig' make[3]: *** [libs/EXTERNAL/CMakeFiles/synlig.dir/build.make:104: /home/runner/work/vtr-verilog-to-routing/vtr-verilog-to-routing/libs/EXTERNAL/synlig/src/synlig-stamp/synlig-install] Terminated make[2]: *** [CMakeFiles/Makefile2:1515: libs/EXTERNAL/CMakeFiles/synlig.dir/all] Terminated make[1]: *** [Makefile:146: all] Terminated make: *** [Makefile:90: all] Terminated Error: Process completed with exit code 143.

There is not a time limit on these tests as far as I am aware. What you are seeing can happen when you are using too many resources. For example too many threads, too much memory, or too much storage.

I have noticed that turning on System Verilog causes a lot of extra libraries to be downloaded and installed on the machine (which appears to cause the compile time to increase by 3x to 4x). Why are all of these libraries necessary? I am wondering if you are installing too much on the GitHub runners and it is causing issues.

limited cpu core

amirarjmand93 · 2025-06-05T17:14:54Z

Hello All,

Synlig has not been maintained for 6 months. Also, it has very dependencies(ext libs) and tests that need deep reformation to match our server building and resource usage restrictions (local pc for running tests has no problem, and all are green but take some hours).

We can switch to some alternatives :

sv2v is a tool that converts SystemVerilog (SV) → Verilog-2005.It parses full synthesizable SV, simplifies it, and rewrites it as plain Verilog. it seems sv2v used internally with yosys sv2v-yosys !?
Surelog is a SystemVerilog parser and elaborator that converts SV source files into a structured UHDM (Surelog had been used in Synlig). Yosys does not natively understand UHDM, but it can use it through the yosys-uhdm plugin( UHDM → RTLIL) , which allows SystemVerilog support via UHDM. The plugin has not been maintained for 2 years!
Slang and yosys-slang : Slang is a fast SystemVerilog parser supporting IEEE 1800-2017/2023. The Yosys-Slang plugin integrates it into Yosys via the read_slang command for direct SystemVerilog loading. Yosys read slang officialy
Siliconcompiler : in SiliconCompiler:

sv2v is used as a conversion step to make SystemVerilog designs compatible with Verilog-only tools.
Slang and SureLog are used as full-featured frontends for parsing and elaborating SystemVerilog in flows that require advanced language support.

I would appreciate it if @vaughnbetz, @hzeller and @AlexandreSinger could take a look and guide us to start new SV integration.

I think number 3, Yosys-Slang, is more rational due to its integration in Yosys and great maintenance.

Also yosys-slang can parse a number of open-source IPs, including:

Black Parrot

BSC Core Tile

OpenHW Group's CV32E40P

Ibex RISC-V Core

Supporters
The following organizations have supported the project and contributed to yosys-slang development:

Microelectronics Design Center of ETH Zürich

PULP platform

Silimate

YosysHQ

OpenTitan: Open source silicon root of trust

RSD RISC-V Out-of-Order Superscalar Processor

Thank you in advance, @loglav03, for handling this issue.

AlexandreSinger · 2025-06-05T19:50:39Z

@amirarjmand93 Thank you for the summary of alternatives! I am sure many people in the VTR community will have opinions about this choice! I recommend that we discuss this at the next VTR Industry Sync (Thursday Meeting).

petergrossmann21 · 2025-06-05T21:00:55Z

Some additional feedback:

Within Silicon Compiler, slang is becoming preferred over Surelog as slang appears to be in a better place with respect to its maintenance.
Silicon Compiler is a full-on flow manager, so adopting it just for its front end support doesn't make a lot of sense to me. Adopting it as an alternative to bespoke Python scripting for managing end-to-end execution of the VTR flow makes more sense if that's something people are interested in. Within that paradigm, solving the yosys/slang integration would be free(ish) since there are pre-defined flows for using them in concert.
The lowest-overhead solution is probably yosys + slang if the read_slang command support is well-maintained, and sv2v if it isn't. sv2v is expressly designed to produce Verilog as its output, so as long as it a) has good enough SystemVerilog syntax coverage and b) produces Verilog yosys can ingest without modification. At that point it's just one more step to execute at the beginning of the flow before running Yosys. I've tried this on a couple of non-VTR benchmarks in the past with success.
Due to high complexity, I would not recommend putting effort into a UHDM to RTLIL interface unless there is an interesting research problem there.

vaughnbetz · 2025-06-05T23:37:49Z

Thanks Amir for the analysis and Alex and Peter for your thoughts! Slang sounds like the front-runner and let's discuss next Thursday.

github-actions bot added the infra Project Infrastructure label Feb 7, 2025

AlexandreSinger force-pushed the feature-ci-system-verilog branch from d5249ad to 56e0afe Compare February 7, 2025 21:43

AlexandreSinger force-pushed the feature-ci-system-verilog branch from 56e0afe to f41ea8c Compare February 7, 2025 22:44

Update CMakeLists.txt

b2c2052

github-actions bot added lang-make CMake/Make code external_libs labels Feb 8, 2025

Update CMakeLists.txt

5d8eb3a

Change install path

amirarjmand93 force-pushed the feature-ci-system-verilog branch 2 times, most recently from 4cadfba to 5d8eb3a Compare February 8, 2025 22:54

amirarjmand93 and others added 6 commits February 8, 2025 19:12

Update golden results and CMakeLists

4a48f33

Update CMakeLists.txt

00595e2

Update CMakeLists.txt

385b981

Update CMakeLists.txt

dcc3258

paths corrected

Update test.yml

8a9125a

check synlig path

Update CMakeLists.txt

f45e2cd

update CPU core

amirarjmand93 added 4 commits February 9, 2025 19:10

Update CMakeLists.txt

ecd77cd

Run the default cmake --build

Update CMakeLists.txt

2033ae8

reduce build proc

Update CMakeLists.txt

0227c24

Synlig Hardcoded Paths resolved

Update paths.py

cabc382

update synlig path

github-actions bot added the lang-python Python code label Feb 10, 2025

amirarjmand93 added 2 commits February 10, 2025 20:16

Update CMakeLists.txt

72b816e

Update test.yml

c3305f5

edited to relaunch the test

amirarjmand93 added 6 commits February 11, 2025 02:30

Update paths.py

965472e

Update CMakeLists.txt

7e3089b

Update CMakeLists.txt

14ab8a2

change CMAKE_BINARY_DIR to CMAKE_CURRENT_SOURCE_DIR. CI use separate build directions.

Update paths.py

6f43262

Update CMakeLists.txt

5a43529

add debug message

Update CMakeLists.txt

3a46e76

Update CMakeLists.txt

d1ae4ab

limited cpu core

[CI] Added System Verilog Regression Tests to GitHub Runners #2885

Are you sure you want to change the base?

[CI] Added System Verilog Regression Tests to GitHub Runners #2885

Conversation

AlexandreSinger commented Feb 7, 2025

Uh oh!

AlexandreSinger commented Feb 7, 2025

Uh oh!

amirarjmand93 commented Feb 7, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

AlexandreSinger commented Feb 7, 2025

Uh oh!

amirarjmand93 commented Feb 7, 2025

Uh oh!

AlexandreSinger commented Feb 7, 2025

Uh oh!

amirarjmand93 commented Feb 7, 2025

Uh oh!

AlexandreSinger commented Feb 7, 2025

Uh oh!

amirarjmand93 commented Feb 7, 2025

Uh oh!

AlexandreSinger commented Feb 7, 2025

Uh oh!

amirarjmand93 commented Feb 8, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

AlexandreSinger commented Feb 8, 2025

Uh oh!

amirarjmand93 commented Feb 8, 2025

Uh oh!

AlexandreSinger commented Feb 8, 2025

Uh oh!

amirarjmand93 commented Feb 9, 2025

Uh oh!

vaughnbetz commented Feb 10, 2025

Uh oh!

amirarjmand93 commented Feb 10, 2025

Uh oh!

amirarjmand93 commented Feb 11, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

AlexandreSinger commented Feb 11, 2025

Uh oh!

amirarjmand93 commented Jun 5, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

AlexandreSinger commented Jun 5, 2025

Uh oh!

petergrossmann21 commented Jun 5, 2025

Uh oh!

vaughnbetz commented Jun 5, 2025

Uh oh!

Uh oh!

amirarjmand93 commented Feb 7, 2025 •

edited

Loading

amirarjmand93 commented Feb 8, 2025 •

edited

Loading

amirarjmand93 commented Feb 11, 2025 •

edited

Loading

amirarjmand93 commented Jun 5, 2025 •

edited

Loading