Compiling CPython for Windows with Debugging Symbols

Compiling CPython for Windows with Debugging Symbols

Play this article

This post has been ported from my old self-hosted blog. The original post was published on 21/11/2022.

Consider this post as a personal note or a handy cheat sheet. Once you've fetched the source code from GitHub, running the build.bat batch script with your desired options is all you need. Just make sure you have Visual Studio C++ installed on your system.

PS C:\Users\tahai\Code\cpython\PCbuild> .\build.bat -t Build -p x64 -c Debug
Using py -3.10 (found 3.10 with py.exe)
Fetching external libraries...
bzip2-1.0.8 already exists, skipping.
sqlite-3.39.4.0 already exists, skipping.
xz-5.2.5 already exists, skipping.
zlib-1.2.13 already exists, skipping.
Fetching external binaries...
libffi-3.4.3 already exists, skipping.
openssl-bin-1.1.1q already exists, skipping.
tcltk-8.6.12.1 already exists, skipping.
Finished.
Using "C:\Program Files (x86)\Microsoft Visual Studio\2019\Community\MSBuild\Current\Bin\msbuild.exe" (found in the Visual Studio installation)
Using py -3.10 (found 3.10 with py.exe)
...
  _lzmamodule.c
     Creating library C:\Users\tahai\Code\cpython\PCbuild\amd64\_uuid_d.lib and object C:\Users\tahai\Code\cpython\PCbuild\amd64\_uuid_d.exp
  connection.c
  cursor.c
  microprotocols.c
  module.c
  prepare_protocol.c
  row.c
  statement.c
  util.c
  blob.c
  _uuid.vcxproj -> C:\Users\tahai\Code\cpython\PCbuild\amd64\_uuid_d.pyd
     Creating library C:\Users\tahai\Code\cpython\PCbuild\amd64\_lzma_d.lib and object C:\Users\tahai\Code\cpython\PCbuild\amd64\_lzma_d.exp
  applink.c
     Creating library C:\Users\tahai\Code\cpython\PCbuild\amd64\_sqlite3_d.lib and object C:\Users\tahai\Code\cpython\PCbuild\amd64\_sqlite3_d.exp
  _lzma.vcxproj -> C:\Users\tahai\Code\cpython\PCbuild\amd64\_lzma_d.pyd
  _sqlite3.vcxproj -> C:\Users\tahai\Code\cpython\PCbuild\amd64\_sqlite3_d.pyd
     Creating library C:\Users\tahai\Code\cpython\PCbuild\amd64\_ssl_d.lib and object C:\Users\tahai\Code\cpython\PCbuild\amd64\_ssl_d.exp
  _ssl.vcxproj -> C:\Users\tahai\Code\cpython\PCbuild\amd64\_ssl_d.pyd
  python.c
  python.vcxproj -> C:\Users\tahai\Code\cpython\PCbuild\amd64\python_d.exe
  C:\Windows\SYSTEM32\ucrtbased.dll is version 10.0.18362.1
  Regenerate test_frozenmain.h
  Programs\test_frozenmain.h written
  Wrote C:\Users\tahai\Code\cpython\PCbuild\amd64\LICENSE.txt
  WinMain.c
  pythonw.vcxproj -> C:\Users\tahai\Code\cpython\PCbuild\amd64\pythonw_d.exe

Build succeeded.
    0 Warning(s)
    0 Error(s)

Time Elapsed 00:00:29.54
PS C:\Users\tahai\Code\cpython\PCbuild>

The build process went smoothly without any errors. You can find the Python binaries and debugging symbols located at PCbuild\amd64, as shown below:

PS C:\Users\tahai\Code\cpython\PCbuild\amd64> dir *.pdb
    Directory: C:\Users\tahai\Code\cpython\PCbuild\amd64
Mode                 LastWriteTime         Length Name
----                 -------------         ------ ----
-a----        21/11/2022     07:09       11816960 libcrypto-1_1.pdb
-a----        21/11/2022     07:10         176128 liblzma_d.pdb
-a----        21/11/2022     07:09        3002368 libssl-1_1.pdb
-a----        21/11/2022     07:10        1150976 pyexpat_d.pdb
-a----        21/11/2022     07:10        1241088 pyshellext_d.pdb
-a----        21/11/2022     07:10       12275712 python312_d.pdb
-a----        21/11/2022     07:10         143360 python3_d.pdb
-a----        21/11/2022     07:10         790528 pythonw_d.pdb
-a----        21/11/2022     07:10         782336 python_d.pdb
-a----        21/11/2022     07:10        6991872 pyw_d.pdb
-a----        21/11/2022     07:10        6967296 py_d.pdb
-a----        21/11/2022     07:10         724992 select_d.pdb
-a----        21/11/2022     07:10        3166208 sqlite3_d.pdb
-a----        21/11/2022     07:10         757760 unicodedata_d.pdb
-a----        21/11/2022     07:10        7139328 venvlauncher_d.pdb
-a----        21/11/2022     07:10        7172096 venvwlauncher_d.pdb
-a----        21/11/2022     07:10         749568 winsound_d.pdb
-a----        21/11/2022     07:10         913408 _asyncio_d.pdb
-a----        21/11/2022     07:10         880640 _bz2_d.pdb
-a----        21/11/2022     07:10        1216512 _ctypes_d.pdb
-a----        21/11/2022     07:10         790528 _ctypes_test_d.pdb
-a----        21/11/2022     07:10        1339392 _decimal_d.pdb
-a----        21/11/2022     07:10        1224704 _elementtree_d.pdb
-a----        21/11/2022     07:10         815104 _hashlib_d.pdb
-a----        21/11/2022     07:10        1216512 _lzma_d.pdb
-a----        21/11/2022     07:10         815104 _msi_d.pdb
-a----        21/11/2022     07:10         765952 _multiprocessing_d.pdb
-a----        21/11/2022     07:10         962560 _overlapped_d.pdb
-a----        21/11/2022     07:10         708608 _queue_d.pdb
-a----        21/11/2022     07:10        1011712 _socket_d.pdb
-a----        21/11/2022     07:10        1036288 _sqlite3_d.pdb
-a----        21/11/2022     07:10        1290240 _ssl_d.pdb
-a----        21/11/2022     07:10         757760 _testbuffer_d.pdb
-a----        21/11/2022     07:10        1282048 _testcapi_d.pdb
-a----        21/11/2022     07:10         684032 _testconsole_d.pdb
-a----        21/11/2022     07:10        1003520 _testembed_d.pdb
-a----        21/11/2022     07:10         667648 _testimportmultiple_d.pdb
-a----        21/11/2022     07:10         815104 _testinternalcapi_d.pdb
-a----        21/11/2022     07:10         724992 _testmultiphase_d.pdb
-a----        21/11/2022     07:10         503808 _testsinglephase_d.pdb
-a----        21/11/2022     07:10         913408 _tkinter_d.pdb
-a----        21/11/2022     07:10         724992 _uuid_d.pdb
-a----        21/11/2022     07:10        1216512 _wmi_d.pdb
-a----        21/11/2022     07:10         847872 _zoneinfo_d.pdb
PS C:\Users\tahai\Code\cpython\PCbuild\amd64>

Fantastic! Now it's time to verify that everything is functioning as expected. We'll launch Windbg and attach it to python_d.exe as shown below.

Attach WinDBG to python_d.exe

Please note that test.py simply contains a os.system() call, which we can trace within the debugger to identify the CPython code implementation of that API:

import os
os.system("whoami")

After encountering the first break instruction exception, we can observe that python312_d.dll has been loaded:

ModLoad: 00007ff7`63eb0000 00007ff7`63ed6000   python_d.exe
ModLoad: 00007ff9`848d0000 00007ff9`84ac8000   ntdll.dll
ModLoad: 00007ff9`82aa0000 00007ff9`82b5f000   C:\Windows\System32\KERNEL32.DLL
ModLoad: 00007ff9`82020000 00007ff9`822f2000   C:\Windows\System32\KERNELBASE.dll
ModLoad: 00007ff9`33700000 00007ff9`33726000   C:\Windows\SYSTEM32\VCRUNTIME140D.dll
ModLoad: 00007ff8`f8260000 00007ff8`f8423000   C:\Windows\SYSTEM32\ucrtbased.dll
ModLoad: 00007ff8`e3980000 00007ff8`e450c000   C:\Users\tahai\Code\cpython\PCbuild\amd64\python312_d.dll
ModLoad: 00007ff9`83860000 00007ff9`838cb000   C:\Windows\System32\WS2_32.dll
ModLoad: 00007ff9`78150000 00007ff9`7815a000   C:\Windows\SYSTEM32\VERSION.dll
ModLoad: 00007ff9`838d0000 00007ff9`839f5000   C:\Windows\System32\RPCRT4.dll
ModLoad: 00007ff9`834d0000 00007ff9`8356e000   C:\Windows\System32\msvcrt.dll
ModLoad: 00007ff9`823b0000 00007ff9`823d7000   C:\Windows\System32\bcrypt.dll
ModLoad: 00007ff9`847c0000 00007ff9`8486e000   C:\Windows\System32\ADVAPI32.dll
ModLoad: 00007ff9`84330000 00007ff9`843cc000   C:\Windows\System32\sechost.dll
(4808.556c): Break instruction exception - code 80000003 (first chance)
ntdll!LdrpDoDebuggerBreak+0x30:
00007ff9`849a0950 cc              int     3

This module exports CPython APIs. By using the examine command in Windbg against python312_d.dll, we can identify the implementation of os.system():

0:000> x python312_d!*os*system
*** WARNING: Unable to verify checksum for C:\Users\tahai\Code\cpython\PCbuild\amd64\python312_d.dll
00007ff8`e3ace3c0 python312_d!os_system (struct _object *, struct _object **, int64, struct _object *)

Perfect! We've discovered that os_system() is the C implementation of the Python API os.system(). Since we have symbols available, the debugger displays the function prototype. Now, the final step is to set a breakpoint on that API and let the debugger resume execution.

When the breakpoint is hit, a new Windbg panel will open, displaying the source code for os_system(). Let the game begin!

os_system source code in WinDBG

Did you find this article valuable?

Support Taha Draidia by becoming a sponsor. Any amount is appreciated!