I just had to figure out how to do this so I figured a quick blog post is in order to save other people time in future.
If you ever need to use windbg to debug a SQL Server crash dump, or you want to capture call stacks using extended events (e.g. when debugging excessive spinlock contention), you’ll need the correct symbol file (sqlservr.pdb) to go with sqlservr.exe of the instance you’re interested in.
[Edit: May 2016] Download and install the Windows debugging tools from the Windows SDK here. This works on Windows Vista through 10 and Windows Server through 2016.
Now you’ll have a tool called symchk in the folder where windbg resides (for my laptop, “C:\Program Files (x86)\Windows Kits\10\Debuggers\x64”) – this is what will pull down sqlservr.pdb.
You need to point symchk at the executable you’re interested in, tell it where to put the sqlservr.pdb, and tell it the location of the Microsoft symbol server.
For me, the following worked:
C:\Users\Paul>cd C:\Program Files\Microsoft SQL Server\MSSQL12.SQL2014\MSSQL\Binn
C:\Program Files\Microsoft SQL Server\MSSQL12.SQL2014\MSSQL\Binn>“C:\Program Files (x86)\Windows Kits\10\Debuggers\x64\symchk” sqlservr.exe /s SRV*c:\symbols*http://msdl.microsoft.com/download/symbols
(Some people need to use https instead of http I’ve heard.)
Then go to the c:\symbols directory, and find the directory called sqlservr.pdb. It will have one or more sub-directories with GUID names, so pick the one with today’s date and then copy the sqlservr.pdb from that directory into the \Binn directory.
Again, the command string to use once you’re in the SQL Server Binn directory is:
“C:\Program Files (x86)\Windows Kits\10\Debuggers\x64\symchk“ sqlservr.exe /s SRV*c:\symbols*http://msdl.microsoft.com/download/symbols
If you get an error “The filename, directory name, or volume label syntax is incorrect.”, the copy-paste inserted weird characters for the double-quotes so delete and reinsert them. Also make sure there’s a space between /s and SRV.
You’ll also need to do this for sqlos.dll, plus on SQL Server 2012+ you’ll need to do it for sqlmin.dll, sqldk.dll, sqllang.dll, sqlboot.dll, and on SQL Server 2014+ you’ll need to do qds.dll and sqltses.dll too otherwise call stacks won’t resolve properly. I usually just replace sqlservr.exe in the command above with *.dll and then I’ve got all the symbol files possible, but you can save time by only doing it for the strictly necessary files.
When you get to the analysis phase, if you don’t have all the correct symbols, your call stack will look something like this (an example where sqlmin.pdb and sqllang.pdb are missing:
XeSosPkg::wait_info::Publish+138 [ @ 0+0x0 SOS_Task::PreWait+176 [ @ 0+0x0 Ordinal1132+9ab [ @ 0+0x0 Ordinal1391+b8e [ @ 0+0x0 Ordinal429+c4e [ @ 0+0x0 Ordinal319+966 [ @ 0+0x0 0x000007FEF20B04E2 0x000007FEF20B0D27 0x000007FEF20B1ACC 0x000007FEF12204C1 0x000007FEF123A54B 0x000007FEF1239C84 0x000007FEF125DC2F 0x000007FEF1254EC7 SOS_Task::Param::Execute+21e [ @ 0+0x0 SOS_Scheduler::RunTask+a8 [ @ 0+0x0 SOS_Scheduler::ProcessTasks+29a [ @ 0+0x0 SchedulerManager::WorkerEntryPoint+261 [ @ 0+0x0 SystemThread::RunWorker+8f [ @ 0+0x0 SystemThreadDispatcher::ProcessWorker+3c8 [ @ 0+0x0 SchedulerManager::ThreadEntryPoint+236 [ @ 0+0x0 BaseThreadInitThunk+d [ @ 0+0x0 RtlUserThreadStart+21 [ @ 0+0x0
Where it should resolve every frame in the call stack and look something like this (the same example with all pdbs in place):
XeSosPkg::wait_info::Publish+138 [ @ 0+0x0 SOS_Task::PreWait+176 [ @ 0+0x0 EventInternal::Wait+1e3 [ @ 0+0x0 FCB::SyncWrite+104 [ @ 0+0x0 DBMgr::CopyModel+fe [ @ 0+0x0 DBMgr::CreateAndFormatFiles+966 [ @ 0+0x0 CStmtCreateDB::CreateLocalDatabaseFragment+682 [ @ 0+0x0 DBDDLAgent::CreateDatabase+f7 [ @ 0+0x0 CStmtCreateDB::XretExecute+8fc [ @ 0+0x0 CMsqlExecContext::ExecuteStmts<1,1>+400 [ @ 0+0x0 CMsqlExecContext::FExecute+a33 [ @ 0+0x0 CSQLSource::Execute+866 [ @ 0+0x0 process_request+73c [ @ 0+0x0 process_commands+51c [ @ 0+0x0 SOS_Task::Param::Execute+21e [ @ 0+0x0 SOS_Scheduler::RunTask+a8 [ @ 0+0x0 SOS_Scheduler::ProcessTasks+29a [ @ 0+0x0 SchedulerManager::WorkerEntryPoint+261 [ @ 0+0x0 SystemThread::RunWorker+8f [ @ 0+0x0 SystemThreadDispatcher::ProcessWorker+3c8 [ @ 0+0x0 SchedulerManager::ThreadEntryPoint+236 [ @ 0+0x0 BaseThreadInitThunk+d [ @ 0+0x0 RtlUserThreadStart+21 [ @ 0+0x0
There’s also a video by Erin that shows the process to go through – see here.
Enjoy!
23 thoughts on “How to download a sqlservr.pdb symbol file”
That’s the tip of the month ! I wonder why the pdb file was not shipped anymore from yukon on, do you know the reason ? Thx
Nope – don’t know – would make sense to ship it rather than having to jump through hoops to get it.
I started using WinDbg for the first time ever this week and I stumbled upon this blog post.
I’m having trouble hammering my system to create a spinlock so that I get an extended event, so I can’t prove this myself, but if you were to set _NT_SYMBOL_PATH to
symsrv*symsrv.dll*c:symbols2*http://msdl.microsoft.com/download/symbolsfor the sql server process, would sql server be able to use the pdb files in the symbol server cache as opposed to the directory itself to resolve the stack trace for extended events?I’ve recently discovered that _NT_SYMBOL_PATH is respected by tools in the microsoft compiler toolchain like link.exe, lib.exe, editbin.exe and dumpbin.exe, so I would suspect sqlservr.exe might behave the same way. If it didn’t behave the same way, I’d certainly file a connect bug suggesting it did.
I don’t know Justin – ask Jonathan.
Even just having SQL Server sitting doing nothing, it’s taking spinlocks in the background. What event are you trying to hit? Check out my spinlocks category where I show how to catch SOS_SCHEDULER_YIELD in XEvents.
I’ll give that a whirl. I’ve never used extended events, and I’m more curious about how SQL, and other things use PDBs.
I have worked though a dozen or so posts relating to sqlservr.pdb availability for SQL 2008 R2 SP2 and SP3. I have been unable to download the sysmbolf files for for SQL 2008 R2 SP1. Can you provide any guidance? I always get the message
SYMCHK: sqlservr.exe FAILED – sqlservr.pdb mismatched or not found
SYMCHK: FAILED files = 1
SYMCHK: PASSED + IGNORED files = 0
no matter what I try.
Thanks
I can confirm the same issue, debug symbols for SQL Server 2008 R2 SP2 cannot be downloaded at the moment.
I have SQL Server 2008 R@ SP2 and I get;
SYMCHK: FAILED files = 0
SYMCHK: PASSED + IGNORED files = 0
Not sure what your problem is or how to debug it – it always works for me.
I get the same error as AA_Online and AFomchenko, with SQL 10.50.3720 (2008R2 Dev edition, 64bit):
C:\Program Files\Microsoft SQL Server\MSSQL10_50.MSSQLSERVER\MSSQL\Binn>”C:\Program Files\Debugging Tools for Windows (x64)\symchk.exe” sqlservr.exe /s SRV*c:\symbols*http://msdl.microsoft.com/download/symbols
SYMCHK: sqlservr.exe FAILED – sqlservr.pdb mismatched or not found
SYMCHK: FAILED files = 1
SYMCHK: PASSED + IGNORED files = 0
C:\Program Files\Microsoft SQL Server\MSSQL10_50.MSSQLSERVER\MSSQL\Binn>
I will install SP2 for SQL to get version 10.50.4000 and try again. Maybe pdb will be available for that build version.
Can these be installed on the server that we are going to troubleshoot itself?
Yes
I did this but I am not getting all the correct symbol files. I did sqlos.dll, *.dll, sqlservr.exe and this is the result I get for IO_COMPLETION. I am really frustrated trying to figure what is causing this wait. What am I missing?
0x0000000000976D94
0x0000000000093F5F
0x0000000000094195
0x00000000000929B6
0x000000000056CB6A
0x000000000056CA1F
0x000000000090F5A7
GetIUMSForMsxml+967a81 [ @ 0+0x0
0x000000000090F471
0x000000000056CC80
0x00000000000F3213
0x00000000000F396A
0x00000000000E4516
0x00000000000F0296
0x00000000000F039D
0x00000000000D5E4D
0x0000000000976D94
0x0000000000093F5F
0x0000000000094195
0x00000000000929B6
0x000000000056CB6A
0x000000000056CA1F
0x000000000056CBE0
0x000000000056CC8B
0x00000000000F3213
0x00000000000F396A
0x00000000000E4516
0x00000000000F0296
0x00000000000F039D
0x00000000000D5E4D
0x00000000000E5519
0x00000000000D5E4D
0x0000000000976D94
0x0000000000093F5F
0x0000000000094195
0x00000000000929B6
0x000000000056CB6A
0x000000000056CA1F
0x000000000056CBE0
0x000000000090F3E2
0x000000000056CC80
0x00000000000F3213
0x00000000000F396A
0x00000000000E4516
0x00000000000F0296
0x00000000000F039D
0x00000000000D5E4D
0x00000000000E5519
0x0000000000976D94
0x0000000000093F5F
0x0000000000094195
0x00000000000929B6
0x000000000056DB13
0x000000000056D8E7
0x00000000000F3293
0x00000000000F391A
0x00000000000E4531
0x00000000000E7860
0x00000000001569C7
0x00000000000E5435
0x00000000000E500B
0x000000000015CDE1
0x000000000015CC3A
_____SQL______Process______Available+682c59 [ @ 0+0x0
0x0000000000976D94
0x0000000000093F5F
0x0000000000094195
0x00000000000929B6
0x000000000090F8F5
0x000000000065012F
0x00000000000DA23B
0x00000000000DA107
0x00000000000D9C81
0x00000000000D9DC9
0x00000000000D9D42
0x00000000000E7860
0x00000000001569C7
0x00000000000E5435
0x00000000000E50D8
0x000000000015CDE1
Looks like you didn’t copy them into the correct directory, or didn’t turn on the trace flag. Worst case is you might need to bounce SQL Server. Which version are you using?
Is there a way to get the symbols from a machine that doesn’t have a SQL Server installed? My databases are behind a firewall which prevents an Internet connection.
Thanks, Nenad
I don’t know as I’ve never had to do that – Google is your friend here. Thanks
Please NEVER delete this post! I’ve referred to it now a couple times over the last few years. Thanks
No danger of that – I use it myself whenever I need to update symbol files.
Great post and great video! Is there a way to understand trace symbols now that I decoded ?
Thank you
Thanks. Quite a lot of the function names are understandable, especially in the Storage Engine portion of the code. If there’s a particular call stack you can’t work out, by all means shoot me an email. But apart from that, there aren’t any public repositories of what it all means.
Error: SYMCHK: sqlservr.exe FAILED – sqlservr.pdb mismatched or not found
Workaround for me (use https not http):
”C:\Program Files\Debugging Tools for Windows (x64)\symchk.exe” sqlservr.exe /s SRV*c:\symbols*httpS://msdl.microsoft.com/download/symbols
After running the command:
symchk sqlservr.exe /s SRV*C:\Symbols\*https://msdl.microsoft.com/download/symbols
I am getting only this:
SYMCHK: FAILED files = 0
SYMCHK: PASSED + IGNORED files = 0
And there is nothing under C:\Symbols directory
Nothing I can suggest except to make sure you’re in the binn directory that contains sqlservr.exe