100% found this document useful (1 vote)
23 views

Full download Dynamic SQL: Applications, Performance, and Security in Microsoft SQL Server - Second Edition Edward Pollack pdf docx

Server

Uploaded by

achuriberiao
Copyright
© © All Rights Reserved
Available Formats
Download as PDF, TXT or read online on Scribd
100% found this document useful (1 vote)
23 views

Full download Dynamic SQL: Applications, Performance, and Security in Microsoft SQL Server - Second Edition Edward Pollack pdf docx

Server

Uploaded by

achuriberiao
Copyright
© © All Rights Reserved
Available Formats
Download as PDF, TXT or read online on Scribd
You are on page 1/ 55

Download Full Version ebook - Visit ebookmeta.

com

Dynamic SQL: Applications, Performance, and


Security in Microsoft SQL Server - Second Edition
Edward Pollack

https://ebookmeta.com/product/dynamic-sql-applications-
performance-and-security-in-microsoft-sql-server-second-
edition-edward-pollack/

OR CLICK HERE

DOWLOAD NOW

Discover More Ebook - Explore Now at ebookmeta.com


Instant digital products (PDF, ePub, MOBI) ready for you
Download now and discover formats that fit your needs...

Start reading on any device today!

Expert Performance Indexing in Azure SQL and SQL Server


2022 Fourth Edition Edward Pollack

https://ebookmeta.com/product/expert-performance-indexing-in-azure-
sql-and-sql-server-2022-fourth-edition-edward-pollack/

ebookmeta.com

Expert Performance Indexing in Azure SQL and SQL Server


2022, Fourth Edition: Toward Faster Results and Lower
Maintenance Both on Premises and in the Cloud Edward
Pollack
https://ebookmeta.com/product/expert-performance-indexing-in-azure-
sql-and-sql-server-2022-fourth-edition-toward-faster-results-and-
lower-maintenance-both-on-premises-and-in-the-cloud-edward-pollack/
ebookmeta.com

Pro SQL Server 2019 Wait Statistics: A Practical Guide to


Analyzing Performance in SQL Server - Second Edition
Enrico Van De Laar
https://ebookmeta.com/product/pro-sql-server-2019-wait-statistics-a-
practical-guide-to-analyzing-performance-in-sql-server-second-edition-
enrico-van-de-laar/
ebookmeta.com

Scholarly Research in Music Shared and Disciplinary


Specific Practices 2nd Edition Sang-Hie Lee

https://ebookmeta.com/product/scholarly-research-in-music-shared-and-
disciplinary-specific-practices-2nd-edition-sang-hie-lee/

ebookmeta.com
Scientific American Mind Scientific American

https://ebookmeta.com/product/scientific-american-mind-scientific-
american/

ebookmeta.com

Cost and Optimization in Government : An Introduction to


Cost Accounting, Operations Management, and Quality
Control 3rd Edition Aman Khan
https://ebookmeta.com/product/cost-and-optimization-in-government-an-
introduction-to-cost-accounting-operations-management-and-quality-
control-3rd-edition-aman-khan/
ebookmeta.com

Syntactic and Semantic Variation in Copular Sentences


Insights from Classical Hebrew 1st Edition Daniel J.
Wilson
https://ebookmeta.com/product/syntactic-and-semantic-variation-in-
copular-sentences-insights-from-classical-hebrew-1st-edition-daniel-j-
wilson/
ebookmeta.com

The Political Economy of Eastern Europe 30 years into the


Transition New Left Perspectives from the Region 1st
Edition Agnes Gagyi
https://ebookmeta.com/product/the-political-economy-of-eastern-
europe-30-years-into-the-transition-new-left-perspectives-from-the-
region-1st-edition-agnes-gagyi/
ebookmeta.com

The World of States, 2nd Edition John L. Campbell

https://ebookmeta.com/product/the-world-of-states-2nd-edition-john-l-
campbell/

ebookmeta.com
Nolo's Encyclopedia of Everyday Law: Answers to Your Most
Frequently Asked Legal Questions, 12th Edition The Editors
Of Nolo The Editors Of Nolo
https://ebookmeta.com/product/nolos-encyclopedia-of-everyday-law-
answers-to-your-most-frequently-asked-legal-questions-12th-edition-
the-editors-of-nolo-the-editors-of-nolo/
ebookmeta.com
Dynamic SQL
Applications, Performance, and Security
in Microsof t SQL Server

Second Edition

Edward Pollack
Dynamic SQL
Applications, Performance, and
Security in Microsoft SQL Server
Second Edition

Edward Pollack
Dynamic SQL: Applications, Performance, and Security in Microsoft SQL Server
Edward Pollack
Albany, NY, USA

ISBN-13 (pbk): 978-1-4842-4317-6 ISBN-13 (electronic): 978-1-4842-4318-3


https://doi.org/10.1007/978-1-4842-4318-3
Library of Congress Control Number: 2018967497

Copyright © 2019 by Edward Pollack


This work is subject to copyright. All rights are reserved by the Publisher, whether the whole or part of the
material is concerned, specifically the rights of translation, reprinting, reuse of illustrations, recitation,
broadcasting, reproduction on microfilms or in any other physical way, and transmission or information
storage and retrieval, electronic adaptation, computer software, or by similar or dissimilar methodology now
known or hereafter developed.
Trademarked names, logos, and images may appear in this book. Rather than use a trademark symbol
with every occurrence of a trademarked name, logo, or image we use the names, logos, and images only
in an editorial fashion and to the benefit of the trademark owner, with no intention of infringement of
the trademark.
The use in this publication of trade names, trademarks, service marks, and similar terms, even if they are not
identified as such, is not to be taken as an expression of opinion as to whether or not they are subject to
proprietary rights.
While the advice and information in this book are believed to be true and accurate at the date of publication,
neither the authors nor the editors nor the publisher can accept any legal responsibility for any errors or
omissions that may be made. The publisher makes no warranty, express or implied, with respect to the
material contained herein.
Managing Director, Apress Media LLC: Welmoed Spahr
Acquisitions Editor: Jonathan Gennick
Development Editor: Laura Berendson
Coordinating Editor: Jill Balzano
Cover image designed by Freepik (www.freepik.com)
Distributed to the book trade worldwide by Springer Science+Business Media New York, 233 Spring Street,
6th Floor, New York, NY 10013. Phone 1-800-SPRINGER, fax (201) 348-4505, e-mail orders-ny@springer-
sbm.com, or visit www.springeronline.com. Apress Media, LLC is a California LLC and the sole member
(owner) is Springer Science + Business Media Finance Inc (SSBM Finance Inc). SSBM Finance Inc is a
Delaware corporation.
For information on translations, please e-mail rights@apress.com, or visit www.apress.com/
rights-permissions.
Apress titles may be purchased in bulk for academic, corporate, or promotional use. eBook versions and
licenses are also available for most titles. For more information, reference our Print and eBook Bulk Sales
web page at www.apress.com/bulk-sales.
Any source code or other supplementary material referenced by the author in this book is available to
readers on GitHub via the book’s product page, located at www.apress.com/978-1-4842-4317-6. For more
detailed information, please visit http://www.apress.com/source-code.
Printed on acid-free paper
To Theresa, Nolan, and Oliver: the best players 2-4 (aka: family)
I could ever have asked for!
Table of Contents
About the Author��������������������������������������������������������������������������������������������������� xiii

About the Technical Reviewer���������������������������������������������������������������������������������xv


Acknowledgments�������������������������������������������������������������������������������������������������xvii

Introduction������������������������������������������������������������������������������������������������������������xix

Chapter 1: What Is Dynamic SQL?���������������������������������������������������������������������������� 1


Understanding Dynamic SQL�������������������������������������������������������������������������������������������������������� 1
A Simple Example�������������������������������������������������������������������������������������������������������������������� 1
The EXEC Statement���������������������������������������������������������������������������������������������������������������� 2
Data Type to Use���������������������������������������������������������������������������������������������������������������������� 3
Dynamic Execution Process���������������������������������������������������������������������������������������������������������� 3
Dynamic SQL in Action������������������������������������������������������������������������������������������������������������������ 4
Advantages of Dynamic SQL��������������������������������������������������������������������������������������������������������� 7
Optional or Customized Search Criteria����������������������������������������������������������������������������������� 7
Customizable Everything��������������������������������������������������������������������������������������������������������� 7
Optimize SQL Performance����������������������������������������������������������������������������������������������������� 8
Generate Large Amounts of T-SQL or Text, Fast!��������������������������������������������������������������������� 8
Execute SQL Statements on Other Servers or Databases������������������������������������������������������� 9
Do the Impossible!������������������������������������������������������������������������������������������������������������������� 9
Dynamic SQL Considerations������������������������������������������������������������������������������������������������������ 10
Apostrophes Can Break Strings��������������������������������������������������������������������������������������������� 10
NULL Can Break Strings�������������������������������������������������������������������������������������������������������� 10
Difficult to Read and Debug��������������������������������������������������������������������������������������������������� 11
Permissions and Scope Are Different������������������������������������������������������������������������������������ 11
Dynamic SQL Cannot be used in Functions��������������������������������������������������������������������������� 12

v
Table of Contents

Dynamic SQL Style���������������������������������������������������������������������������������������������������������������������� 12


Document Thoroughly����������������������������������������������������������������������������������������������������������� 12
Debugging Dynamic SQL������������������������������������������������������������������������������������������������������� 16
Write Dynamic SQL Just Like Standard T-SQL����������������������������������������������������������������������� 19
String Sizes and Truncation��������������������������������������������������������������������������������������������������� 20
Management Studio Text Display������������������������������������������������������������������������������������������ 21
Sp_executesql����������������������������������������������������������������������������������������������������������������������� 22
Building Strings via Concatenation��������������������������������������������������������������������������������������� 23
Notes on Apostrophes����������������������������������������������������������������������������������������������������������� 29
Conclusion���������������������������������������������������������������������������������������������������������������������������������� 30

Chapter 2: Protecting Against SQL Injection���������������������������������������������������������� 31


What Is SQL Injection?���������������������������������������������������������������������������������������������������������������� 31
Cleansing Inputs������������������������������������������������������������������������������������������������������������������������� 39
Parameterizing Dynamic SQL����������������������������������������������������������������������������������������������������� 42
Schema Name and Square Brackets������������������������������������������������������������������������������������������ 48
Effective Spacing������������������������������������������������������������������������������������������������������������������������ 50
Properly Type Inputs�������������������������������������������������������������������������������������������������������������������� 50
Blind SQL Injection���������������������������������������������������������������������������������������������������������������������� 51
Detection and Prevention������������������������������������������������������������������������������������������������������������ 53
Security Testing��������������������������������������������������������������������������������������������������������������������� 53
Scanning of Application Traffic���������������������������������������������������������������������������������������������� 54
Log Review���������������������������������������������������������������������������������������������������������������������������� 54
Code Review�������������������������������������������������������������������������������������������������������������������������� 55
Software Patching����������������������������������������������������������������������������������������������������������������� 55
Limit URL Length������������������������������������������������������������������������������������������������������������������� 56
Use Views and/or Masking for Sensitive Data����������������������������������������������������������������������� 57
Vendor Software�������������������������������������������������������������������������������������������������������������������� 57
Login Pages��������������������������������������������������������������������������������������������������������������������������� 58
Conclusion���������������������������������������������������������������������������������������������������������������������������������� 60

vi
Table of Contents

Chapter 3: Large Scale Searching�������������������������������������������������������������������������� 61


Why Use Dynamic Searches?����������������������������������������������������������������������������������������������������� 61
Custom Search Grids������������������������������������������������������������������������������������������������������������������ 67
Search Grid Considerations��������������������������������������������������������������������������������������������������������� 79
Disallow Blank Searches������������������������������������������������������������������������������������������������������� 80
Data Paging��������������������������������������������������������������������������������������������������������������������������� 80
Conditional Paging����������������������������������������������������������������������������������������������������������������� 83
Search Limitations����������������������������������������������������������������������������������������������������������������� 86
Input-Based Search��������������������������������������������������������������������������������������������������������������� 87
Result Row Count������������������������������������������������������������������������������������������������������������������ 93
Additional Filtering Considerations��������������������������������������������������������������������������������������������� 99
Alternatives������������������������������������������������������������������������������������������������������������������������������� 101
Conclusion�������������������������������������������������������������������������������������������������������������������������������� 102

Chapter 4: Permissions and Security������������������������������������������������������������������� 103


The Principle of Least Privilege������������������������������������������������������������������������������������������������� 103
Granular Permissions vs. Role Permissions������������������������������������������������������������������������������ 106
Dynamic SQL and Ownership Chaining������������������������������������������������������������������������������������� 107
Changing Security Context On-the-Fly�������������������������������������������������������������������������������������� 112
Where Do Security Disasters Come From?������������������������������������������������������������������������������� 120
Users, Passwords, and Inconvenience�������������������������������������������������������������������������������������� 124
Dynamic SQL Maintenance������������������������������������������������������������������������������������������������������� 126
Cleaning House������������������������������������������������������������������������������������������������������������������������� 138
Login and User Usage��������������������������������������������������������������������������������������������������������������� 140
Auditing Users and Logins�������������������������������������������������������������������������������������������������������� 142
Memory Consumption��������������������������������������������������������������������������������������������������������������� 146
Row Level Security������������������������������������������������������������������������������������������������������������������� 153
Signing Stored Procedures������������������������������������������������������������������������������������������������������� 158
Conclusion�������������������������������������������������������������������������������������������������������������������������������� 163

vii
Table of Contents

Chapter 5: Managing Scope���������������������������������������������������������������������������������� 165


What Is Scope?������������������������������������������������������������������������������������������������������������������������� 165
Why Is Scope Important?���������������������������������������������������������������������������������������������������������� 168
Managing Scope in Dynamic SQL��������������������������������������������������������������������������������������������� 169
Using OUTPUT in Dynamic SQL������������������������������������������������������������������������������������������������� 170
Table Variables and Temporary Tables�������������������������������������������������������������������������������������� 175
Table Variables�������������������������������������������������������������������������������������������������������������������� 175
Temporary Tables���������������������������������������������������������������������������������������������������������������� 177
Global Temporary Tables������������������������������������������������������������������������������������������������������ 181
Using Permanent Tables for Temporary Storage����������������������������������������������������������������������� 185
Output Data Directly to a Table from Dynamic SQL������������������������������������������������������������������� 187
Conclusion�������������������������������������������������������������������������������������������������������������������������������� 190
Cleanup������������������������������������������������������������������������������������������������������������������������������������� 191

Chapter 6: Performance Optimization������������������������������������������������������������������ 193


Query Execution������������������������������������������������������������������������������������������������������������������������ 193
Parsing�������������������������������������������������������������������������������������������������������������������������������� 193
Binding�������������������������������������������������������������������������������������������������������������������������������� 193
Optimization������������������������������������������������������������������������������������������������������������������������ 194
Execution����������������������������������������������������������������������������������������������������������������������������� 194
Optimization Tools��������������������������������������������������������������������������������������������������������������������� 194
Query Execution Plan���������������������������������������������������������������������������������������������������������� 195
STATISTICS IO���������������������������������������������������������������������������������������������������������������������� 198
STATISTICS TIME������������������������������������������������������������������������������������������������������������������ 199
Use All of These Tools!��������������������������������������������������������������������������������������������������������� 200
Dynamic SQL vs. Standard SQL������������������������������������������������������������������������������������������������ 200
Query Parsing and Binding�������������������������������������������������������������������������������������������������� 200
Execution Plan Caching������������������������������������������������������������������������������������������������������� 202
Simplifying Queries������������������������������������������������������������������������������������������������������������� 206
Paging Performance����������������������������������������������������������������������������������������������������������������� 215
Filtered Indexes������������������������������������������������������������������������������������������������������������������������ 228

viii
Table of Contents

Cardinality��������������������������������������������������������������������������������������������������������������������������������� 233
Statistics������������������������������������������������������������������������������������������������������������������������������ 234
Sys.dm_db_stats_properties���������������������������������������������������������������������������������������������� 244
Trace Flag 2371������������������������������������������������������������������������������������������������������������������� 245
Back to Dynamic SQL���������������������������������������������������������������������������������������������������������� 246
Query Hints������������������������������������������������������������������������������������������������������������������������������� 247
Conclusion�������������������������������������������������������������������������������������������������������������������������������� 254
Cleanup������������������������������������������������������������������������������������������������������������������������������������� 254

Chapter 7: Scalable Dynamic Lists����������������������������������������������������������������������� 257


What Is a Dynamic List?������������������������������������������������������������������������������������������������������������ 257
Using XML to Create a Dynamic List����������������������������������������������������������������������������������������� 260
Set-Based String Building��������������������������������������������������������������������������������������������������������� 264
Revisiting Security�������������������������������������������������������������������������������������������������������������������� 267
STRING_AGG����������������������������������������������������������������������������������������������������������������������������� 274
Conclusion�������������������������������������������������������������������������������������������������������������������������������� 278

Chapter 8: Parameter Sniffing������������������������������������������������������������������������������ 279


What Is Parameter Sniffing?����������������������������������������������������������������������������������������������������� 279
Parameter Sniffing Examples��������������������������������������������������������������������������������������������������� 282
Identifying Parameter Sniffing�������������������������������������������������������������������������������������������������� 297
Design Considerations�������������������������������������������������������������������������������������������������������������� 298
Query Execution Details������������������������������������������������������������������������������������������������������ 299
The Red Herrings����������������������������������������������������������������������������������������������������������������� 302
Parameter Values���������������������������������������������������������������������������������������������������������������� 307
Local Variables�������������������������������������������������������������������������������������������������������������������� 308
Forcing Cardinalities to the Optimizer��������������������������������������������������������������������������������� 316
Dynamic SQL����������������������������������������������������������������������������������������������������������������������� 320
Trace Flag 4136������������������������������������������������������������������������������������������������������������������� 322
Fix Bad Business Logic�������������������������������������������������������������������������������������������������������� 322
Conclusion�������������������������������������������������������������������������������������������������������������������������������� 325
Cleanup������������������������������������������������������������������������������������������������������������������������������������� 325

ix
Table of Contents

Chapter 9: Dynamic Pivot and Unpivot����������������������������������������������������������������� 327


PIVOT����������������������������������������������������������������������������������������������������������������������������������������� 327
UNPIVOT������������������������������������������������������������������������������������������������������������������������������������ 337
Additional Examples������������������������������������������������������������������������������������������������������������������ 343
Multiple PIVOT Operators���������������������������������������������������������������������������������������������������������� 347
Multiple UNPIVOT Operators������������������������������������������������������������������������������������������������������ 350
Classification Using PIVOT and CASE���������������������������������������������������������������������������������������� 358
Conclusion�������������������������������������������������������������������������������������������������������������������������������� 360

Chapter 10: Solving Common Problems��������������������������������������������������������������� 361


Collation Conflicts��������������������������������������������������������������������������������������������������������������������� 361
The Problem������������������������������������������������������������������������������������������������������������������������ 362
The Solution������������������������������������������������������������������������������������������������������������������������� 369
Organizing and Archiving Data�������������������������������������������������������������������������������������������������� 372
The Problem������������������������������������������������������������������������������������������������������������������������ 372
The Solution������������������������������������������������������������������������������������������������������������������������� 374
Customized Database Objects�������������������������������������������������������������������������������������������������� 380
The Problem������������������������������������������������������������������������������������������������������������������������ 380
The Solution������������������������������������������������������������������������������������������������������������������������� 380
A Note on System Tables���������������������������������������������������������������������������������������������������������� 388
Conclusion�������������������������������������������������������������������������������������������������������������������������������� 389

Chapter 11: Applications of Dynamic SQL����������������������������������������������������������� 391


Database Backups�������������������������������������������������������������������������������������������������������������������� 391
Saving Generated Scripts���������������������������������������������������������������������������������������������������������� 399
Saving Scripts to a Table����������������������������������������������������������������������������������������������������������� 399
Executing TSQL on Other Servers��������������������������������������������������������������������������������������������� 409
Generating Schema from Metadata������������������������������������������������������������������������������������������ 412
Building a Solution�������������������������������������������������������������������������������������������������������������������� 412
Conclusion�������������������������������������������������������������������������������������������������������������������������������� 420

x
Table of Contents

Chapter 12: Index Usage and Maintenance���������������������������������������������������������� 421


Index Defragmentation�������������������������������������������������������������������������������������������������������������� 421
Index Rebuild����������������������������������������������������������������������������������������������������������������������� 423
Index Reorganization����������������������������������������������������������������������������������������������������������� 424
Creating an Index Maintenance Solution����������������������������������������������������������������������������� 424
Index Usage Statistics��������������������������������������������������������������������������������������������������������������� 430
Missing Index Statistics������������������������������������������������������������������������������������������������������������ 466
Conclusion�������������������������������������������������������������������������������������������������������������������������������� 485

Index��������������������������������������������������������������������������������������������������������������������� 487

xi
About the Author
Edward Pollack has over 18 years of experience in database
and systems administration and architecture, developing a
passion for performance optimization and making things
go faster. He has spoken at many SQL Saturdays, 24 Hours
of PASS, and PASS Summit. This led him to organize SQL
Saturday Albany, which has become an annual event for
New York’s Capital Region.
In his free time, Ed enjoys video games, sci-fi & fantasy,
traveling, and cooking exceptionally spicy foods. He lives in
the subarctic icescape of Albany, NY with his wife Theresa,
his sons Nolan and Oliver, and an impressive collection of
video game-related plushies and figures.

xiii
About the Technical Reviewer
Kathi Kellenberger is the editor of Simple-Talk at Redgate Software, and Data Platform
MVP with over 20 years experience working with SQL Server. She is the author of several
books, including Beginning T-SQL, Beginning Reporting Services, and Expert T-SQL
Window Functions. When she is not working, she enjoys spending time with friends and
family, singing, and climbing the stairs of tall buildings.

xv
Acknowledgments
The SQL Server community is vast, made up of user groups, companies, professionals,
colleges, and organizations that create a network of like-minded individuals all looking
to further their knowledge, while at the same time helping others.
My interest in database administration was borne of some masochistic curiousity,
but the resources to learn, grow, and share that knowledge were made possible by
more people than I can count, each of whom has volunteered countless hours for the
betterment of others.
Thank you to the Professional Association of SQL Server; the Capital Area SQL Server
Group and its founders, Dan Bowlin and Joe Barth; to Autotask, a company that has
given me great amounts of professional freedom to explore database technologies in my
free time; Matt Slocum for organizing and letting me be a part of SQL Saturday Rochester
(the first I spoke at); APress for the opportunity to write and support throughout the
process; SQL Shack for standing behind my writing for years; my friends, who are always
there for me, no matter what life has thrown at us; to the many volunteers who organize,
speak, write, blog, and otherwise improve the world in their free time; and to my family
for having immense patience when I've come up with crazy ideas like this one.

xvii
Introduction
Dynamic SQL is a tool that is often described in bits and pieces, when a need for code
arises and time is limited. This book is an opportuntiy to put as many of those fragments
as possible together into a meaningful journey, from defining the technology to delving
into its deepest and most complex aspects. This is a dive into many topics that are
extremely important when working with any database. We will intentionally delve
deeper into performance optimization, application development, and security than may
seem necessary.

What is This Book?


This is meant to be a discussion of smart database design and architecture, with a focus
on dynamic SQL. If any topic that is covered in an aside feels incomplete, it is because
there simply isn't room in these pages for a thorough analysis of all of them without
losing focus on why we are here. Dynamic SQL is a tool that is often underused, misused,
or overused. The many tangents into other arenas of design and development serve as
guides to keep us on track and emphasize the value of well-written database queries, as
well as ensuring that we use dynamic SQL for the correct applications.
Each chapter delves into a specific topic and attempts to go into as much detail as
possible, while also providing multiple examples to demonstrate it in the simplest way
possible. If you have never written a line of dynamic SQL, this will be an opportunity
to learn, practice, and immediately apply it. If you already have experience in writing
and using dynamic SQL, this will be a chance to learn new applications while getting a
refresher on those you have worked with in the past.
Most examples in this book will reference the Microsoft AdventureWorks sample
database, which provides basic database structures that can be freely experimented
with. Queries are compatible with any version of AdventureWorks, but were tested most
heavily with AdventureWorks 2016.

xix
Introduction

Intended Audience
Anyone with a healthy interest in database administration or development can benefit
from the topics covered within this book. Each chapter starts out with basic definitions
and examples, providing an easy entry point for professionals with any level of
experience. We transition into more advanced techniques, allowing you to not only learn
the basics of an important subject, but also gain access to scripts and ideas that could be
tested and used to solve problems you may face in your everyday experiences.
If you have a particular interest in database security or optimziation, then you will
appreciate the focus on these topics in each chapter. SQL Injection gets an exhaustive
review, with many different aspects and examples presented to ensure a thorough
explanation of this important topic! Every chapter, regardless of topic, will reference
performance whenever possible. It is an oft-made mistake that a database is designed
with little data and few users, ignoring the possibility that it will one day grow into a
behemoth. Reminders are placed throughout this book to consider query performance
at all times, even when performance may seem “good enough.”

Contacting the Author


We can only grow personally and professionally if we are willing to consider other
viewpoints and revise our own to improve.
I love hearing from anyone who has ideas, questions, applications, video game
recommendations, or criticism. Please contact me at ed7@alum.rpi.edu and let me
know what I can do to improve the content of this book, or address any questions or
problems you may have.

xx
CHAPTER 1

What Is Dynamic SQL?


T-SQL is a scripting language that expands with each new release of SQL Server. Success
in the world of database development and administration requires flexibility and the
ability to adapt constantly to new situations, technologies, and demands. Many of the
challenges we face are unknowns, or situations in which we cannot know exactly the
data we will be working with until runtime. In order to find the optimal solution in the
face of unknowns, one of the best tools at our disposal is dynamic SQL.

Understanding Dynamic SQL


Dynamic SQL is quite simple to understand, and once acquainted, the number of
applications can become staggering. Dynamic SQL seeks to solve scenarios where we
want to operate on one or many objects, but do not know all of the pertinent details
as we write our code. Parameters can be passed into our code in order to persist sets
of important values, but what do we do when the structure of our T-SQL is defined by
these values?

A Simple Example
Starting with a very simple select statement, we will build a starting point for
understanding dynamic SQL:

SELECT TOP(10) * FROM Person.Person;

This statement returns 10 rows from the table Person.Person, including all columns
in the table. What if we wanted to select data from a table, but did not know the name
of the table until runtime? How would we substitute the variable table name into our

1
© Edward Pollack 2019
E. Pollack, Dynamic SQL, https://doi.org/10.1007/978-1-4842-4318-3_1
Chapter 1 What Is Dynamic SQL?

T-SQL? Before answering that question, let’s introduce dynamic SQL by simply rewriting
the preceding query so that we are executing it as a character string, rather than standard
T-SQL:

DECLARE @sql_command NVARCHAR(MAX);


SELECT @sql_command = 'SELECT TOP 10 * FROM Person.Person';
EXEC (@sql_command);

In this example, we have defined a character string called @sql_command that


will be used to hold our dynamic SQL. What is the dynamic SQL? It’s the string that
we are building and then later executing. In this case, it is the same select statement
from before, with no alterations. After we set the value of our @sql_command, it is then
executed, providing the same results as before.

The EXEC Statement


EXEC is used to execute @sql_command. EXECUTE may also be used, as they are
equivalent statements. Other ways to execute dynamic SQL will be presented later in
this book, in response to the need for further flexibility or security. Remember to always
put parentheses around the @sql_command string. Here’s an example that omits the
parentheses:

DECLARE @sql_command NVARCHAR(MAX);


SELECT @sql_command = 'SELECT TOP 10 * FROM Person.Person';
EXEC @sql_command;

Failure to do so will result in a somewhat odd error:

Msg 2812, Level 16, State 62, Line 11


Could not find stored procedure 'SELECT TOP 10 * FROM Person.Person'.

The dynamic SQL command string is treated by SQL Server as a stored procedure
when parentheses are not included. Leave them out and you’ll be unable to execute your
SQL string, receiving an error similar to the preceding one.

2
Chapter 1 What Is Dynamic SQL?

Data Type to Use


Note that NVARCHAR(MAX) is used as the data type for our command string. While
we could use VARCHAR, we would potentially be losing data if any extended Unicode
characters were in any of the objects we work with. The size could also be shortened,
but if our command string becomes larger than that size, it will be truncated and our
dynamic SQL will become the source of confusing error messages or logical errors.

For consistency and reliability, use NVARCHAR(MAX) as the data type for your
dynamic SQL command strings.

It may be tempting to use VARCHAR or use a smaller size string to save computing
resources, but as these are scalar variables, the memory used is relatively small and
very temporary. A 10,000 character NVARCHAR string would cost 20KB, whereas the
VARCHAR version would cost 10KB. The difference is minimal and will not have an
impact on any modern computing system. This logic should not be applied to tables,
where computing resources are multiplied by row counts, and additional storage
systems are involved.

Dynamic Execution Process


To understand how dynamic SQL works and the various ways in which it can be
applied to the many problems we encounter, it is important to consider how dynamic
SQL is built. In addition, becoming familiar with the execution process used by SQL
Server in order to parse and run our string of T-SQL will make using dynamic SQL a
much easier process.
All dynamic SQL follows 3 basic steps:

1. Create a string variable that will store our dynamic SQL.


Any variable name may be used.

2. Build a command string and store it in this variable.

3. Execute our command string.

3
Chapter 1 What Is Dynamic SQL?

The benefit of storing our T-SQL command as a string is that we are free to use any
string manipulation commands on it, building it in one or many steps. Now to tackle our
original problem: how to select data from a table that is not defined until runtime. To
accomplish this, we remove Person.Person from the string and replace it with a variable
that we define as shown previously:

DECLARE @sql_command NVARCHAR(MAX);


DECLARE @table_name SYSNAME;
SELECT @table_name = 'Person.Person';
SELECT @sql_command = 'SELECT TOP 10 * FROM ' + @table_name;
EXEC (@sql_command);

The variable @table_name stores the name of the table we wish to query.
Commonly, this would be passed in as a parameter, either from other stored
procedures, or an application that calls this directly. By building it into @sql_command,
we gain the flexibility of querying any table we wish, without hard-coding it ahead
of time. While this is a trivial example (how often will we want to select data in this
fashion?), it provides the basis for thousands of applications, each of which can save
immense time, resources, and complexity. Before diving further into the details of
dynamic SQL and its many uses, let’s look at a more practical (and more complex)
example of dynamic SQL in action.

Dynamic SQL in Action


A common maintenance need is to run T-SQL against many databases on a server.
This maintenance could involve backing up databases, rebuilding indexes, reporting
on critical data elements, or many other applications. If our database list never
changes and no databases are ever renamed, we could hard-code names into each
procedure and not need to worry about changing them in the future. This would work
until the one day when we finally experience those inevitable changes, moving or
renaming databases, ultimately breaking those valuable maintenance procedures. It’s
critical that our maintenance, monitoring, and reporting jobs operate with the highest
level of reliability possible.
Listing 1-1 shows a common example of a statement that could be used to run a
backup against a single database, storing it on a local drive.

4
Chapter 1 What Is Dynamic SQL?

Listing 1-1. Simple Backup Statement

BACKUP DATABASE AdventureWorks2014


TO DISK='E:\SQLBackups\AdventureWorks2014.bak'
WITH COMPRESSION;

This T-SQL will back up the AdventureWorks2014 database to the SQLBackups folder
on the E drive, using compression. If we want to perform a custom database backup on
a subset of databases that all begin with the text “AdventureWorks,” we would need to
build T-SQL that could adapt to collect a list of all databases with that name, and then
perform backups on each of them separately. The following T-SQL shows one way that
this could be accomplished, using dynamic SQL.

Listing 1-2. Dynamic SQL Built to Back Up All Databases Starting with
“AdventureWorks”
DECLARE @database_list TABLE
      (database_name SYSNAME);

INSERT INTO @database_list


      (database_name)
SELECT
      name
FROM sys.databases
WHERE name LIKE 'AdventureWorks%';

DECLARE @sql_command NVARCHAR(MAX);


DECLARE @database_name SYSNAME;

DECLARE database_cursor CURSOR LOCAL FAST_FORWARD FOR


SELECT database_name FROM @database_list
OPEN database_cursor
FETCH NEXT FROM database_cursor INTO @database_name;

WHILE @@FETCH_STATUS = 0
BEGIN
      SELECT @sql_command = '
      BACKUP DATABASE [' + @database_name + ']
      TO DISK="E:\SQLBackups\' + @database_name + '.bak"
      WITH COMPRESSION;'
5
Chapter 1 What Is Dynamic SQL?

      EXEC (@sql_command);

      FETCH NEXT FROM database_cursor INTO @database_name;


END

CLOSE database_cursor;
DEALLOCATE database_cursor;

This T-SQL is certainly more complex than the first backup statement that we
looked at. Let’s break it apart in order to understand what is going on here, and why it
works. We can then focus on the dynamic SQL that provides the backbone of this set of
statements.

1. Populate a table variable with a list of database names.

2. Go through a loop, one time per database.

3. Build a dynamic SQL command string that takes into account the
current database name.

4. Execute the dynamic backup statement.

5. Continue iterating through the loop until all relevant databases


have been backed up.

We declare a number of variables here:

@database_list: Contains all databases that match our search


criteria. In this case, any database that starts with the word
“AdventureWorks” will be included.

@sql_command: This is the command string that will contain our


dynamic SQL statement.

@database_name: Holds the name of the database that is currently


being backed up.

database_cursor: A cursor that will be used to iterate through all


databases named in @database_list.

Much of this example is setup for the loop. The critical portion is where we substitute
the database name and backup file name with @database_name. This allows us to
generate a backup statement that will not only back up each database, regardless of how

6
Chapter 1 What Is Dynamic SQL?

many there are, but will name the backup file using that name. We could just as easily
append additional information onto the file name, such as the date, time, or server
name, if it were important.
Backups are a perfect use of dynamic SQL, as we can continue to customize and add
time-saving functionality into our code, such as:

1. Whether to use compression

2. Determining if subfolders should be used (or not) for backup files


3. Should we perform a FULL, DIFF, or TLOG?

4. Should this backup be COPY_ONLY?

Advantages of Dynamic SQL


There are many reasons why we would want to incorporate dynamic SQL into our
everyday arsenal of SQL Server tools. In addition, there are many specific challenges for
which dynamic SQL is the optimal solution. Discussing these scenarios will highlight
why an entire book can be written on this topic.

Optional or Customized Search Criteria


Search boxes are one of the most common tools used in the development of web pages
or applications. For simple searches, we may only need to pass in a single variable for
evaluation. In more powerful web searches, we may be able to choose between many
criteria, of which each could be evaluated with AND or OR conditions. While we could
write a very long SELECT statement with left joins to every possible table involved, we
would likely end up with an immense, inefficient, and unwieldy pile of T-SQL. Dynamic
SQL allows us to build up a select string that only queries the tables necessary to satisfy a
given search.

Customizable Everything
Adding joins or WHERE clauses are only the beginning. With dynamic SQL, any
statement can be customized to provide greater flexibility to your code. Want to group
by a column based on a dynamic search? The solution is to write the GROUP BY clause

7
Chapter 1 What Is Dynamic SQL?

as dynamic SQL, altering it as needed to fit the needs of each specific situation. Want to
generate row numbers for a data set, but won’t know which columns to partition by or
order by until runtime? No problem!
Our preceding example illustrated how we could use dynamic SQL to customize a
backup operation, and customize the name of the backup file. Any conceivable T-SQL
statement can be altered to utilize dynamic SQL, and in doing so, allow for greater
flexibility in any number of day-to-day challenges.
ORM (object-relational mapping) software can allow for similar levels of customization,
but not all companies can (or want) to use software that automatically generates T-SQL,
as performance and complexity can quickly become overwhelming. Like any querying
application, choosing the correct tool is essential to scalability and performance.

Optimize SQL Performance


So far, dynamic SQL has appeared to make things more complicated, adding the need
for temporary variables, loops, and command strings. Despite the seemingly added
complexity, this framework can allow us to reduce the size of the SQL statements that we
typically execute and improve performance.
Dynamic SQL provides an opportunity to customize our statements to match
performance needs. Removing excess objects, adjusting joins and subqueries, and
reducing the size of an SQL statement can result in faster executions times and reduce
resource consumption.
While our scripts may have more lines of T-SQL, the queries that are ultimately
executed by SQL Server will be simpler and perform more reliably.

Generate Large Amounts of T-SQL or Text, Fast!


Sometimes we need to execute large SQL statements that act on a set of many objects.
Other times, we want to generate output text based on data stored in a specific set of
tables. Perhaps we want to generate SELECT statements that will be used to gather
reporting data from any number of sources.
Writing all of this T-SQL by hand could take a very long time, and lead to a significant
opportunity for human error to occur, as we trudge through a time-consuming, boring
task. If the SQL statements involved are to be run on a regular basis, then preparing them
in advance may be impossible if the target tables or other objects involved can change
on a regular basis.
8
Chapter 1 What Is Dynamic SQL?

Using dynamic SQL, we can generate any amount of commands or text without
limit. SQL Server will not tire of this process, no matter how dull it may seem. This is an
opportunity to automate tedious tasks and reduce operator intervention in those that
would end up being busy work. The result is that our jobs become easier, more fun, and
we can focus on more important tasks that demand our attention!

Execute SQL Statements on Other Servers or Databases


A common challenge occurs when you want to run queries against other entities, but do
not know ahead of time what all of those entities are. If those objects can vary, or change
at runtime, then dynamic SQL is a great solution for managing these operations without
having to hard-code object names that are likely to change over time. This reduces the
chances of an application breaking after a software release, configuration change, or
hardware upgrade.
Similarly, in these scenarios, we may have an application with code that runs
in many locations, with references to servers, databases, or other objects that vary
based on environment. Writing slightly different code in each environment would be
inefficient and would result in significantly higher maintenance needs over time. Far
simpler would be to maintain configuration data and write code that processes those
configurations, reading and writing to the database as needed. Dynamic SQL allows for
that configuration data to be easily handled and acted upon, regardless of the complexity
of the operations involved.

Do the Impossible!


Simply put, there are many tasks in SQL Server that would be extremely difficult, or
seemingly impossible without dynamic SQL. Many common maintenance scenarios that
need to iterate across database objects become trivially easy with dynamic SQL.
Have you ever tried to PIVOT or UNPIVOT across a dynamic column list? The
command is powerful, but requires a definitive column list. If the list is not known until
runtime, then the only way to get the data we need is to use dynamic SQL to insert our
customized column list into the statement and then execute it.
We will have many examples of interesting, useful, and fun ways in which dynamic
SQL can make very difficult tasks easy. Stay tuned and enjoy!

9
Chapter 1 What Is Dynamic SQL?

Dynamic SQL Considerations


As with any tool, dynamic SQL shouldn’t be used everywhere blindly, nor is it the
solution to every database problem you’ll encounter. With a discussion of any tool,
it is imperative that we consider its challenges, pitfalls, and complexities prior to
implementing it.

Apostrophes Can Break Strings


As we build dynamic SQL commands, we incorporate other variables and strings into
them. If any of these contain apostrophes, then our command string will be broken.
The resulting command will, if we are lucky, throw an error and not run. SQL injection
is the process of using the variables in dynamic SQL to intentionally close the string
with an apostrophe, and then attempt to execute malicious code. If we do not
cleanse all parameters and inputs prior to building our command statement, we risk
introducing colossal security holes into our code.
Like in application code, it is imperative that we ensure that our inputs are clean
and that unexpected symbols in our parameters will have no negative effect on the
operation of our code. Failure to do so can result in broken code, unexpected behavior,
or catastrophic security holes. Input cleansing is important in all components of an
application, including the database!

NULL Can Break Strings


NULL is a complicated state of affairs. As an absence of value, any attempt to
concatenate a string with NULL will result in NULL. If the dynamic SQL command string
that we build is passed a parameter that is NULL, then our entire statement will become
NULL. The result will likely be T-SQL that does absolutely nothing. This can lead to
troubleshooting nightmares as it becomes unclear why an SQL statement appears to
do nothing. Further, the search for the NULL parameter may be a daunting task if the
statement in question has many inputs.

10
Chapter 1 What Is Dynamic SQL?

Difficult to Read and Debug


Dynamic SQL loses the benefits of color coding that exist in SQL Server Management
Studio (and most text/code editor tools) that you get when you write standard SQL in
the text editor. Within apostrophes, much of the text will be red, including keywords,
strings, and variable names. In addition, the error checking that is performed as you type
does not occur as effectively within a dynamic SQL string. A simple typo that would be
underlined in red normally will not be as apparent when it is within a string.
In order to combat these challenges, we must devise very well-written T-SQL. In
addition to writing very organized code, we have to be even more diligent when
documenting our work. T-SQL that may normally be trivially easy to understand can
be harder to grasp when written as part of a string. Extra time and care must be used
in order to ensure that when we revisit this code in the future, it is still easy to read and
meaningful.
Dynamic SQL always compiles correctly. To SQL Server, it is simply a character
string. The contents of it are not checked for syntax or object validity until runtime.
Effective testing and debugging are the key to ensuring that the T-SQL we write executes
as we expect it to.
A positive side effect of this situation is that it encourages and trains us to write better
code. We are more conscious of spacing, naming, and line breaks, allowing our code
(dynamic SQL or otherwise) to be easier to read.

Permissions and Scope Are Different


Dynamic SQL statements are executed in their own scope. Variables defined within the
string will not normally be available outside of it. In addition, dynamic SQL is executed
with the permissions of the user executing the overall T-SQL code (stored procedure, job,
etc…). It does not execute with the permissions of the owner of the stored procedure or
the user that happened to be executing it recently.
To avoid unexpected errors, permissions conflicts, or other security concerns, it’s
important to consider what users will be running any code that includes dynamic SQL. If
we need to save data from a dynamic SQL statement, or pass parameters in from outside,
then that needs to be explicitly managed in order to get the desired effect.
Scoping in SQL Server is a feature whose purpose is to segregate objects in different
sessions and benefits us by ensuring that different users cannot access data in-flight that
they may not be allowed to see.

11
Chapter 1 What Is Dynamic SQL?

Dynamic SQL Cannot be used in Functions


Simply put, we can use dynamic SQL in stored procedures, ad hoc T-SQL, and jobs, but
it is not allowed within functions. Any attempt to include dynamic SQL within functions
will result in an error:

Msg 443, Level 16, State 14, Procedure fn_test, Line 72


Invalid use of a side-effecting operator 'EXECUTE STRING' within a
function.

SQL Server functions must be deterministic. Inputs and outputs must be in the
form given in the function definition. Dynamic SQL by nature is nondeterministic, and
therefore cannot be used within functions.

Dynamic SQL Style


Writing code that works is very important. Writing code that is easy to understand and
maintainable is equally as important. As someone charged with the creation and upkeep
of immense numbers of database objects, you must always consider how easy it will be
to read, understand, troubleshoot, and upgrade these objects at any point in the future.
Because dynamic SQL tends to be harder to read, extra care should be taken to ensure
that our T-SQL is well written, effectively documented, and that objects/variables are
named according to reasonable conventions. These design considerations will save your
future self considerable time, as well as show your colleagues that you care about their
well-being and the future of your organization.
These tips apply to all types of coding, but will be of particular benefit when writing
T-SQL, and especially when implementing dynamic SQL.
The rules of good dynamic SQL design begin here, but will continue to be built upon
throughout the rest of this book. Consider any efforts on your part to write maintainable
code, whether it utilizes dynamic SQL or not.

Document Thoroughly
This is the mantra that is repeated to anyone who has ever written a line of code, a script,
or a nontechnical process. Your documentation explains how your code works, why it is
written as it is, and serves as a guide when changes will inevitably be made. T-SQL that

12
Chapter 1 What Is Dynamic SQL?

may not normally warrant documentation will become harder to read when dynamic
SQL is applied. Consider creating additional documentation to supplement this added
complexity.
The first and simplest way to document your work is to include a header at the top
of your file. This header provides basic information on who created this code, some
revision notes, its purpose, and a quick overview of how it works. Understanding the
reasons behind why a stored procedure was created can be as useful as knowing how it
works. More importantly, it is possible to discern the function of code by reading through
it and scratching one’s head a bit. It isn’t possible to figure out the original request
that spurred the creation of that code without either having some existing application
knowledge that others may not have or asking other developers for help.
Consider the following header for a simple backup script:

Listing 1-3. Header Comments, Documenting a Hypothetical Backup Script

/*    8/1/2018 Edward Pollack


      Backup routing for AdventureWorks databases

      As a result of ticket T1234, logged on 7/21/2018, it became necessary


      to selectively back up a limited set of AdventureWorks databases via a
      SQL Server Agent job.  The job can have its schedule adjusted as needed
      to fit the current needs of the business.

      Dynamic SQL is used to iterate through each database, performing the


      backup and naming the resulting file using the database name, date,
      time, and source server. */

This header tells the reader the following:

1. The date that this code was written, to provide context into when
it came about

2. The author, which allows future developers to know where to go


with questions

3. Background into why this was written and the problem that was
being addressed

4. A brief description of how it works and any special features that


are used

13
Chapter 1 What Is Dynamic SQL?

This short documentation block answers most of the common questions that a
developer may have about your code. The things we consider obvious while writing
T-SQL may not be so obvious to someone else reading this years later. Our own code
is always easier to read than that of others, and this is easy to forget when buried in
development projects. As time passes, though, even our own code can be hard to
understand as we become more detached from the details of how we wrote it.
When writing code that involves dynamic SQL, we must consider documenting
thoroughly, but also not go overboard and explain every single line of T-SQL. Let’s take
our backup routine from earlier and add some meaningful documentation to it.

Listing 1-4. Backup Script Sample, with Documentation Added

-- This will temporarily store the list of databases that we will back up
below.
DECLARE @database_list TABLE
      (database_name SYSNAME);

INSERT INTO @database_list


      (database_name)
SELECT
      name
FROM sys.databases
WHERE name LIKE 'AdventureWorks%';
-- This WHERE clause may be adjusted to back up other databases besides
those starting with "AdventureWorks".

DECLARE @sql_command NVARCHAR(MAX);


DECLARE @database_name SYSNAME;
DECLARE @date_string VARCHAR(17) = CONVERT(VARCHAR, CURRENT_TIMESTAMP, 112) +
'_' + REPLACE(RIGHT(CONVERT(NVARCHAR, CURRENT_TIMESTAMP, 120), 8), ':', ");

-- Use a cursor to iterate through databases, one by one.


DECLARE database_cursor CURSOR FOR
SELECT database_name FROM @database_list
OPEN database_cursor
FETCH NEXT FROM database_cursor INTO @database_name;

14
Exploring the Variety of Random
Documents with Different Content
concrete work; except that the drain pipe used will be paid for per
linear foot at the price named in the contract which price shall cover
the cost of furnishing and putting the pipe in place complete.
PART II
INSTRUCTIONS TO INSPECTORS ON
STREET PAVING WORK
INTRODUCTORY NOTE

Any general code of instructions for inspectors on street paving


work must necessarily be little more than tentative or suggestive, as
applied to any one city.
That here presented is intended for use in connection with the
specifications in Part I of this book. Different specifications might
make necessary many changes in the instructions that follow. The
local or special conditions in any city might also make them
inapplicable without material modifications and additions.
The author believes that in every city where considerable street
pavement work is prosecuted and a number of inspectors employed
on such work, some such code of instructions to inspectors should be
formulated and used. Even where the inspectors are of high
character and fully competent, the adoption of a system of rules and
regulations relating to their work will tend to unify procedure,
prevent misunderstandings and promote good discipline.
The preparation of such a code of instructions requires no little
time and thought, especially where there are no precedents to
suggest what is needed, or to be used as rough patterns. It is with the
hope that the code here presented may be found useful in this way,
even though it may be not appropriate for adoption in any given city,
that it is offered to municipal engineers.
GENERAL REMARKS ABOUT INSPECTION
AND INSPECTORS

The necessity of having competent and honest inspectors upon any


public work during its construction need not be here argued. It is
generally understood and recognized in the case of work carried out
by contract. Where the contractor is incompetent, careless or
unreliable, inspection is absolutely necessary if good work is to be
secured. Even where the contractor thoroughly understands his
business and intends to faithfully comply with the specifications and
to do all his work in a proper manner, the inspector cannot safely be
dispensed with. The contractor cannot at all times be on the work;
whatever may be his intention and instructions, foremen and
laborers cannot be depended upon to exercise the proper care and
judgment, even where they have no inclination, motive or interest to
do otherwise, which is not always the case. It seems to be deeply and
almost irradicably fixed in the mind of the average foreman on
municipal work that he is expected to show his ability and skill by
evading or circumventing the strict requirements of the
specifications and the watchfulness of the municipal agents,
whatever his employer may say to the contrary. Even when fairly
careful, honest and conscientious he may need friendly oversight. He
is naturally and properly anxious to save all the money he can for his
employer, whether from a desire to loyally serve that employer’s
interest, or to enhance his reputation for doing work cheaply. He
may not appreciate the importance of minor requirements of the
specifications and may believe they can be ignored without real
detriment to the utility of the completed work. Like other men he
may sometimes be careless or forgetful, however good his intentions.
These human qualities are not confined to foremen on contract
work. They are liable if not likely to be found in the foremen on work
done for the municipality by the direct, or day’s work system. They
may be encouraged by the impression or belief that they will not be
held by the municipal authorities to as strict an observance or
accountability as they would be if the work were being done by a
contractor—which is too often true. The absence of some of the
motives of foremen employed by contractors is, therefore, not a
sufficient reason for dispensing with inspectors on work done by the
city direct.
It should be trite to say that inspectors should be chosen with
some regard to their qualifications for the work they are expected to
do. One cannot, of course, expect to secure experts for such positions
at the rate of compensation usually paid; but it is reasonable to
require that an appointee to such positions shall possess the
following qualifications:
That he shall be honest, trustworthy and loyal.
That he shall be a man of at least average intelligence and common
sense.
That he shall have some practical knowledge of or experience in
the work he is employed to inspect.
That he shall be a man of good personal habits.
That he shall be habitually punctual, industrious, and alert, and
shall possess a fair share of that quality commonly called “backbone.”
It will not, I am sure, be denied that a very large number of the
inspectors found upon municipal public work fall short in one or
more of these qualifications.
Inspectors who habitually expect or accept compensation or
gratuities from the contractor violate the first requirement, and their
number is astonishingly large. These are useless, or worse than
useless, to the municipality. There are not a few contractors who will
take advantage of the presence on the work of such inspectors to do
things that they would not do if thrown wholly on their own honor
and responsibility. Contractors can hardly be blamed for expecting
some compensation for the money thus exacted or accepted by the
inspector from them.
The requirement that the appointee shall have some practical
knowledge of and experience in the work he is expected to do, while
often disregarded, is of real importance. Good workmanship
embraces many small details that are essential, and inferable from
the specifications, but cannot be stated at length therein, and the
inspector should have a good working knowledge of these.
Unfortunately, large numbers of inspectors are employed for
personal or political reasons, or because more competent men are
not available. They know little or nothing from practical knowledge
and experience about the work they are assigned to inspect. Efficient
service cannot be expected from such, even if they possess the other
necessary qualifications. The obvious remedy, where a sufficient
number of qualified inspectors cannot be obtained at call, would be
to train up a permanent force by having the promising tyros serve a
sufficient apprenticeship under older and more experienced
inspectors; but this is seldom done.
Inspectors, whether appointed by the engineer in charge or not,
should be absolutely under his control, including the power of
dismissal without unnecessary formalities. Where they are under
civil service regulations and can be dismissed only upon charges and
after a hearing before some third person, the obstacles in the way of
promptly getting rid of an incompetent or unfaithful inspector are
often so great as to be discouraging if not deterrent. He may be
clearly below a reasonable standard of honesty or efficiency but it
may be very difficult to formulate distinct charges and support them
with positive evidence. Thus, one may be morally certain that an
inspector is accepting gratuities from a contractor, but unable to
prove it by legal evidence. If in such cases the judgment and integrity
of the responsible engineer cannot be trusted, he is not fit, himself,
for the position he occupies.
The question of how much authority and power shall be given to
the inspector is always a troublesome one. If he be too much limited
or restricted, he cannot properly discharge his duties or command
the necessary respect from the contractor and his employees; if he be
given too much authority and is disposed to be arbitrary or
unreasonable, the contractor may be unjustly treated. So far as the
definite requirements of the specifications are concerned there can
usually be no room for going astray, but in the numberless little
details not specifically covered in the specifications, or where their
language and intent is not clear, differences and disputes may result
which require a degree of knowledge and judgment that the
inspector may not possess, and where his decisions might be against
the interests of the municipality or unjust to the contractor. In such
cases the matter should always be referred to the engineer or his
assistant.
Where inspectors are employed upon work done directly by the
municipality the relation between them and the superintendent or
foreman should be clearly defined and understood. Briefly stated,
this relation should be the same as that between the inspector and
the contractor on contract work.
INSTRUCTIONS FOR THE GUIDANCE OF
INSPECTORS ON STREET PAVING AND
INCIDENTAL WORK

GENERAL

1. The chief duty of the inspector is to see that the work to which
he is assigned shall be carried out in full and strict accordance with
the plans and specifications therefor, and with such additional
instructions as may from time to time be given by the engineer.
2. These instructions are intended to supplement but in no case to
take the place of the specifications for the work, which must be fully
and strictly complied with, unless they are changed or modified by
the engineer.
3. Wherever the words “the engineer” are used in these
instructions they refer to the chief engineer in charge of the work for
the municipality, or his authorized deputies or agents.
Wherever the words “the contractor” are used they refer to the
person, or firm, or corporation under contract to perform the work to
which you are assigned, or his agents or employees engaged upon the
work.
4. You will report to and be under the exclusive direction and
control of the engineer.
5. Upon being assigned to any work you will obtain copies of the
plans and specifications under which it is to be done, and will
carefully read and acquaint yourself with all their provisions and
requirements. If any part of these plans and specifications are not
clearly understood by you, you will ask the engineer for explanations
and instructions, and will be governed thereby. Failure to
understand or to be familiar with the plans and specifications will in
no case be accepted as an excuse for not complying with and
enforcing them.
You will also obtain, study and familiarize yourself with all the
blank forms, reports, etc., that are to be used by you on the work, so
that they may be correctly utilized; you will also obtain a proper
supply of such apparatus, tools, stationery, etc., as you will need
upon the work.
6. In inspecting any work done under contract you are to assume
that the contractor is bound to carry out in good faith the plans and
specifications attached to his contract, and that he may be, and must
be, held to a strict compliance with them unless you receive
instructions to the contrary from the engineer. You will be held
responsible for the enforcement of these plans and specifications so
far as they relate to the doing of the work.
7. In case the contractor or his agents shall neglect or refuse to
comply with the plans and specifications or to perform the work in a
proper manner, or in accordance with the instructions of the
engineer, you will direct him to suspend the particular work in fault
and will report all the facts promptly to the engineer and await his
instructions, which, when received, you will proceed to enforce.
8. You will be expected to be upon the work at all times when work
is in progress, whether during ordinary working or office hours, or
not. But in case the contractor shall work continuous day and night
forces, or otherwise so conduct the work as to require the presence of
an inspector for unusual or unreasonable hours of service, you will
notify the engineer and ask instructions or assistance. If for any
reason you are to be unable to be present whenever work is in
progress you will promptly notify the engineer in time to enable him
to supply a substitute.
9. All reports and records required of you by the engineer must be
promptly made up and forwarded to him at the time and in the
manner he may direct. You will promptly acknowledge in person or
writing any instructions received from the engineer and in case such
instructions are not clearly understood by you, you will ask for
explanation or advice.
You will keep a notebook or diary in which all occurrences or
matters of interest relating to the work will be recorded each day,
which book will be delivered to the engineer on the completion of the
work, or whenever your connection therewith may end. All diaries,
notebooks and records kept on the work are the property of the city.
10. While your first duty is to see that the work is done in
accordance with the plans and specifications, you will, so far as
consistent with this duty, be considerate of the interests of the
contractor and facilitate his work as much as practicable. Materials
should be inspected promptly upon delivery and in such a manner as
not to delay or discommode the contractor more than is necessary.
Errors, defects or delinquencies in the work should be detected and
called attention to promptly, so as to avoid tearing up and re-
construction as far as practicable.
11. Orders and instructions should be given to the contractor or his
foremen or representatives on the work and not to the laborers,
unless in urgent cases.
12. You shall not at any time, before, during, or after the
completion of the work, ask or accept from the contractor or any of
his agents or employees, or from any other person whom you may
have reason to believe is acting for the contractor, any money, gifts,
gratuities or other considerations of value, whether as compensation
for services rendered him or otherwise. The penalty for violation of
this rule will be dismissal from the service followed by such legal
prosecution as the engineer may direct. This rule does not apply to
the ordinary small courtesies common between gentlemen associated
in business.
13. It is a condition of your employment that your services shall be
loyal to the interests of the municipality, efficient, and satisfactory to
the engineer, and that whenever in his opinion your service falls
short of this standard he may dismiss you at any time, without
previous notice.
STREET PAVING WORK

14. Sub-foundation.—The usual procedure in preparing the


sub-foundation for a pavement is:
The removal of the old pavement if one exists on the street.
The setting of new or the resetting of the old curbing. (If concrete
combined curb and gutter is used it is not constructed until after the
rough grading is done.)
The rough grading of the street.
The fine grading of the street.
In this work the main things for the inspector to look after are the
following:
If an old pavement is to be taken up and any part of it is to be
reserved for use by the city, see that the material is not injured in
handling and is disposed of as the specifications or the instructions
of the engineer require.
15. Curbing.—New curb stone should be inspected as soon as
practicable after it is delivered on the street so that the contractor
may have time to replace any that may be rejected. Rejected stones
should be plainly marked on their tops with red paint, and the
contractor required to remove them from the street. Stones that are
defective in dressing should be given some distinctive mark and the
attention of the contractor called to them so that the necessary
redressing may be done well in advance of the setting, and
reinspected.
If the old curb stone is to be redressed and reset, it should be
inspected as soon as possible after it is taken out and all stones that
are not suitable for re-use rejected and removed from the work. The
redressing of the stones should be looked after carefully. The
tendency in this work is toward the use of old stones that do not, or
cannot be made to comply with the specifications, and toward laxity
in the dressing.
Setting Curbing.—In the setting of curbing the points to be
looked after are:
To see that the trenches are excavated to the full dimensions so as
to provide space for the use of the full quantity required of concrete,
gravel or broken stone in which the curb is to be embedded.
To see that the stones are set to true grade and alignment.
That the concrete, gravel or crushed stone used is of the proper
quality, the proper quantity used, and that it is placed and
compacted so as to fill the spaces designated under and around the
stone. The tendency of the workmen is to treat these details as
unimportant and to slight the work.
After the curbing is set any spaces between it and the edge of the
sidewalk are to be cleaned out, filled and trimmed with Portland
cement mortar.
16. Rough Grading.—If plows and scrapers are used in the
rough grading, see that they are not allowed to penetrate or to
disturb the natural soil below the true grade, and to prevent this it is
better to require that the plowing and scraping shall not approach
nearer than one or two inches to the true grade.
If embankment or filling is necessary, the filling material must be
deposited in regular layers not more than six inches deep and each
layer must be well compacted by rolling or ramming before the next
layer is applied.
17. Fine Grading.—The fine grading (with pick or mattock and
shovel), to bring the sub-grade to the true grade and contour
required, should be done with reasonable accuracy and care.
Allowance must be made for compression under the roller, so that
when completed the surface shall not vary more than half an inch
below nor more than a quarter of an inch above the true grade, in
order that room for the full thickness of concrete foundation may be
provided, and no more. Humps, depressions and irregularities of the
surface are to be avoided. The fine grading will be completed by
thoroughly rolling or tamping the surface to make the soil firm and
dense. On some soft or wet, clayey soils, excessive rolling may
produce a plastic or “liver-like” condition and the rolling should be
discontinued where such a tendency appears.
18. Work Incidental to Grading.—Where it is a part of the
contractor’s work to reconstruct or readjust catch basins, manholes,
covers, and other structures, he should be required to keep this work
done well in advance of the laying of the concrete foundation. If this
work is delayed until after the main body of the concrete is in place,
leaving openings to be filled later, hurriedly and perhaps carelessly,
it may result in weak patches in the concrete around these structures,
where, in fact, the concrete should be strongest.

CONCRETE FOUNDATIONS

19. The concrete foundation of a pavement is a most important


part of the work, and must not be slighted in any way.
20. Quality of Materials.—The quality of the materials used for
making the concrete are fully described in the specifications, and
they must be substantially and uniformly adhered to.
The Portland cement delivered on the street will presumably have
been previously inspected and accepted by the engineer, but any
change in the kind, or brand, or apparent quality of the cement
delivered after the beginning of the work should not be permitted
without the approval of the engineer. Barrels or bags of cement not
plainly marked with the brand, or name of the manufacturer must be
rejected. You will frequently make balls of stiff mortar from the
cement and sand used (1 part cement, 2 parts sand) and set them
aside for observation. If in eight hours these balls do not set up hard
the fact should be promptly reported to the engineer.
The sand and stone will naturally vary slightly from time to time,
but should not differ materially from the requirements of the
specifications or the standards adopted by the engineer. Stone or
gravel that contains any considerable quantity, say more than 5%, of
foreign matter, such as soil or refuse, should be rejected. Stone
containing an excessive quantity of fine fragments or screenings
bunched together in the pile, should be mixed with the mass of larger
stone before it is used, so as to produce a mass of fairly even
composition. Stone that is coated with mud, or that is not sound and
hard should be thrown out.
Sand that is moderately coarse and does not contain more than
7½% of clay or soil, and is free from vegetable refuse, may be used.
21. Storing Materials.—Cement stored on the street must be
stacked up on dry lumber floors at least four inches from the ground
and effectually covered to protect it from rain or snow. Packages of
cement which, when turned out for use contain hard lumps should
be rejected.
Sand and stone must be stored on tight lumber floors to prevent
their admixture, when being shoveled up, with street soil or other
impurities. This requirement must be enforced.
22. Proportioning Concrete.—The proportions of cement,
sand and stone or gravel named in the specifications must be strictly
adhered to.
A barrel of Portland cement will be considered as four cubic feet
and a standard bag of cement as one cubic foot.
It is usual for the contractor to measure the sand and stone by
wheelbarrow loads. In that case you will measure and determine the
capacity, in cubic feet, of the convenient or average wheelbarrow
load, and determine the size and number of loads each of sand and
stone required for one batch of concrete (if made by hand) and
thereafter see that the quantities thus determined are used, and the
wheelbarrow loading uniform. To measure the contents of a
wheelbarrow load, use a square box holding one cubic foot. It cannot
be accurately measured on the barrow.
As a check upon the proportioning of materials, you should
occasionally keep a record of the quantities of cement, sand and
stone used for, say, one or two hours, and mark and measure up the
quantity of concrete made with them, and from this data compute
the ratios of the materials used. (In ordinary street concrete,
proportioned 1:3:6, the volume of rammed concrete made will just
about equal the volume of the stone used.)
Where the concrete is mixed by a machine with automatic feeding
devices, reliance must not be placed upon the machine for proper
proportioning. Frequent tests of the materials used and the volume
of concrete turned out should be made as described above, and any
necessary corrections made in the feeding devices of the machine.
This test should be made daily if practicable. The eye will usually
detect any material change in the composition of the concrete, and
whenever such changes are observed the composition ought to be at
once checked up by measurement.
23. Mixing Concrete.—See that the specifications are followed.
Use sufficient water to make what is commonly called a wet concrete
but not so wet that free water will drain from the mixed batch before
ramming. The test for proper mixing will be that all the fragments of
stone are completely covered with mortar.
24. Placing Concrete on Street.—All concrete must be placed
and rammed before it begins to stiffen or set. Concrete should be so
handled that the mortar will segregate or separate from the stone as
little as possible. When in place on the street it should make a mass
of uniform consistency and appearance. The concrete should, as soon
as placed on the street, be graded to a proper thickness and surface,
and then well compacted by ramming. Insist on good ramming; this
part of the work is not often properly done. The surface of the
concrete should be brought to the true grade by proper grading and
tamping. Where necessary to raise the surface, additional concrete,
made of small stone, may be applied during the tamping. Neat
mortar should not be used for this purpose. Sweeping mortar over
the surface with street brooms should not be permitted—it is useful
only to cover up defects and to produce an extra smooth surface
(which is not usually desirable). To measure the thickness of the bed
of concrete, lay a straight-edge upon the surface with the end of the
straight-edge projecting a foot or two beyond the edge of the
concrete, and measure the space from the lower edge of the straight-
edge down to the sub-foundation.
Care must be taken to finish the surface of the completed concrete
at the proper distance below and truly parallel to the grade and
contour of the finished pavement. This is more important in the case
of sheet asphalt pavements, since irregularities in the concrete will
cause variation in the thickness of the finished asphalt surface, which
will affect the durability of the pavement. The grade stakes for the
concrete should therefore be carefully set and worked to, and humps
or depressions between the stakes should be avoided.
25. In hot, dry, weather the concrete must be kept damp by
watering it until it is at least five days old, and horse-travel over it
must be prohibited until it has set up hard enough not to be injured
thereby.
26. Old Paving Stone Foundations.—If a foundation is to be
made of old paving stone blocks reset, the important points to be
looked after are,
The preparation of the sub-foundation.
The sand bed.
The setting of the blocks.
The grouting of the blocks.
The sub-foundation should be prepared, where necessary, as
stipulated in Sects. 16 and 17 of these instructions. This part of the
work is likely to be carelessly done unless proper work is insisted
upon.
The sand used as a cushion course must be of good quality and
reasonably clean. The use of mixed sand and soil taken from the
street excavation should not be permitted.
The blocks, after being cleaned from adhering dirt, should be set
with reasonable care, well bedded in the sand cushion, and laid at
such an elevation below the completed pavement surface, that after
being well rammed, the surface will be at the proper elevation below
the pavement datum.
The grouting of the joints is the most important part of the work.
See that the mortar is made and used in accordance with the
specifications, and that the joints are completely filled.
27. Broken Stone Foundations.—The specifications for
broken stone foundations are sufficiently explicit, and compliance
should be insisted upon. Thorough rolling is especially important.

SHEET ASPHALT PAVEMENTS

28. The manufacture of the mixture for sheet asphalt pavement,


and the laying of it on the street, involve a number of separate
elements or operations. The quality and durability of the pavement
will depend largely upon the skill, intelligence and accuracy with
which each and all parts of the work are carried out. The inspector,
both at the mixing plant and on the street, must give constant and
intelligent attention to the work.
29. Inspecting at Paving Plant.—Inspectors at the plant where
the asphalt paving mixture is made will be provided with the
following apparatus:

1 nest of standard sieves, Nos. 10, 30, 50, 80, 100 and 200
2 thermometers for sand
1 tank thermometer
2 standard flow-plates and one dozen cartridges for same
1 balance for weighing sand
6 dishes, glass or porcelain, 4 inches in diameter.
30. The more important things for the Plant Inspector to watch
during the manufacture of asphalt paving mixture are:

a. Quality and grading of the sand.


b. Temperature of heated sand.
c. Uniform quality of the refined asphalt and flux.
d. Temperature of melted bitumens.
e. Uniform consistency of asphaltic cement.
f. Correct ratios of sand and asphaltic cement.
g. Thorough mixing of materials.
31. Sand.—The general quality and grain-size of the sand,
required by the specifications, must be closely watched and adhered
to.
It is hardly ever possible to secure a natural sand that will conform
strictly to the theoretical composition, but substantial compliance
should be required unless the Chief Engineer directs otherwise. After
any sand has been approved for a given contract or street, all the
sand thereafter used for that street should conform closely to the
standard so approved; for any material change in the quality of the
sand will change the quality of the pavement produced, unless the
proportions of the other materials are changed accordingly. Any
material change in the sand should therefore be promptly reported to
the Chief Engineer.
32. Sand for the sieve tests should be taken from the hot-sand box
and allowed to become cold. From the sample thus taken, weigh out
50 grams. Arrange the nest of sieves with the No. 200 at the bottom
followed by Nos. 100, No. 80, No. 50, No. 30 and No. 10 at the top.
Place the weighed sample upon the No. 10 sieve and thoroughly
shake the nest until all the sand that will, has passed through the
various sieves; then weigh the various portions, beginning with that
which has passed the No. 200 (deducting in each case the weight of
the dish). The portion remaining on the No. 200, is the quantity
passing the No. 100, etc. The sum of all these portions should be 50
grams, though it will usually be slightly less because of unavoidable
waste. In use, keep the sieves clean and as free as possible from
adhering grains by using a bristle brush.
33. Temperature of the Sand.—The temperature of the heated
sand should be frequently noted by plunging the thermometer into
the mass. It should not fall below 315° F. nor exceed 375° F. It is very
important to avoid over-hot sand, because, in the mixer, the very
thin films of bitumen covering the hot-sand grains are readily and
quickly scorched by the excessive heat in the sand.
34. Refined Asphalt and Fluxing Oil.—Unless otherwise
notified by the Chief Engineer you will assume that the stock of
refined asphalt on hand at the beginning of the work has been
inspected and is satisfactory. You will, however, note the appearance
of the material and the character and marking of the barrels or
packages, and will not permit the use of any new shipments of
materials of different appearance or brand without authority from
the Chief Engineer. The above instructions apply also to the oil used
for fluxing.
35. Ratio of Asphalt and Flux.—You will frequently note and
record the weight of asphalt and of flux used in charging the melting
tanks.
36. Temperature in Melting Tanks.—The melting kettles are
usually equipped with fixed thermometers, but their reading should
be frequently checked with your detached tank thermometer. You
will see that the temperatures in the kettles are kept within the limits
named in the specifications. See that the melted asphaltic cement is
kept constantly agitated as required by the specifications.
37. Uniformity of Asphaltic Cement.—It is very important
that the asphaltic cement shall be of uniform composition and
consistency, or hardness.
When the composition and consistency of the asphaltic cement to
be used for the surface-course of any street has been determined
upon and approved by the engineer, you will take out and preserve a
sufficiently large quantity to be used thereafter as a standard, and
each subsequent tank of cement, before it is used, should be tested
by comparing it with this standard in the following manner, using
the flow-plate apparatus:
Melt and fill two cartridges with the standard cement, and two
with the cement to be tested, marking them for identification. When
cold, carefully remove with a wetted dull knife any surplus cement
from the ends of the cartridges so as to leave them exactly full and
the metal ends clean. Immerse them in cold water until they are all of
the same temperature. Then place and fasten these filled cartridges
on the grooved plate with their lower ends exactly to the zero line on
the plate, the standard samples alternating with the new. Then place
the plate, at an inclination of about 45°, in front of a source of
uniformly distributed heat, as, for instance, near a steam boiler or
large steam pipe, or (when the weather is favorable) in the sunshine,
facing the sun. Leave the plate in this position until the bitumen in
the cartridges shall have melted and flowed down the grooves an
average of from four to five inches. Then measure the lengths of flow
of each from the zero line on the plate. If the average flow of the new
samples differs more than 5 per cent. from that of the standards, the
new cement should be regarded as unsatisfactory and should be
corrected by adding more asphalt or flux, as may be required.
(If a penetration apparatus is supplied or is available, the
penetration method may be used instead of this flow-method for
determining the consistency of the cements. Proper instructions for
its use will accompany the apparatus.)
38. Ratio of Sand and Cement.—When the proper ratio of
sand and asphaltic cement has been determined for any street you
will see that this ratio is carefully adhered to. The correctness of the
weighing scales used should be frequently tested.
39. Mixing.—When the sand and asphaltic cement are properly
mixed the appearance of the mass will be perfectly uniform
throughout, with every grain of sand entirely and evenly coated with
cement. The mixing should, however, be continued somewhat
beyond the attainment of this condition, so as to insure
thoroughness. After the proper result has been arrived at, either the
time in the mixer, or the number of revolutions of the mixer shaft
should be noted and the standard thus determined substantially
adhered to thereafter. With the standard mixers commonly used, the
hot sand and the filler (dust) should first be put into the mixer and
mixed for about fifteen seconds; then the asphaltic cement added
and the mixing continued as long as necessary (in the usual mixer
not less than sixty seconds).
40. Records.—You will keep complete records of the results of
your observations and tests in a book which will be provided for the
purpose, and will make such reports to the Engineer as he may
direct.
41. Inspection on the Street.—The leading things to be kept in
mind by the street inspector, in the laying of asphalt pavement on
the street, are the following:

a. Temperature of mixtures when applied to the street.


b. Proper preparation of the street surface before material is laid.
c. Laying and rolling of the binder-course.
d. Laying of the surface-course.
e. Rolling the surface-course.
42. Inspector’s Equipment.—The inspector will be supplied
with the following apparatus:

1 armored thermometer reading from 200° F. to 500° F.


1 graduated spatula.
1 twelve-foot straight-edge.
1 fifty-foot tape line.

You might also like