Troubleshooting Sharepoint The Complete Guide To

Tools Best Practices Powershell Oneliners And

Scripts 1st Edition Simpkins download

https://ebookbell.com/product/troubleshooting-sharepoint-the-
complete-guide-to-tools-best-practices-powershell-oneliners-and-
scripts-1st-edition-simpkins-32704104

Explore and download more ebooks at ebookbell.com

Here are some recommended products that we believe you will be
interested in. You can click the link to download.

Troubleshooting Relationships On The Autism Spectrum Ashley Stanford

https://ebookbell.com/product/troubleshooting-relationships-on-the-
autism-spectrum-ashley-stanford-46093366

Troubleshooting And Repairing Diesel Engines 5th Edition Paul Dempsey

https://ebookbell.com/product/troubleshooting-and-repairing-diesel-
engines-5th-edition-paul-dempsey-46137860

Troubleshooting Microsoft Teams Enlisting The Right Approach And Tools

In Teams For Mapping And Troubleshooting Issues 1st Edition Balu N
Ilag

https://ebookbell.com/product/troubleshooting-microsoft-teams-
enlisting-the-right-approach-and-tools-in-teams-for-mapping-and-
troubleshooting-issues-1st-edition-balu-n-ilag-46285446

Troubleshooting Java Meap V08 All 12 Chapters Laurentiu Spilca

https://ebookbell.com/product/troubleshooting-java-
meap-v08-all-12-chapters-laurentiu-spilca-47532178
Troubleshooting And Supporting Windows 11 Creating Robust Reliable
Sustainable And Secure Systems 1st Edition Mike Halsey

https://ebookbell.com/product/troubleshooting-and-supporting-
windows-11-creating-robust-reliable-sustainable-and-secure-
systems-1st-edition-mike-halsey-48671120

Troubleshooting The Extrusion Process A Systematic Approach To Solving

Plastic Extrusion Problems 3rd Edition Noriega Mpe

https://ebookbell.com/product/troubleshooting-the-extrusion-process-a-
systematic-approach-to-solving-plastic-extrusion-problems-3rd-edition-
noriega-mpe-50166080

Troubleshooting Tivoli Using The Latest Features Ibm Redbooks

https://ebookbell.com/product/troubleshooting-tivoli-using-the-latest-
features-ibm-redbooks-51388430

Troubleshooting Electricalelectronic Systems 3rd Edition Glen A Mazur

https://ebookbell.com/product/troubleshooting-electricalelectronic-
systems-3rd-edition-glen-a-mazur-51425974

Troubleshooting And Maintaining Pcs Allinone For Dummies 4th Dan

Gookin

https://ebookbell.com/product/troubleshooting-and-maintaining-pcs-
allinone-for-dummies-4th-dan-gookin-53016744
Troubleshooting
SharePoint
The Complete Guide to Tools, Best Practices,
PowerShell One-Liners, and Scripts
—
Stacy Simpkins
 Troubleshooting
SharePoint
The Complete Guide to Tools,
Best Practices, PowerShell One-Liners,
and Scripts

Stacy Simpkins
Troubleshooting SharePoint
Stacy Simpkins
Brandon, Florida, USA
ISBN-13 (pbk): 978-1-4842-3137-1 ISBN-13 (electronic): 978-1-4842-3138-8
https://doi.org/10.1007/978-1-4842-3138-8
Library of Congress Control Number: 2017960834
Copyright © 2017 by Stacy Simpkins
This work is subject to copyright. All rights are reserved by the Publisher, whether the whole or part of the
material is concerned, specifically the rights of translation, reprinting, reuse of illustrations, recitation,
broadcasting, reproduction on microfilms or in any other physical way, and transmission or information storage
and retrieval, electronic adaptation, computer software, or by similar or dissimilar methodology now known or
hereafter developed.
Trademarked names, logos, and images may appear in this book. Rather than use a trademark symbol with
every occurrence of a trademarked name, logo, or image we use the names, logos, and images only in an
editorial fashion and to the benefit of the trademark owner, with no intention of infringement of the trademark.
The use in this publication of trade names, trademarks, service marks, and similar terms, even if they are
not identified as such, is not to be taken as an expression of opinion as to whether or not they are subject to
proprietary rights.
While the advice and information in this book are believed to be true and accurate at the date of publication,
neither the authors nor the editors nor the publisher can accept any legal responsibility for any errors or
omissions that may be made. The publisher makes no warranty, express or implied, with respect to the material
contained herein.
Cover image designed by Freepik
Managing Director: Welmoed Spahr
Editorial Director: Todd Green
Acquisitions Editor: Joan Murray
Development Editor: Laura Berendson
Technical Reviewer: Samarjeet Singh Tomar
Coordinating Editor: Jill Balzano
Copy Editor: Kim Burton-Weisman
Compositor: SPi Global
Indexer: SPi Global
Artist: SPi Global
Distributed to the book trade worldwide by Springer Science+Business Media New York,
233 Spring Street, 6th Floor, New York, NY 10013. Phone 1-800-SPRINGER, fax (201) 348-4505, e-mail
orders-ny@springer-sbm.com, or visit www.springeronline.com. Apress Media, LLC is a California LLC
and the sole member (owner) is Springer Science + Business Media Finance Inc (SSBM Finance Inc).
SSBM Finance Inc is a Delaware corporation.
For information on translations, please e-mail rights@apress.com, or visit http://www.apress.com/
rights-permissions.
Apress titles may be purchased in bulk for academic, corporate, or promotional use. eBook versions
and licenses are also available for most titles. For more information, reference our Print and eBook Bulk
Sales web page at http://www.apress.com/bulk-sales.
Any source code or other supplementary material referenced by the author in this book is available to
readers on GitHub via the book’s product page, located at www.apress.com/9781484231371. For more
detailed information, please visit http://www.apress.com/source-code.
Printed on acid-free paper
This book is dedicated to Saanvi, Owen, Willow, Oaklyn, and Weston.
Contents

About the Author����������������������������������������������������������������������������������������������������� ix

About the Technical Reviewer��������������������������������������������������������������������������������� xi
Acknowledgments������������������������������������������������������������������������������������������������� xiii
Introduction�������������������������������������������������������������������������������������������������������������xv

■
■Chapter 1: Least-Privileged SharePoint Builds������������������������������������������������������ 1
Why Least Privilege���������������������������������������������������������������������������������������������������������� 1
An Ounce of Prevention Is Worth a Pound of Cure���������������������������������������������������������������������������������� 1
Local Group Membership������������������������������������������������������������������������������������������������������������������������ 5
Ask the Domain Controllers�������������������������������������������������������������������������������������������������������������������� 6
Database Permissions for Farm Account Vs Install Account������������������������������������������������������������������ 7
File System Permissions for Members of the WSS_Admin_WPG Local Group��������������������������������������� 7
Logging File Paths�������������������������������������������������������������������������������������������������������������������������������� 12
Registry Permissions���������������������������������������������������������������������������������������������������������������������������� 14
Application Pool Accounts�������������������������������������������������������������������������������������������������������������������� 15
WSS_WPG Registry Access������������������������������������������������������������������������������������������������������������������ 16
Application Pool Accounts in IIS����������������������������������������������������������������������������������������������������������� 16
PowerShell to Reset Local Permissions and Files�������������������������������������������������������������������������������� 18
Inspecting for Least Privilege��������������������������������������������������������������������������������������������������������������� 18

Next Steps���������������������������������������������������������������������������������������������������������������������� 37
■
■Chapter 2: Key Settings of a Good Build�������������������������������������������������������������� 39
PowerShell Aliases��������������������������������������������������������������������������������������������������������� 40
Verb-Noun���������������������������������������������������������������������������������������������������������������������� 40
All PowerShell cmdlets Are Objects������������������������������������������������������������������������������� 40

v
■ Contents

Running Administratively and the SharePoint Management Console���������������������������� 41

Variable Instantiation����������������������������������������������������������������������������������������������������� 42
Objects as a Form of Troubleshooting���������������������������������������������������������������������������� 45
Avoiding Scrolling Truncation����������������������������������������������������������������������������������������� 51
Enumerating Sites���������������������������������������������������������������������������������������������������������� 53
Step 1��������������������������������������������������������������������������������������������������������������������������������������������������� 55
Step 2��������������������������������������������������������������������������������������������������������������������������������������������������� 55

PowerShell Script to Create Central Administration������������������������������������������������������� 57

PowerShell Script to Create Service Applications���������������������������������������������������������� 61
Building a Farm with AutoSPInstaller����������������������������������������������������������������������������� 72
MSDTC and DCOM Settings�������������������������������������������������������������������������������������������� 75
Network Service Permissions���������������������������������������������������������������������������������������� 82
Local Security for the Farm Account������������������������������������������������������������������������������ 82
Next Steps���������������������������������������������������������������������������������������������������������������������� 92
■
■Chapter 3: More Key Settings to a Good Build����������������������������������������������������� 93
COM+ Security for User Profile Sync����������������������������������������������������������������������������� 93
App Fabric and Distributed Cache�������������������������������������������������������������������������������������������������������� 94

User Profile Synchronization���������������������������������������������������������������������������������������� 105

Patching����������������������������������������������������������������������������������������������������������������������� 110
Publishing Infrastructure vs. Minimal Download Strategy������������������������������������������� 112
Account Management�������������������������������������������������������������������������������������������������� 113
Logging Locations and Levels�������������������������������������������������������������������������������������� 114
Path-based vs. Host-named Site collections��������������������������������������������������������������� 116
HNSC or HHSC�������������������������������������������������������������������������������������������������������������� 123
Next Steps�������������������������������������������������������������������������������������������������������������������� 130

vi
 ■ Contents

■
■Chapter 4: Files, Virtual Mappings, and IIS Settings����������������������������������������� 131
Got Weird Stuff?����������������������������������������������������������������������������������������������������������� 134
SharePoint IIS Site Directories������������������������������������������������������������������������������������� 138
Virtually Mapped Folders���������������������������������������������������������������������������������������������� 140
SharePoint Web Services��������������������������������������������������������������������������������������������� 143
What About Registry?��������������������������������������������������������������������������������������������������� 165
■
■Chapter 5: SQL��������������������������������������������������������������������������������������������������� 177
PowerShell������������������������������������������������������������������������������������������������������������������� 211
Configuring SharePoint-Integrated Reporting with SQL Server 2012/2014����������������� 215
Scenario 1������������������������������������������������������������������������������������������������������������������������������������������ 216
Scenario 2������������������������������������������������������������������������������������������������������������������������������������������ 217

■
■Chapter 6: SQL Backup and Restore and Useful CLI Commands����������������������� 239
Event ID 5586��������������������������������������������������������������������������������������������������������������� 255
■
■Chapter 7: Search Configuration and Troubleshooting�������������������������������������� 261
■
■Chapter 8: Service Application Troubleshooting����������������������������������������������� 327
■
■Chapter 9: ULS Viewer��������������������������������������������������������������������������������������� 371
■
■Chapter 10: Tools: Network Packet Tools and Page Performance��������������������� 401
Wireshark��������������������������������������������������������������������������������������������������������������������� 401
Fiddler�������������������������������������������������������������������������������������������������������������������������� 407
NetMon and Message Analyzer������������������������������������������������������������������������������������ 411
Developer Dashboard��������������������������������������������������������������������������������������������������� 414
Webalizer���������������������������������������������������������������������������������������������������������������������� 418
Indihiang���������������������������������������������������������������������������������������������������������������������� 423
SPS Farm Report utility������������������������������������������������������������������������������������������������ 425
Process Monitor (ProcMon)������������������������������������������������������������������������������������������ 428

vii
■ Contents

■
■Chapter 11: Tools: SharePoint Health Analyzer Demystified����������������������������� 439
SharePoint Health Analyzer Tool����������������������������������������������������������������������������������� 439
Performance Analysis of Logs (PAL) Tool for SharePoint���������������������������������������������� 442
SharePoint Feature Administration and Cleanup Tool�������������������������������������������������� 463
The SharePoint Manager Tool��������������������������������������������������������������������������������������� 468
Wrap Up������������������������������������������������������������������������������������������������������������������������ 471

Index��������������������������������������������������������������������������������������������������������������������� 473

viii
About the Author

Stacy Simpkins is a SharePoint engineer with Rackspace, the number-

one managed cloud company. He is passionate about SharePoint and
loves helping customers understand and get the most out of SharePoint.
Prior to Rackspace, Stacy worked with the federal government as an IT
specialist and across multiple industries (food, legal, manufacturing,
health insurance, and professional services) architecting and developing
small, medium, and large SharePoint environments as a consultant. As
a consultant, he served as a solutions architect for Magenium Solutions
and as a senior consultant for Sogeti LLC. Stacy holds numerous
Microsoft Certifications. During his limited free time, he enjoys blogging
about SharePoint and other Microsoft products, speaking at user group
meetings, and leading the Tampa Bay SharePoint user group.

ix
About the Technical Reviewer

Samarjeet Singh Tomar is a SharePoint Engineer for the Blue Cross Blue
Shield Association (BCBSA), a national federation of 36 independent,
community-based and locally operated Blue Cross and Blue Shield
companies. He is passionate about SharePoint and .Net Core, Tableau,
Angular, D3, Power-BI and helping customers and business in automate
and visualization. Prior to BCBSA, Samar worked with various industry
domains and service area. He is passionate about learning and
implementing different technology and build scalable solution using
proven practices. During his limited free time, he enjoys blogging about
SharePoint and other technologies, he loves travelling and playing
computer games.

xi
Acknowledgments

I’d like to thank my fellow Rackspace SharePoint engineers for their contributions: Scott Fawley, J. T. Shoupe,
Stephen Swinney, Danny Pugh, Mike Ross, Mike Clarke, Jarod Oliver, Daocheng Li (Richard), Mark Watts,
Ryan Holderread, Brad Slagle, and Tray Harrison. Originally, I had planned to provide a short bio of
everyone on this list; however, we weren’t able to pull them all together before printing. To everyone on
this list, I sincerely thank you for your fanatical support and the awesome SharePoint knowledge, and the
wisdom you’ve shared with me over the last year.

xiii
Introduction

This introduction covers, at a high level, the topics that this book discusses. The book assumes that you
already have a development SharePoint environment that you can use to perform the exercises. If you don’t
have a development farm and are not sure about the steps needed to create one, you should get a copy of
my book Building a SharePoint 2016 Home Lab: A How-To Reference on Simulating a Realistic SharePoint
Testing Environment (Apress, 2016). Although it is possible to read each chapter independently, there are
parts of chapters that build off previous chapters and/or assume some requisite SharePoint knowledge.
The following is the 40,000-foot view.

Chapter 1. Least-Privileged SharePoint Builds

This chapter thoroughly discusses building a SharePoint farm using least privileging. It starts to peel
away the troubleshooting onion, layer by layer, and explains why a least-privileged build is important for
troubleshooting.

Chapter 2. Key Settings of a Good Build

This chapter is the first of two parts that cover the key settings of a good build. You’ll learn about SQL aliases,
MSDTC, to IIS WAMREG and DCOM, Network Service, and the local security needs of a farm account.

Chapter 3. More Key Settings of a Good Build

This chapter finishes the discussion on key settings in the file system as they relate to App Fabric and
Distributed Cache, User Profile Synchronization, publishing infrastructure, account management, logging
locations and levels, and path-based vs. host headers, also known as host named site collections.

Chapter 4. Files, Virtual Mappings, and IIS Settings

This chapter explores the changes that SharePoint makes to a Windows server file system and discusses how
this relates to IIS. It looks at IIS logging and opens the discussion that surrounds the connection between IIS
logs, SharePoint logs, and Windows logs.

Chapter 5. Database and Security Operations

This chapter opens SQL Server Management Studio and looks at the SQL Server settings, database settings,
server roles, database mappings, SQL logging, and various PowerShell and/or command-line operations as
they relate to SharePoint database security operations from within SSMS and/or SQL Server configuration.

xv
■ Introduction

Chapter 6. SQL Backup and Restore, and Useful CLI

This chapter covers a few more SQL-related topics, such SQL database backup and restore options,
unattached restores, SQL file restores, and PowerShell site collection backup and restore. We look at some
Windows OS commands that yield helpful troubleshooting information, including systeminfo, ncpa.cpl,
msinfo32, SC, and others as I talk about finding answers to troubleshooting questions.

Chapter 7. Search Configuration and Troubleshooting

This chapter peels back a deeper layer of the troubleshooting onion as it relates to issues with search, search
configuration with PowerShell, and the search service application. We look at some cool scripts and take a
fairly good dive into search.

Chapter 8. Troubleshooting Services

This chapter looks at troubleshooting User Profile Synchronization Connections, Excel Services, Office Web
app connections, and patching Office Web apps. We look at managed metadata term stores and discuss
the connection to the User Profile Service. I’ll discuss web.config modifications and using PowerShell to
determine if the web.config is modified. Along with looking at web.config, PowerShell interrogates timer
jobs, log levels, and databases. Finally, PowerShell is used to unprovision and provision services.

Chapter 9. Tools: ULS, merge-splogfile, and Other

PowerShell cmdlets
This chapter’s primary focus centers on ULS logs, ULS viewer, merge-splogfile, and other PowerShell
cmdlets that pertain to Windows logs. It discusses the numerous settings of ULS viewer and some various
scenarios and methods. The chapter explains the connection between SharePoint and Windows event logs
and helps the reader understand how to decipher what the logs are saying and how to use the logging system
and configure it.

Chapter 10. Tools: Network Packet Tools and Page

Performance
This chapter discusses the use of ProcMon, WireShark, Fiddler, NetMon, developer dashboard, and more! It
also covers a few more tools used to look at network packets, IIS logs, and page load performance.

Chapter 11. Tools: SharePoint Health Analyzer Demystified

This chapter discusses the SharePoint Health Analyzer report, the Performance Analysis of Logs (PAL) tool
for SharePoint, the SharePoint Manager tool, the SharePoint feature admin tool, and finally, a summation of
the three chapters on troubleshooting tools.

xvi
 ■ Introduction

Commonly Used Shortcuts

In this book, we use keyboard shortcuts, the run bar, and commands quite a bit. Table-A lists some of the
commands with a brief description.

Table-A. Keyboard Shortcuts and Commands Used in This Book

Command\Keyboard Shortcut Description of Run Command

Windows key + R Opens the run bar
Cmd Opens the Command window
Comexp Opens the Component Services manager
Compmgmt.msc Opens the Computer Management console
ipconfig Opens the ipconfig information
nslookup Opens a command-line interface to DNS
Ncpa.cpl Opens the network connections
Regedit Opens the registry editor
Control netconnections Opens the network connections
Msinfo32 Opens the system information
Sysdm.cpl Opens the system properties
Services.msc Opens the Services console
Dsa.msc Opens the Active Directory users and computers
Dnsmgmt.msc Opens the Domain Name System manager
Gpmc.msc Opens the Group Policy Manager
Control Panel Open the control panel
Lusrmgr.msc Open the Local Users and Groups administration console
Notepad Opens Notepad
Adsiedit.msc Opens the Active Directory Service Interface editor

Summary
The goal of this book is to provide you with a much broader troubleshooting arsenal for SharePoint and
perhaps a deeper understanding of how the file system relates to the databases. We do not delve into
unsupported activities, such as table modifications, as that would not be in best practice; however, there are
a couple points in the book where we come close, as we look into certain tables inside the SharePoint SQL
Server database tables. No animals were hurt during the making of this book and all of the tools you see used
in this book are available free of charge and are downloadable on the Internet.

xvii
CHAPTER 1

Least-Privileged SharePoint Builds

Why Least Privilege

In this chapter, you’re introduced to least-privileged SharePoint builds. It is important to understand the
components of a least-privileged build because it aids in troubleshooting the odd behaviors that can arise
when builds that were once least privileged have been modified. Least-privileged SharePoint builds follow
the best practice recommendations of Microsoft, and as a result, offer better performance.
As you read through Chapter 1 (and the entire book), you don’t need to have a SharePoint environment
to follow along; but it would definitely be a plus and you’ll get more out each chapter and the chapter
exercises, if you have a farm. If you don’t have a farm and do not know how to build one, you should
purchase a copy of my book Building a SharePoint 2016 Home Lab: A How-To Reference on Simulating a
Realistic SharePoint Testing Environment (Apress, 2016). This book moves along at a little slower pace than
the book in your hands. With that said, let’s get going.

An Ounce of Prevention Is Worth a Pound of Cure

Knowing if a farm is least privileged is often half the battle in troubleshooting various issues with SharePoint.
When SharePoint is installed using an administrative account, a common mistake is to use the same account
for all services. This happens when the same account that is used to install or set up SharePoint is also used
to access or connect to the databases that are stored on SQL Server. The account used to access the SQL
databases is known as the farm account, which should not be a local administrator.

■■Note The only time the farm account is a local administrator is during a User Profile service setup
and configuration.

It’s really easy to make the mistake of using the install account for the farm account. The post setup
Configuration Wizard (psconfiggui.exe) prompts for the farm account. This is where that “ounce of planning
is worth a pound of cure,” because even though there are blogs and TechNet forums posts that advise on
methods of how this account can be modified after the fact, it is always cleaner, and in your best interest, to
plan a farm account separate from the install account—before installing SharePoint.
Once the setup account has been erroneously given as the farm account, and the databases are created,
the cat is out of the bag. The best way to correct this is too start with a fresh build. There are a couple of
methods that you can use to determine if the farm you’re working with is over-privileged. Method number
one is the Windows operating system’s Services console.

© Stacy Simpkins 2017 1

S. Simpkins, Troubleshooting SharePoint, https://doi.org/10.1007/978-1-4842-3138-8_1
Chapter 1 ■ Least-Privileged SharePoint Builds

For example, if you open the services console (services.msc) and notice that all the SharePoint services
are running under an account that looks like the farm account (say, something like 2013Farm), it’s probably
a safe bet that you’re not working with a least-privileged farm. Figure 1-1 shows a farm that was installed in
an over-privileged fashion.

Figure 1-1. Farm account used as the identity for all services

The only Windows operating system service related to SharePoint that the farm account should run
is the SharePoint timer service (SPTimerV4). The farm account should not be used to run the SharePoint
administration service (SPAdminV4) since this service performs automated changes that require local
administrator permission on the server.
The farm account would never be used to run the search services, as this would be worse than using the
search service administration account as the crawler account. In both cases, SharePoint search results would
include unpublished versions and would show these versions in search queries to users who shouldn’t
be able to read them until they were published. This is why we always use a search service account for the
SharePoint Search Host Controller service (SPSearchHostController) and for the SharePoint Server Search
15 Service (OSearch15). A separate SharePoint service account is then used as the default content account,
otherwise known as the crawler, or crawl account.
If you’ve never least privileged a SharePoint environment, you’re probably starting to see that it is not
as easy as just inserting the binaries and running the Configuration Wizard to completion, and possibly
the farm Configuration Wizard, all with the same login account. As I mentioned earlier, this is a common
occurrence, and one that is easily rectified by a farm rebuild using PowerShell scripts to build the farm and
provide the least-privileged access.
So what do to if you’re seeing an account listed for most of the services, you can make sure that this is
the case by running the following PowerShell:

(Get-SPFarm).DefaultServiceAccount.Name

This one-liner returns the farm account. If the two match up, then it’s up to you to determine how to go
about least privileging the farm.

2
 Chapter 1 ■ Least-Privileged SharePoint Builds

Figure 1-2 shows the results of running the PowerShell one-liner.

Figure 1-2. defaultServiceAccount is the farm account

You might be dealing with a farm that has many solutions deployed. These solutions might not like
having to live in an environment where they cannot run in some form of “over privilege.” Before completely
throwing out the seemingly over-privileged build, you should dig a little deeper and open IIS Manager
(inetmgr.exe). Once you have Internet Information Services (IIS) Manager open, the identities that the
application pool accounts are using will give another indication of whether the environment is somewhat
least privileged, or if it is possibly over-privileged to some extent. In other words, the Windows operating
system Services console and the PowerShell one-liner are not the end-all/be-all decision makers deciding
whether the farm is too bad off from a least-privileged standpoint.
If you open the IIS Manager and see something similar to Figure 1-3, there was an attempt to least
privilege the farm, and it may be salvageable. You might be able to adjust the various service identities using
Central Administration and/or PowerShell, and be completely fine.

Figure 1-3. IIS Manager shows signs of least privilege

I say “maybe” because if the same account used to install SharePoint was used for the farm account, my
experience has shown me that it is best to rebuild this type of farm. If you know for certain that that was not
the case, then you should proceed with looking at the rest of the least-privileged settings—before making
your determination. If you’re not sure, there’s another troubleshooting step to possibly yield the desired
results; these are to determine what has happened to the farm that is exhibiting some form of over-privilege.
Hopefully, it is not due to the setup account erroneously used as the install and the farm account.
3
Chapter 1 ■ Least-Privileged SharePoint Builds

The account that was used to run the Configuration Wizard is the owner of both the Central
Administration and the configuration databases in SQL. This account should not be the farm account. The
farm account is the account that should be running the SharePoint Timer Service and the identity that the
Central Administration web site is running with when looking at the application pools within IIS Manager.
I know that I’ve said that a couple of times, but it is very important to drive this point into the root of your
SharePoint least privileging knowledge.
Figures 1-4 and 1-5 show that an account other than 2013Farm was used to create the farm’s Central
Administration and configuration databases.

Figure 1-4. Central admin content database is owned by the installer, as are all databases

Figure 1-5. The configuration database is owned by the account used to install or set up SharePoint

4
 Chapter 1 ■ Least-Privileged SharePoint Builds

This means that the farm account that runs the Central Administration site in Figure 1-3 was not used as
the setup account.
From looking at the accounts used to run the SharePoint services in Figure 1-1, there is more work to
be done to get this farm to a least-privileged state; and we still have not decided if the farm is going to need a
rebuild, as we haven’t looked at the SQL database logins, SQL settings, registry permissions, or any of the file
system permissions. One thing is certain, though: we have determined that the farm was not installed with
the farm account. A setup or install account was used, and so far we know that various Windows SharePoint
Services are running over-privileged.
The identities used by the various application pools in IIS look legit. That is, they look as if they are least
privileged. We noticed that the application pool that hosts most of the SharePoint service applications is
running under a different account than the application pool that serves the content to the web application
that hosts the host named site collections. This is because the method that installed this farm utilized
PowerShell to create the application pool that hosts the SharePoint service applications. A little later in this
chapter, we’ll look more deeply at IIS Manager, the identities used to run the various application pools, and
some of the various file locations that SharePoint reaches into from within IIS.

Local Group Membership

The only IT service account that should be a member of the local administrators group on any server in
the farm is not a SharePoint service account at all; it is the SharePoint install or setup account. It is often
thought of as a service account because it is used to perform administrative functions in SharePoint, such as
installing the farm and performing the initial configuration. This setup account is needed to set up the farm
up in a least-privileged fashion.
Earlier, I mentioned the farm account needing local administrator membership for the configuration of
the User Profile service and I forgot to mention that after the User Profile service application is configured
and the User Profile synchronization service is synchronizing, that the farm account should be removed
from the local administrators group on all servers in the farm. It is OK to leave the setup account in the local
administrators group to log in administratively and to set up new service applications and perform other
administrative duties.
Speaking of local groups, SharePoint creates three of them during installation and the farm account is
added to all three of these groups. When providing a consultant with farm account-esque access to your farm,
remember that the consultant’s account does not and should not be added to the WSS_RESTRICTED_WPG_V4
local group, as this group should only contain the farm account. If you’re looking at a farm for least privilege
and you notice accounts other than the farm account have membership in the WSS_RESTRICTED_WPG_V4
local group, chances are good that there is some over-privileged code running somewhere in this farm. If the
code is properly written, it should not be necessary to modify this group.
When a SharePoint farm is created, the account that is entered into the Configuration Wizard
(psconfiggui.exe), as the farm account, is automatically added to each of the following groups:
• WSS_ADMIN_WPG
• WSS_RESTRICTED_WPG_V4
• WSS_WPG
This automatic group population actually happens during setup; and then each time that a server is
joined to the farm, via the Configuration Wizard, or via the command-line psconfig.exe or PowerShell. The
setup user account is automatically added to
• WSS_ADMIN_WPG
• IIS_WPG

5
Chapter 1 ■ Least-Privileged SharePoint Builds

It also has elevated privileges in SQL Server, as does the farm account, but with a slight twist that I’ll
discuss in just a minute. If you ever notice a disparity in the accounts in these groups, there are really only
three ways that this can happen. The first is that the server has gremlins in it. The second is that someone
manually modified the membership. Finally, the third is via code or solution deployment. I like the first way
because it is the most common explanation.

Ask the Domain Controllers

If you ever encounter a farm with disparity between the three Windows SharePoint Services worker
process groups, you should start asking questions. If you see a user that does not belong in the group,
you should ask is when the user was added. You can open the domain controller and look at the security
logs for event ID 47—a member was added to a security-enabled local group. You can do this manually
using the event viewer (eventvwr.msc), or you can use a totally awesome piece of PowerShell that a good
friend of mine, Mr. J. T. Shoupe, a fellow SharePoint engineer at the world’s number-one managed cloud
company, Rackspace, introduced to me.

$spservers=Get-SPServer | where {$_.Role -ne "Invalid"}

foreach($spserver in $spservers)

$filename=$spserver.name
write-host ------------------------- $filename -------------------------
get-winevent -FilterHashtable @{Logname='System';ID=5138} -MaxEvents 3 | select TimeCreated,
ID, Message

In this example, J. T. was looking for instances where the IIS web server was unable to communicate
with the Windows Process Activation Service (WAS). Because application pools depend on WAS to function
properly, you may have to restart the application pool on a schedule if you see a lot of 5138 event IDs. The
real point I’m trying to make here is that the part of the script that reads ID=5138 could easily be changed to
4732, and the part that reads Logname=‘System’ could be replaced with Logname=‘Security’ if you wanted
to scour the security log for event ID 4732. You can always look for more than three events by changing –
MaxEvents 3 to –MaxEvents 4, or a number higher than 3.
The way to use this PowerShell is to open a SharePoint Management Shell and paste it in after you’ve
adjusted it for your logname, ID, and MaxEvents. Don’t worry if you don’t understand all the PowerShell
at the moment; in an upcoming chapter, we’ll dig into PowerShell a little bit further and look at how it has
some really awesome troubleshooting capabilities. Let’s keep talking about “the who” part of this query.
Another question that can be answered by the domain controllers logs is when the local security group
was changed, searching for event ID 4735. It might even tell you who made the change. Chances are good
that the change was made by a service account, which narrows the “who-done-it” to those people who have
or had access to the passwords. Hopefully, that was or is a small list.
Solutions could be written in such a way that they modify the membership of local groups. You can use
a list of deployed solutions to find yourself a good starting point for the search in the domain controllers to
determine if any group memberships were changed at the same time or right around the time of a solution
deployment. To get such a list, manually click through each deployed solution to look at the last time it was
deployed, or use this PowerShell:

Get-SPsolution | sort lastoperationendtime | ft name, lastoperationendtime

6
 Chapter 1 ■ Least-Privileged SharePoint Builds

The use of the sort-object cmdlet is purposefully left at the default of ascending so that the most recently
deployed solutions are at the bottom of the list that is generated. This gives you a timeline of when solutions
were deployed. Then you can use J. T.’s script to determine if any local group memberships changed around
the same time.
It is a good idea to have all the solutions in your farm documented with what they do and what changes
they make to the file system, registry, IIS, and so forth. Most governance documents specify that each
solution should be thoroughly documented in such a way that the “hit by a bus” theory is protected. Not that
I’d wish any developer to get run over by a bus, or hit by one, or backed over by one, because that would not
be good. It would also “not be good” to have an undocumented solution make unwanted changes to security
groups, service identities, and or application pool identities.

Database Permissions for Farm Account Vs Install Account

In SQL Server, there’s a login known as sysadmin or SA, which is, for the most part, the god of SQL. Then,
there are accounts that have the fixed server role of sysadmin; not to be confused with SQL login SA. And
finally, there are accounts that have both db_creator and securityadmin. When an account has db_creator
and securityadmin, it essentially is the same as having a public login and sysadmin. The farm account
that is used to connect to the databases is given db_creator and security admin during the initial farm
configuration; and for the farm to function, these fixed server roles should remain after the farm is created.
The farm account is not a member of the local administrators group on SQL Server.
The install account is a member of the local administrators group on every application, web front
end, distributed cache, search, and SQL Server in the farm. The install account also has db_creator and
securityadmin fixed server roles. Both accounts have db_owner of the server farm configuration database
and of the server farm Central Administration content database. The install or setup account needs to
be able to log in to the computer running SQL Server in order for the install configuration wizards or
PowerShell cmdlets to succeed.
After the farm is created, the farm account has db_owner on every SharePoint database. With
SharePoint 2013, a manual change is required for the Performance Point database, wherein the db_owner
has to be manually added.
The final difference between the farm account and the install account is that the farm account has
membership in the WSS_CONTENT_APPLICATION_POOLS role for the configuration database and for
the Central Administration content database. Membership in this role gives the farm account elevate
permissions to a subset of stored procedures.

File System Permissions for Members of the WSS_Admin_WPG

Local Group
This section discusses a few file system paths that the WSS_Admin_WPG local group has, for the
most part, full control over. Oddly enough, a file system path that this group does not have full
control over, but instead can only modify, is the infamous root folder of the hive, which is located
at %COMMONPROGRAMFILES%Microsoft Shared\Web Server Extensions\15, with the path
%COMMONPROGRAMFILES% = c:\program files\common files. This is the directory for the core
SharePoint 2013 files. In SharePoint 2010, the path is %COMMONPROGRAMFILES%Microsoft Shared\Web
Server Extensions\14.
If the access control list (ACL) is modified for this folder in any way, all sorts of things start to go haywire;
for example, solution deployments do not function properly or a feature activation fails or does not activate
correctly. Figure 1-6 shows the contents at the root of the hive. I’ve always found it strikingly odd that the
members of WSS_Admin_WPG can only modify at this folder level when the group has full control over a
plethora of other Windows system folders and only a few of the hive’s subfolders. As you read on, pay special

7
Chapter 1 ■ Least-Privileged SharePoint Builds

attention to which folders inherit their permissions from the 15 hive, so that if you ever need to determine if
manual changes were made to the file system permissions, you’ll have a good starting point.

Figure 1-6. The 15 hive folders

The directories directly beneath the hive that inherit and only allow the farm account to modify these
directories and all the subfolders and files are as follows: BIN, client, HCCab, Help, ISAPI, Policy, Resources,
Template, UserCode, WebClients, and WebServices.
Of the folders that inherit permissions directly from the root of the hive, the BIN folder is one of the
most heavily accessed folders because it contains the OWSTIMER, PSCONFIG, SPMETAL, WSStracing, and
WSSAdmin files. There are a lot of other .dll and .exe files in this folder that are responsible for supporting
SharePoint. The local service on each server has read\execute on this directory. If this directory is modified,
parts of SharePoint will start to fail; and if it is removed, SharePoint will break.
The local service also has read rights to the key in registry that contains the document conversion service.
The Client folder contains files for the support of Microsoft Online; whereas, the HCCab folder contains
.cab files that are broken down in such a way as to represent the various languages installed in the system;
they are also used in the help system. Speaking of the help system, the Help folder holds a compiled HTML
file that serves the SharePoint Help system.
When looking at IIS, you’ll notice that some of the folders have a shortcut icon but other folders do not
have the icon. The folders with the shortcut icon are virtual folders that map to various locations within
the global assembly cache (GAC). GAC is a term used to describe areas on the file system that hold key
SharePoint files. The ISAPI folder is part of this GAC that contains numerous web service (.asmx) files known
as web service discovery pages (.aspx) files. The ISAPI folder also is home to dynamic link library (.dll) files

8
 Chapter 1 ■ Least-Privileged SharePoint Builds

that support the operations for SharePoint that are handled through web services. The ISAPI folder has a
shortcut icon in IIS because it is a virtually mapped folder; that is, its files do not reside under the
default %SystemDrive%\inetpub\wwwroot\wss\VirtualDirectories location; but instead, they live inside
C:\Program Files\Common Files\Microsoft Shared\Web Server Extensions\15\isapi and are mapped in IIS
to the virtual folder named _vti_bin.
The Policy folder also inherits from the root and it contains files that redirect assemblies. Different
versions of SharePoint support different levels of redirection; for example, SharePoint 2013 supports the
redirection of SharePoint 2010 and 2007 assemblies.
The Resources folder contains .resx files that are used to localize SharePoint. In other words, these files
are used to represent different languages. The default install of SharePoint has the base set of files that do not
have a language identifier, and then, for the most part, a corresponding file that has the language identifier.
For example, core.resx, which contains descriptions for web parts, is accompanied by core.en-US.resx. I said
“for the most part” because some files do not have language agnostic files. These resource files are copied
by language packs as you add them. The default install of SharePoint is in English. It is a really good idea to
never modify these files manually. The same is true with most IIS settings and changes made in the Windows
Services console. We need to allow SharePoint to handle these changes as much as possible. Sometimes,
we’ll need to take things into our own hands, but hopefully, this is not very often.
The TEMPLATE folder is where you’ll find the most development taking place. I’d wager this folder
and its subfolders, FEATURES and IMAGES, are the three that are most heavily targeted by developers. The
TEMPLATE folder has folders inside it that support customizations made to the farm. The TEMPLATE folder
also has a plethora of folders that contain out-of-the-box SharePoint features and core files for SharePoint
sites. Modifications to ACLs on this folder cause odd behavior within SharePoint. The ADMIN subfolder
contains the master pages and templates for the Central Administration web site, along with other core
features for Search, Secure Store Service, Business Connectivity Services, and content deployment. The
LAYOUTS subfolder contains a plethora of files that are used for all sorts of administrative actions within
SharePoint sites. Whenever you’ve navigated to site settings or site content, you have accessed files inside of
the LAYOUTS subfolder. The virtual directory, which is exposed inside IIS, is named _layouts.
The TEMPLATE folder is also home to the CONTROLTEMPLATES subfolder, which contains files that
are used in list item forms. These templates control the layout of the list item forms. Along the same line of
thought, there is a subfolder under the TEMPLATE folder named DocumentTemplates, which houses a file
named wkpstd.aspx. The wkpstd.aspx file is used to create document libraries; so, if you’re having trouble
creating document libraries, check that the ACL of the DocumentTemplates folder has not been changed
and that the date of the wkpstd.aspx is not recent. A recent date on this file could indicate a modification that
should not have been made.
When you create copies of sites in the form of site templates, the SiteTemplates folder is used. It
contains the base files used in the process of creating a site template for blogs, team sites, wiki sites, meeting
workspaces, Tenant Administration, and Central Administration. Table 1-1 summarizes the site templates
that are available in different versions of SharePoint On-Premises and SharePoint Online.

9
Chapter 1 ■ Least-Privileged SharePoint Builds

Table 1-1. Available Site Templates

Category Site Type Site Site Office Office 365 SharePoint SharePoint SharePoint
Collection 365 for Server Server 2013 or Online
for small medium Foundation SharePoint
business or large 2013 Server 2016
business
Collaboration Team Yes Yes Yes Yes Yes Yes Yes
Collaboration Blog Yes Yes Yes Yes Yes Yes Yes
Collaboration Project Yes Yes Yes Yes No Yes Yes
Collaboration Community Yes Yes No Yes No Yes Yes
Enterprise Document Yes Yes No Yes No Yes Yes
Center
Enterprise Records Yes Yes No Yes No Yes Yes
Center

The TEMPLATE folder’s IMAGES subfolder contains shared files that are shared by all the SharePoint
web applications on the server. These files are image files and they are accessible by the _layouts/images
virtual directory. There is a subfolder of the TEMPLATE folder named SQL that contains stored procedures
for SQL Server. There is a subfolder named THEMES under the TEMPLATE folder that provides the files used
in SharePoint themes. Knowing this is important when troubleshooting issues with any of these.
The WorkflowActivities subfolder contains only one .dll file; so, if there are workflow issues, you can
easily rule out the file system as the issue by checking the subfolder for a file named Microsoft.SharePoint.
WorkflowServices.Activities.dll, which has the same date on all of the servers in your farm.
The XML subfolder contains XML files that provide support for the files used to render some of the
SharePoint field and schema definition, which helps with the look and feel by mapping the JavaScript files
used by the different actions in SharePoint. This folder gets enhancements and the addition of field types
and definitions, which are added by SP, CU, and/or platform additions; for example, Project Web app (PWA)
and SQL Server Reporting Services (SSRS) integration adds more XML files to this folder.
By no means does this do justice to the awesome power of the files that I just mentioned. There is a
reason that all the directories inherit—with the exception of the ADMISAPI, CONFIG, and Logs directories.
One of the reasons is that it makes it hard for code to perform any sort of action that would alter ACLs, which
is intentional because changes to ACLs in the SharePoint hive can have detrimental impacts.
The UserCode folder under the root of the hive inherits its permissions, giving the farm account only
modify, as it contains files used in support of sandboxed solutions. The WebClients Folder has numerous
subfolders that contain .config files for client settings for various service applications and services within
SharePoint. If one of them is different from the next, this might result in inconsistent behavior in a service
application. There may be modifications to one of the servers in a load balanced farm. The WebServices
folder contains web.config files for the application root in a subfolder named root. It has web.config files for
quite a few of the key service applications. In an upcoming exercise, you’ll see that the WebServices folder
houses web.configs for Secure Store Service, Topology Services, PowerPoint Conversion, BCS, Subscription
Settings, and Security Token.
Now that we’ve covered the directories that inherit from the hive, let’s talk about one of the directories
that does not inherit its permission from the hive: the ADMISAPI directory. This directory contains files
related to SOAP services for the Central Administration site. The members of the WSS_ADMIN_WPG
group have full control over this folder, its subfolders, and files. If your farm is exhibiting issues with remote

10
 Chapter 1 ■ Least-Privileged SharePoint Builds

site creation, or if it is experiencing weird behavior, such as things sometimes working and sometimes
not working, take a look at the directories access control list and look for any changes. Later, in one of the
exercises, you’ll notice that this folder is mapped in IIS to the _vti_adm virtual folder within IIS. The default
permissions on the file system folder are shown in Figure 1-7. Notice how some are inherited and some are
explicitly given.

Figure 1-7. ADMISAPI default permissions

The CONFIG directory also affects IIS and how web applications behave (as far as provisioning
is concerned. The CONFIG folder has files that are needed for a lot of different SharePoint operations,
including upgrade mapping operations where objects are mapped from one version of SharePoint to the
next—with 2010 to 2013 and 2013 to 2016. If the ACL shown in Figure 1-8 is altered, the problems with web
application provisioning will arise. The same is true if the contents of this directory are modified.

11
Chapter 1 ■ Least-Privileged SharePoint Builds

Figure 1-8. CONFIG directory default permissions

As you’ll notice in the exercises that wrap up this chapter, membership in the local administrators group
grossly changes the number of privileges an account or service that runs under that account possesses.
This is why the farm account is removed from the local administrators group after a User Profile service is
configured in SharePoint 2010 or 2013; it is not even required in the local admins group in SharePoint 2016
due to the changes in the FIM (forefront identity manager) service.

Logging File Paths

The default directory for the SharePoint Unified Logging System is located in %COMMONPROGRAMFILES%\
Microsoft Shared\Web Server Extensions\15\LOGS, or, in other words, C:\Program Files\Common Files\
microsoft shared\Web Server Extensions\15\LOGS. Just change the 15 for a 14 if working with SharePoint
2010, or to a 12 if working with SharePoint 2007. It seems as if Microsoft doesn’t like the number 13, or
maybe the SharePoint team didn’t since they skipped right over it when going from SharePoint 2007’s hive to
SharePoint 2010.
It’s a best practice to move as much logging and writing off the OS drive as possible. This logging
directory is able to be relocated after a build is completed; whereas, some of the SharePoint directories
cannot be relocated once the farm is up and online. Only at install, can you move the parent directory,
located at %ProgramFiles%\Microsoft Office Servers\15.0, by opting to change the drive location during the

12
 Chapter 1 ■ Least-Privileged SharePoint Builds

install. Figure 1-9 shows the SharePoint 2010 install. All you need to change is the C:\ drive to a D:\ or E:\
drive. This is a one-time event, and if you exercise this option, all future servers need to have the requisite
D:\ or E:\ drive. If you move one, you might as well move both: to move the search index location and in case
you decide to expand your search topology in the future.

Figure 1-9. Default file paths one-time only move option

I do not want to confuse the files that are located underneath %ProgramFiles%\Microsoft Office
Servers\15.0 or %ProgramFiles%\Microsoft Office Servers\15.0\Logs with the location of the ULS logs. As I
stated earlier, the ULS logging default location is under the Hive\Logs folder. Since this is defaulted to the OS
drive, it’s a best practice to move ULS logging to D:\ or E:\. This can be done via PowerShell, which we’ll look
at later on in some exercises in an upcoming chapter.
%ProgramFiles%\Microsoft Office Servers\15.0\Logs is where runtime diagnostic logs are generated
(not stored). If you’re having trouble with logging, check that the ACL has not been modified and that the
WSS_ADMIN_WPG local group has explicitly applied full control over this folder and its subfolders and files.
Not to confuse things, but the same is true about the permissions for wherever the ULS logs are writing; that
is, the WSS_ADMIN_WPG has full control over that location as well.

13
Chapter 1 ■ Least-Privileged SharePoint Builds

The files and folders underneath %ProgramFiles%\Microsoft Office Servers\15.0 include a

directory named WebServices. If you’re having trouble with services such as search or Excel, you
should check this directory for any “tomfoolery”—err, modifications. The Data directory (located at
%ProgramFiles%\Microsoft Office Servers\15.0\Data) is the root of all search functionality, so if you’re
having search troubles, make sure that this folder’s ACL has not been modified.
The WSS_ADMIN_WPG group has full control over the default location for IIS web sites in SharePoint.
This is located at C:\Inetpub\wwwroot\wss. If you’re having trouble with administrative functions that
enumerate or make changes to sites and subsites, make sure that this directory has not been altered.
Speaking of site resolution locations, the WSS_ADMIN_WPG local group has read/write capability at the
location of the Windows operating system’s HOSTS file. Since the dawn of time, this location has always been
located at %windir%\System32\drivers\etc\HOSTS.
Finally, the WSS_ADMIN_WPG can provide access to and has full control over the SharePoint cache,
including the Cache config, which is a subfolder of %AllUsersProfile%\ Microsoft\SharePoint. The config
folder in use is a GUID named folder underneath %AllUsersProfile%\ Microsoft\SharePoint\config;
inconsistencies can arise here that may cause timer jobs to fail. Clearing this cache location may also solve
issues with psconfig.exe and psconfiggui.exe when either of them receives errors.

Registry Permissions
The WSS_ADMIN_WPG local group provides elevated access to various locations in registry that perform
and house critical settings. These are not to be altered, as issues will arise.
If you’re having trouble with document conversions, check that the WSS_ADMIN_WPG group has only
read and write permissions over the following locations:
• HKEY_LOCAL_MACHINE\Software\Microsoft\Office Server\15.0\
LoadBalancerSettings
• HKEY_LOCAL_MACHINE\Software\Microsoft\Office Server\15.0\
LauncherSettings
The local system has to have read permissions on the LauncherSettings for document conversions to
work. The Local SYSTEM also has full control over the following registry location for machines to join the farm:

HKEY_LOCAL_MACHINE\Software\Microsoft\Shared Tools\Web Server Extensions\15.0\Secure

If you’re having trouble provisioning services, check that this key has not been altered and make sure
that LOCAL SYSTEM and WSS_RESTRICETED_WPG have full control:

HKEY_LOCAL_MACHINE\Software\Microsoft\Shared Tools\Web Server Extensions\15.0\Secure\FarmAdmin

If you’re having trouble with joining a server to the farm or with general SharePoint functions, check
that the WSS_ADMIN_WPG group has full control over the following locations:
• HKEY_LOCAL_MACHINE\Software\Microsoft\Shared Tools\Web Server
Extensions\15.0\Secure
• HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Office Server\15.0
If you’re having trouble with search, check the WSS_ADMIN_WPG group for full control over these locations:
• HKEY_LOCAL_MACHINE\Software\Microsoft\Office Server\15.0\Search
• HKEY_LOCAL_MACHINE\Software\Microsoft\Shared Tools\Web Server
Extensions\15.0\Search

14
 Chapter 1 ■ Least-Privileged SharePoint Builds

If SharePoint is behaving oddly, check that the WSS_ADMIN_WPG has read permissions at this location:

HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Office Server

If you’re having trouble opening Central Administration or with odd logging behavior in your farm
account, check this location for congruency on all of your servers:

HKEY_LOCAL_MACHINE\Software\Microsoft\Shared Tools\Web Server Extensions\15.0\WSS

Application Pool Accounts

Application pool accounts in IIS are automatically added to the WSS_WPG local group by SharePoint when
application pools are created using the GUI or via PowerShell. The WSS_WPG group has read and execute
access on the following directories:
• C:\Inetpub\wwwroot\wss
• %ProgramFiles%\Microsoft Office Servers\15.0
Members of the group have the ability to modify the contents of web.configs and to make changes to
sites that do not involve permissions; members also have access to the server-side SharePoint binaries that
are not located in the hive.
The application pool accounts have read access on the following location:

%AllUsersProfile%\ Microsoft\SharePoint

The following gives application pools the ability to interact with the files in the configuration cache,
among other files that are located under this directory:

%ProgramFiles%\Microsoft Office Servers\15.0\WebServices

If you are experiencing issues with services such as search or Excel, it is important to check the
WebServices directory to make sure that the WSS_WPG group has read access.
The application pool accounts have read access on the following hive locations, including all
subfolders and files:
• %COMMONPROGRAMFILES%\Microsoft Shared\Web Server Extensions\15\ADMISAPI
• %COMMONPROGRAMFILES%\Microsoft Shared\Web Server Extensions\15\CONFIG
And if you’re having the type of troubles that I explained in the WSS_ADMIN_WPG section, you need to
keep these two directories in mind for the WSS_WPG group with read access.
Finally, the WSS_WPG group has modify permissions on the ULS logging location. If logging is not
happening, make sure that this group has the proper permissions on the following location (when using the
default logging location):

%COMMONPROGRAMFILES%\Microsoft Shared\Web Server Extensions\15\LOGS

15
Chapter 1 ■ Least-Privileged SharePoint Builds

WSS_WPG Registry Access

The application pool accounts that are members of the WSS_WPG group have read access on the following
registry locations:
• HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Office Server\15.0
• HKEY_LOCAL_MACHINE\Software\Microsoft\Shared Tools\Web Server
Extensions\15.0\Secure
• HKEY_LOCAL_MACHINE\Software\Microsoft\Shared Tools\Web Server
Extensions\15.0\WSS
If the 15.0\Secure is modified, you might find it very difficult to add a machine to the farm, or to run
SharePoint. If the 15.0\WSS is altered, diagnostic logging will probably fail and there might be issues with
adding servers to the farm or running the Configuration Wizard and/or its command-line equivalent.
The application pool accounts have both read and write access at the following locations:
• HKEY_LOCAL_MACHINE\Software\Microsoft\Office Server\15.0\Diagnostics
• HKEY_LOCAL_MACHINE\Software\Microsoft\Office Server\15.0\
LoadBalancerSettings
• HKEY_LOCAL_MACHINE\Software\Microsoft\Office Server\15.0\
LauncherSettings
The same is true for these locations, as was true for the WSS_ADMIN_WPG group, with respect to the
difference in each group’s permissions, as far as what can happen if modified or altered (e.g., problems with
diagnostic logging and/or issues with the load-balancer-handling document conversion).

Application Pool Accounts in IIS

Least-privilege SharePoint service applications app pool accounts are not local administrators on the box;
they should not need to be farm admins or to have WSS_Admin_WPG membership. If SharePoint service
applications app pool accounts need membership in any of the these, then the code only works if it has
greater than least privileging, as you saw with what the WSS_Admin_WPG group can do to the file system.
When a content web application is created to store site collections, a managed account is used to run the
application pool. Using the regular domain user account that does not have membership in any elevated
group gives the content web application least privileging, because this account is not used to run Windows
services as service applications; they only run the application pool that serves the web applications that
houses site collections, sites, subsites, and content.
Speaking of service applications and SharePoint, it is a best practice to break out the Search Service
application into its own application pool and then to run the other SharePoint service applications under
a Shared Hosted Services application pool. In a truly least-privileged farm, the Secure Store Service
application has its own service application pool; it is not hosted inside the same application pool as the
other service applications. The Shared Hosted Services application pool houses all of the various service
applications in SharePoint and it runs under a managed account named something along the lines of
2013svcapps, or SP_SA_AP for SharePoint service applications application pool account. When a service
application is created, the account used to run the service application is automatically added to the
WSS_WPG local group. It is given different permissions within SQL, depending on the service applications
that uses it.
When the Shared Hosted Services application pool is created, the account used to run it is
automatically assigned the SP_DATA_ACCESS role for any existing content databases. It is assigned the
WSS_CONTENT_APPLICATION_POOLS role associated with the farm configuration database and with the
Central Administration content database.

16
 Chapter 1 ■ Least-Privileged SharePoint Builds

When it comes to SharePoint 2013, the IIS application pool that houses the search service application
uses a regular domain user that is not a member of domain admins, local admins, WSS_Admin_WPG, or
any elevated group. When the search service application is created in SharePoint 2013, part of that process
should create a separate application pool in IIS that uses a special service account for search, usually named
something like SP_search. Please note that this is not the default content access account. The default access
account is also called the crawl account, which is used to access content in all the sites by being manually
assigned full read permissions on the web applications that host the sites.
The Excel services unattended account is a regular domain user that is used in conjunction with the
secure store service to create the unattended service account and allow Excel services in SharePoint 2013 to
contact external content from data sources that require a user name and password. This account must be a
domain user account and must not be a member of the local administrators group.
The My Sites application pool account is another regular domain user account that has no
administrative privileges on the local server other than membership in WSS_ADMIN_WPG. It is
automatically added to the WSS_ADMIN_WPG and WSS_WPG local groups when the service application
that the My Sites web application utilizes is provisioned. The My Sites web application has the “allow
self-service site creation” enabled as one of its requirements, without which My Sites would not be able to
provision for each user. The account is assigned to the WSS_CONTENT_APPLICATION_POOLS role that is
associated with the farm configuration database and with the Central Administration content database. It
gets SP_DATA_ACCESS to all of the content databases.
The WSS_CONTENT_APPLICATION_POOLS database role is associated with the farms configuration
database and the Central Administration site content database. The role makes it possible for its members to
query and update the site map and have read-only access to parts of the configuration database.
TechNet says, “The secure WSS_SHELL_ACCESS database role on the configuration
database replaces the need to add an administration account as a db_owner on the configuration database.”
(https://technet.microsoft.com/en-us/library/cc678863.aspx).
When you use the add-spshelladmin PowerShell cmdlet to add a user name, you’re only adding that
user to the configuration database’s WSS_SHELL_ACCESS role. J. T. has a handy one-liner that adds an
admin user to all the content databases by using the following:

Get-SPContentDatabase | Add-SPShellAdmin –UserName Domain\UserName -verbose

After running this command, the user that you specified in the user parameter value is added to the
WSS_SHELL_ACCESS role on all content databases. By adding a user to the role, you are giving them execute
access to all the stored procedures for the database, as well as the ability to read and write on all the database tables.
Because the SP_DATA_ACCESS role replaces the db_owner role in SharePoint 2013 to some degree,
it is the role that should be used to grant object model level access to databases during upgrades and new
deployments. It provides the following permissions:
• Grants EXECUTE or SELECT on all SharePoint stored procedures and functions
• Grants SELECT on all SharePoint tables
• Grants EXECUTE on user-defined types where the schema is dbo
• Grants INSERT on the AllUserDataJunctions table
• Grants UPDATE on the Sites view
• Grants UPDATE on the UserData view
• Grants UPDATE on the AllUserData table
• Grants INSERT and DELETE on the NameValuePair tables
• Grants CREATE table permission

17
Chapter 1 ■ Least-Privileged SharePoint Builds

Central Administration’s application pool runs under the same account that runs the timer service:
the farm account. This is why the farm account should not be a local administrator, as that would give this
site more privilege than it needs to operate. The farm account is also used to run the Security Token Service
application pool that is responsible for web service calls related to authentication. The farm account runs
the Topology Services application pool, as well, which is the pool responsible for overall control of what runs
where and on which servers via IIS. We’ll dive a little deeper into this in Chapters 2 and 3.

PowerShell to Reset Local Permissions and Files

So what do you do if you think something has been changed in your farm in one of the file systems, folders,
or registry settings? PowerShell to the rescue—and/or the post-setup configuration wizard.
If you think that something has been changed with respect to an ACL, you can use
Initialize-SPResourceSecurity on each of your farm’s servers to enforce security on the local server in all
the files, folders, and registry keys. You could also run psconfig.exe –cmd secureresources, which is the
command-line equivalent of the PowerShell cmdlet.
Unfortunately, this does not handle database permissions for the members of the WSS_WPG group.
The Install-SPFeature cmdlet is used during a new install and after joining a farm. It also scans the
system for any missing features and then reinstalls them in the farm. This works great for finding any missing
features that may have been erroneously removed by undocumented development.

Inspecting for Least Privilege

In this section, you’ll perform some exercises on the identities that are running SharePoint. You’ll look at
local group memberships, the identities that are running IIS application pools and where they are changed
in SharePoint. You’ll create a farm admin account, inspect IIS and the file system, and restore the file
system–level security.
In this first exercise, let’s look at the accounts that are running Windows SharePoint Services.

WINDOWS SHAREPOINT SERVICES

This exercise looks at Windows SharePoint Services and SharePoint Central Administration to determine
which accounts are utilized to run Windows SharePoint Services at the operation system level. Next, you
learn how to modify the account that each service is using so that SharePoint is aware of the changes.

Verify Identity Using services.msc

1. Open the Services management console by typing services.msc on the run bar
and clicking OK.
2. In the name column, scroll down the alphabetically sorted list to the services
starting with SharePoint*. Take a look at the identity that is used to run the
SharePoint Timer Service. The following screenshot shows the SPTimerV4 service is
running under the farm account as expected.

18
 Chapter 1 ■ Least-Privileged SharePoint Builds

This screenshot in was taken during a farm install when search had not been provisioned. After search
was provisioned, the account used to run the SharePoint Search Host Controller and SharePoint Server
Search 15 changed to 2013SearchSvc, as shown in the following screenshot.

■■Note fter the least-privileged farm is fully created, the SharePoint user code host runs under a
least-privileged account; the same is true for other services, such as the search services.

19
Chapter 1 ■ Least-Privileged SharePoint Builds

Verify Identity using PowerShell

1. Open a command line administratively by right-clicking the Windows logo and

clicking Command Prompt (Admin).
2. Once the command line opens, type PowerShell, and click Enter.
3. After the prompt returns, type the following code (also shown in the following
screenshot) and then press Enter.

gwmi win32_Service -filter "name='sptimerv4'" | ft name, startname

You can see that the SharePoint timer service is set to start using the account named 2013Farm.

Verifying the Farm Account Identity in Central Admin

1. Open Central Administration and click Security. Then, under General Security, click
Configure Service Accounts.
2. Click the drop-down menu and select the farm account. You should see the same
account that you saw in the Windows OS–level Services console (services.msc).
If you ever run into a situation where this account does not match what is in Windows, your best bet
is to rebuild the farm, if at all possible. If a rebuild it not feasible, then this is where you make changes
to any of the accounts in use by Windows SharePoint Services, and you follow any changes with an
IIS reset in every server in your farm. You should avoid making the changes directly in the Windows
operating system console, or in the IIS Management console; since SharePoint is not aware of this, it
will most likely cause issues.

In the next exercise, you’ll look at how to invoke the local group management console from the
command line and check group membership, as well as a quick way to verify group membership using the
net command.

20
 Chapter 1 ■ Least-Privileged SharePoint Builds

LOCAL GROUP MEMBERSHIP

Open the Local Group Management Console

In this exercise, you open the local users and group management console administratively to look at
group membership.
1. Open an administrative command line. Type Lusrmgr.msc and press Enter. The
local users and groups management console opens.
2. Click groups and then open the administrators group. Make a mental note of the
members that you see in this group, thinking about what I discussed in earlier. Note
that the farm account is (hopefully) not a member of the administrators group.
3. Open the WSS_WPG group at the very bottom of the list of groups. Note how the
various service accounts that run service and content application pools in IIS are
all members of this group, along with NT Authority\LOCAL SERVICE, NT Authority\
NETWORK SERVICE, and NT AUTHORITY\SYSTEM, as shown in the following
screenshot.

21
Chapter 1 ■ Least-Privileged SharePoint Builds

4. Open the WSS_ADMIN_WPG group. You should expect to see the installer account,
the farm account, and the BUILTIN\Administrators as members of this group, as
shown in the following screenshot.

5. Open the WSS_Restricted_WPG _V4 group. Note how the farm account is the only
identity allowed to be a member of this group.

■■Note The WSS_Restricted_WPG_V4 group should never allow any identities other than the farm account,
as this would surely over-privilege the farm.

22
 Chapter 1 ■ Least-Privileged SharePoint Builds

6. Open the IIS_IUSRS group, as shown in the following screenshot, and note that the
identities used in IIS are members of this group. Read the description of this group.

Check Group Membership Using the Command Line

1. At the command line, type net localgroup administrators and press Enter.
2. At the command line, type net localgroup WSS_ADMIN_WPG and press Enter.
3. At the command line, type net localgroup WSS_WPG and press Enter.
4. At the command line, type net localgroup WSS_Restricted_WPG_V4 and press Enter.
5. At the command line, type net localgroup IIS_IUSRS and press Enter.
The command-line method of the check local group membership is much faster, as long as you know
the group names.

Now let’s take a look at the user accounts that the Internet Information Services (IIS) Manager is using.
We already know that we should see different accounts in use by various application pools. Let’s take a look!

23
Chapter 1 ■ Least-Privileged SharePoint Builds

IIS IDENTITIES AND HOW THEY MAP TO SHAREPOINT

This exercise compares the service accounts that are in use by SharePoint application pools. It also
looks at the Service Accounts Credential Management page in Central Administration.
1. Open the IIS Manager. A quick shortcut to this program is always a good idea in
any SharePoint farm. You can open it by opening a run bar, typing inetmgr, and
pressing Enter or clicking OK.
2. Once the IIS Manager opens, expand the server node and click Application Pools.
Once the application pools are visible, adjust the column widths so that the values
are clearly visible, as shown in the following screenshot.

3. Navigate back to the Service Account Management page in Central Administration

by opening Central Administration and clicking Security. Then, under General
Security, click Configure Service Accounts.
4. Hopefully, you’ll find that the identity shown in Farm Account on the Service
Accounts page in Central Administration matches what is shown in IIS. In the
following screenshot, you can see that the Central Administration application pool is
running under the farm account.

■■Note Earlier in this chapter, we identified the farm account via PowerShell by running the following cmdlet,
which should agree with what you discovered in this exercise:

(Get-SPFarm).DefaultServiceAccount.

24
 Chapter 1 ■ Least-Privileged SharePoint Builds

m
N
ae

5. Click the drop-down menu on the Service Accounts page and select Service
Application Pool - SharePoint Hosted Services. Note how this account matches up
with the application pool named df8a3a42-fa06-48ee-b26a-5caf4ed4931b. The
fact that this application pool in IIS uses the same identity is all well and good, but
other than exploring the application pool to view the applications, how can we be
certain that this is the SharePoint application pool?
6. Open an administrative SharePoint Management Shell and type the following:

Get-SPServiceApplicationPool
Get-SPServiceApplicationPool | ft Name, ProcessAccountName, ID, -auto
Get-SPServiceApplicationPool | ft Name, ID, -auto

PowerShell returns the name of the SharePoint Service application pools along with the associated
GUID, as shown in the following screenshot.

25
Chapter 1 ■ Least-Privileged SharePoint Builds

7. Another method to identify which IIS application pool is the pool used by a
SharePoint service application is to have the Service Accounts page open and
the service application pool selected (similar to what’s shown in the following
screenshot), and then open IIS Manager.

8. Right-click the service application pool that serves 14 applications, and then click
View Applications, as shown in the following screenshot.

26
 Chapter 1 ■ Least-Privileged SharePoint Builds

9. Once the window changes, adjust the physical path so that you can see the
mappings, as shown in the following screenshot.
The following farm was installed in such a way that some of the files were not stored on the operating
system drive, but instead were stored on the D:\ drive.

Now that we’ve definitely identified which application pool in IIS is serving up the bulk of the service
applications in this farm, let’s make sense of those virtual paths in the first column.
In order to do this, we need to open the sites node and then drive to the SharePoint Web Services, as
shown in the following screenshot. Don’t worry if your farm does not have physical paths to a different
drive, because the different drive does not affect least privilege. The only reason that these different

27
Chapter 1 ■ Least-Privileged SharePoint Builds

paths would exist is if the option to store binaries on a different drive was chosen during the
SharePoint install.
If you’re having trouble joining a server to an existing farm, check the physical paths in IIS and then
adjust your install of SharePoint accordingly if you see paths other than C:\.
OK, let’s discuss the virtual path a little.

10. Expand the SharePoint Web Services node under Sites and choose one of the web
services to explore. In the following screenshot, I choose the web service with
the name that started with 1cca9199ade. After clicking Explore, I found that it is
mapped to a location on the D drive of the server.

28
 Chapter 1 ■ Least-Privileged SharePoint Builds

After having looked at all this, I can’t help but recall that proverbial question: If a tree falls in the forest
and no is there to hear it, does it make a sound? Quite obviously, the tree makes a sound when it falls; no one
hears the sound, but the sound is there nonetheless.
When it comes to SharePoint, you could ask this question: If a SharePoint farm is least privileged for
safety reasons, but 35 people know the farm account ID and password, is it really safe? I’ve seen this before,
and I’d argue that the farm is least privileged but in need of a governance document. Anyways, I wanted to
bring that up because you might encounter a situation where you need to give a user farm admin privileges
without giving away the farm account. The next exercise discusses how this is accomplished.

MAKING A FARM ADMIN

There might come a day when your company hires consultants to come into your environment and
perform a review of SharePoint. The consultant will definitely ask for access to your farm with farm
admin privileges. Personally, I’d want to hover over the consultant and watch them work; but since that
might be viewed as highly offensive, we instead need to create a farm admin account.

■■Note The farm admin account that we are creating is similar to the setup user, not “the farm account.” This
account should be able to perform everything the setup user account is capable of performing.

29
Chapter 1 ■ Least-Privileged SharePoint Builds

1. Log in to every server in the farm and add the user’s account to the Local
Administrators group.
2. Add the user’s account to the Farm Admin SharePoint Group.
3. Add the user’s account to SQL Server with sysad and db_creator fixed server roles.
4. Add the user’s account to the Shell_Admin_Access role of every content database.
5. Add the user’s account to the Shell_Admin_Access role of the configuration
database and the Central Administration content database.
6. Verify that the account was added to the WSS_ADMIN_WPG and WSS_WPG groups.

INSPECTING IIS AND THE FILE SYSTEM AND RESTORING ORDER

In this exercise, you look at the ACLs on a couple of the hive folders. On a couple of registry keys,
you look at virtual mappings in IIS that map to various locations in the hive. Then, you learn about
PowerShell and command-line commands to reset these permissions.

■■Note At no time do we change any of the permissions.

File System and IIS Mappings

1. Log on to the server in your farm that runs Central Administration. Open the IIS
Manager (inetmgr.msc).
2. Open Windows Explorer and navigate to the root of the SharePoint hive. This
example uses a SharePoint 2013 farm, so navigate to C:\Program Files\Common
Files\microsoft shared\Web Server Extensions\15.
3. Right-click the ADMISAPI folder and then click Properties. Click the Security tab and
then the Advanced button. Note in the following screenshot that the folder is owned
by SYSTEM and that there are some permissions that are explicitly granted to the
WSS_ADMIN_WPG and WSS_WPG groups. There are also inherited permissions.

30
 Chapter 1 ■ Least-Privileged SharePoint Builds

4. Close the permissions windows by cancelling or using the red X. Close Windows
Explorer so that you are looking at IIS Manager.

5. Open the Central Administration site, as shown in the following screenshot.

31
Chapter 1 ■ Least-Privileged SharePoint Builds

6. Click Explore (see the following screenshot) to navigate to the virtual directory
named _vti_adm. Note where it maps.

7. Note that the directory maps to C:\Program Files\Common Files\Microsoft Shared\

Web Server Extensions\15\admisapi.

32
 Chapter 1 ■ Least-Privileged SharePoint Builds

8. Since Windows Explorer is open to the hive, let’s look at one of the folders that
inherits its permissions from the SharePoint Hive’s root folder. This example uses
a SharePoint 2013 farm, so navigate back to the 15 hive. Remove the \admisapi
portion from C:\Program Files\Common Files\Microsoft Shared\Web Server
Extensions\15\admisapi in Windows Explorer so that you’re at the root, as shown in
the following screenshot.

9. Right-click the folder named BIN, and then click Properties. Click the Security
tab and then the Advanced button. Note in the following screenshot that all the
permissions are inherited. There are zero explicitly given permissions and the
WSS_WPG group does not have permission to this folder.

■■Tip Knowing how SharePoint permissions are supposed to be set, where they inherit and do not inherit, and
what this affects, helps troubleshoot issues. It might not solve them, but it helps you rule out possible culprits.

33
Chapter 1 ■ Least-Privileged SharePoint Builds

10. Take a few minutes to look at the virtual folders inside your SharePoint sites and
the non-virtual folders.
The folders that have shortcut icons like the _vti_adm (shown in the previous screenshot) do not
map to a location within the IIS web root. The folders that do not have a shortcut icon contain
SharePoint-related files within the default IIS root for SharePoint, which is usually located at
%SystemDrive%\inetpub\wwwroot\wss\VirtualDirectories.
Each web application gets a unique folder under this location and then each site has mappings to these
various locations. Table 1-2 provides a high-level analysis of the IIS to file system mappings.

34
 Chapter 1 ■ Least-Privileged SharePoint Builds

Table 1-2. IIS Mapped Folders

Folder Name Virtual Central Mapped Folder Path

Admin
Only
_admin Yes Yes %CommonProgramFiles%\Microsoft Shared\Web Server
Extensions\15\template\admin
_app_bin No No C:\inetpub\wwwroot\wss\VirtualDirectories\<SITEF
OLDER>\_app_bin
_controltemplates Yes No %CommonProgramFiles%\Microsoft Shared\Web Server
Extensions\14\template\controltemplates
_layouts Yes No %CommonProgramFiles%\Microsoft Shared\Web Server
Extensions\14\template\layouts
_login Yes --** %CommonProgramFiles%\Microsoft Shared\Web Server
Extensions\15\template\identitymodel\login
_vti_adm Yes Yes %CommonProgramFiles%\Microsoft Shared\Web Server
Extensions\15\admisapi
_vti_bin Yes No %CommonProgramFiles%\Microsoft Shared\Web Server
Extensions\15\isapi
_vti_pvt No No C:\inetpub\wwwroot\wss\VirtualDirectories\<SITEF
OLDER>\_vti_pvt
_windows Yes --** %CommonProgramFiles%\Microsoft Shared\Web Server
Extensions\15\template\identitymodel\windows
_wpresources Yes No %CommonProgramFiles%\Microsoft Shared\Web Server
Extensions\wpresources
App_Browsers No No C:\inetpub\wwwroot\wss\VirtualDirectories\
<SITEFOLDER>\App_Browsers
App_GlobalResources No No C:\inetpub\wwwroot\wss\VirtualDirectories\
<SITEFOLDER>\App_GlobalResources
Aspnet_client No No C:\inetpub\wwwroot\wss\
VirtualDirectories\<SITEFOLDER>\aspnet_client
bin No No C:\inetpub\wwwroot\wss\
VirtualDirectories\<SITEFOLDER>\bin
wpresources No No C:\inetpub\wwwroot\wss\
VirtualDirectories\<SITEFOLDER>\wpresources

** Does not exist in Central Administration site mappings

35
Chapter 1 ■ Least-Privileged SharePoint Builds

Registry Locations

In this part of the exercise, we’ll open the registry editor and take a look at the permissions on the root
SharePoint key.
1. Open the registry editor and navigate to HKEY_LOCAL_MACHINE\SOFTWARE\
Microsoft\Office Server\15.0, as shown in the following screenshot.

2. Right-click the 15.0 key and then click Permissions ä Advanced. Note that both
WSS_ADMIN_WPG and WSS_WPG have explicitly assigned permissions.

Resetting Permissions

If you ever need to reset the file, folder, or registry permissions back to their original permissions, you
can perform the following tasks.
3. To reset the permissions using PowerShell, open the SharePoint management
console administratively. Type Initialize-SPResourceSecurity (as shown in the
following screenshot) and press Enter.

36
Discovering Diverse Content Through
Random Scribd Documents
 “The Dartaway is gone!” exclaimed Ned. “So is the sloop! They’ve
stolen our boat! They must suspect something!”
At first Jerry and Bob could not believe it. Then Ned went over
again all that he had heard, telling how he had slipped away to
inform them of how matters stood.
“They must have known it was our boat,” said Jerry. “Was Bill
Berry one of the men?”
“I couldn’t see them plainly,” Ned replied. “Their voices didn’t
either one of them sound like Bill’s though. What’s to be done?”
“We’d better give notice of our boat being stolen,” said Jerry, “and
think of some scheme to get home. After that we’ll have to turn
detectives and get the Dartaway.”
The minstrel show was over when the boys went back to the
pavilion, and the crowd was coming away. The dock was thronged
with persons seeking their boats to make the run for home.
“Hello boys!” called a voice, and they saw Captain Jenkinson, of
the Three Bells coming toward them. “You look as if something had
happened.”
“Something has,” spoke Jerry. “Our boat’s been stolen.”
“Stolen! You don’t mean it. How?”
Ned related how it had happened, saying nothing however of the
conversation he had overheard.
“We’ll tell the manager of the place, and he’ll send out a general
alarm,” said the skipper of the Three Bells. “It isn’t the first time
boats have been stolen from the docks around here. I believe there’s
an organized gang. If we had a decent police force here it wouldn’t
happen so often.”
The manager of the pavilion and amusement resort, who was also
a sort of deputy sheriff, promised the boys to do what he could to
recover their craft. He said he would have notices of the theft posted
at his dock and all the other wharves along the coast.
 “How you boys going to get home?” asked Captain Jenkinson.
“Walk I guess,” replied Jerry with a laugh. “It’s only about five
miles by the shore road.”
“Well I guess you’ll not walk while there’s gasolene in the Three
Bells,” was the hearty answer. “Get aboard. We’ll cruise around a bit,
and maybe we’ll get sight of those rascals on the sloop. They
probably sailed out to sea, towing your boat, and they’ll be likely to
hide it in some cove until the affair blows over.”
Captain Jenkinson had a party of friends aboard his boat, but
there was plenty of room for the boys. There was much sympathy
expressed for them, and every one volunteered to be on the lookout
for the Dartaway.
In the powerful boat quite an area of the bay in the vicinity of the
dock was covered, in a search for a sight of the sloop. It was a fairly
light night, and a sailing vessel could be made out some distance
away.
There were several false alarms, and once the Three Bells gave
chase to a sloop that seemed trying to get away. But when they got
up to her they found it was only a fishing boat getting a start for
early morning work, and there was no sign of the boys’ craft.
“Too bad!” remarked Captain Jenkinson, as he landed the boys at
their dock. “But it’s sure to turn up sooner or later. They’ll not dare
to sell her, and can only keep her hidden. There are not many places
where a boat the size of yours can be tucked away.”
It was kindly and well meant consolation, but the boys never felt
in lower spirits than they did that night. Mrs. Hopkins tried to cheer
them up, but it was hard work. She even suggested they hire
another boat and make a search of the nearby harbors, for the
missing one.
“I guess that’s a good plan,” said Jerry to his chums. “We’ll do it
to-morrow.”
 “What ought we to do about warning Mr. Hardack?” asked Ned, as
they got ready to retire.
“Oh, I almost forgot about that,” came from Jerry. “I guess there
will be time in the morning. Besides, we want to think it over a little
more.”
 CHAPTER XXIV
THE SECRET OF THE COVE

The boys made inquiries the next morning and found it would be
impossible to hire another motor boat. The season was at its height
and all the craft were engaged. Several of their newly-made friends
offered them the use of their boats for a part of each day, but the
boys did not want to take them on this condition. Besides, they
wanted to be away by themselves, as they had much to talk about.
“Hadn’t we better warn Mr. Hardack?” asked Ned, when they had
exhausted the possibilities of hiring a boat.
“I’ve been thinking of it,” Jerry said, “and I don’t see what
particular good it would do. We only know half the story. Besides,
those fellows might only have been talking to hear themselves talk.
It’s a mighty risky thing to wreck a vessel and tamper with a
lighthouse. They wouldn’t dare do it.”
“But it wouldn’t do any harm to tell the keeper what we know,”
Ned insisted.
“Only to get laughed at for our pains, in case nothing happened.
Besides, how are we going to get to the lighthouse without a boat?”
“We can walk around by the shore, it’s not more than ten miles,”
went on Ned.
“Love will find a way,” sang Bob, and he had to jump back to
escape a playful blow Ned aimed at him.
“I suppose we could,” remarked Jerry. “Maybe it wouldn’t do any
harm to go around there and see how the land lays. We can sort of
give him a hint, and warn him to be on the lookout for Bill Berry.”
 “That will be a good plan,” agreed Ned. “When can we start? The
sooner the better.”
“Not until after dinner, I vote,” from Bob. “I can’t walk on an
empty stomach.”
“I don’t believe you ever tried,” Jerry said.
The boys inquired the best way to get to the lighthouse by going
along the shore, and learned it was about eight miles. They started
after lunch, and hard enough walking they found it, as most of the
way lay through deep sand.
“This is awful,” groaned Bob as he toiled along.
“Dry up, Chunky,” from Ned. “It will do you good. Besides, think of
what a service you may be doing.”
“I’m thinking about my chances for supper.”
The boys had gone about half the distance and were going along
a series of high sand dunes, covered with a growth of tall, rank
grass, when they were suddenly halted by a voice calling to them:
“Where you fellows going?”
They looked up, to see a roughly dressed man barring the path.
“I don’t know’s it’s any of your business,” Jerry replied hotly, for he
resented the man’s tone. “This is a free country.”
“That’s all right, my young friend,” the man said with a sneer, “but
this is private property, and we don’t allow any trespassing on it. So
you’d better be off if you know what’s good for you.”
“Can’t we go along the beach?” asked Ned.
“No, sir, you can’t go anywhere along here. We own for two miles
back, and if you try to pass here you’ll get into trouble. So be off!”
The boys hesitated. There was no way of proving that what the
man said was not so, and they did not want to get into trouble. To
get past meant walking two miles back in shore. As they stood
there, wondering what to do, another man came from somewhere in
the tall grass and said to the first:
“The boat’s in the cove. Floated in at high tide.”
“That’ll do!” was the quick retort, as though the first man feared
his companion would say too much. “I’m warning these chaps off
our land.”
“Yes, and they’d better go if they know what’s good for them,”
said the other.
“Oh, we’ll go,” came from Jerry. “We don’t care about walking on
your property. I guess we can manage without doing so.”
The three lads turned and began to walk inland, across the waste
of sand, which was hot with the afternoon sun. The coast at this
point was rather high, there being a series of bluffs, which sloped
abruptly down to the beach.
“What shall we do?” asked Jerry when they were out of earshot of
the men.
“Nothing to do but to go around their place,” said Ned. “It will
mean about four miles more.”
Bob groaned.
“It’ll get some of that fat off you, Chunky,” Jerry remarked with a
laugh.
“It’s all right enough for you to joke,” growled the fleshy youth.
“But I can’t help it.”
“I wonder what he meant by saying the boat was in the cove?”
came from Ned.
“I was thinking about that myself,” Jerry put in. “I didn’t know
there was a cove along here.”
“Let’s take a look,” suggested Ned.
“How can we, without going back and meeting the men?”
 “Easy enough,” came from Jerry. “We’ll walk along for a mile or so,
then cut down along parallel to the coast and walk back toward the
beach. We ought to come out right back of the cove, if there is one,
or very near to it.”
This was voted a good plan to follow, and, with no further
objections from Bob, the boys trudged along. It was hot and hard
work, but they were very anxious to find out the secret of the cove,
as they believed the men had some object in not allowing them to
pass.
The locality was a deserted one. It was half way between two
summer resorts, and there was not a house in sight. It was about as
lonesome a place as one could find in the midst of civilization.
Nothing was to be seen but sand and rank grass.
“Do you s’pose these men had anything to do with stealing our
boat and with the lighthouse plot?” asked Bob, as he and his chums
walked along.
“I’m not good at riddles,” returned Jerry. “First we’ll see what the
cove holds.”
They kept on for an hour and began to work their way in toward
the shore again. They kept a careful watch for the men but saw no
one. They came to a place where the weeds and grass were quite
high. It was tangled together by the wind and they had to struggle
to get through it. Jerry, who was in the lead, emerged on a clear,
sandy place. He gave one look down and uttered a low cry.
“What is it?” called Ned.
“The secret of the cove!” exclaimed Jerry.
He pointed to a small body of water below them as they stood on
a high sand bank. As the boys looked they saw a sailing vessel and
another craft floating near a small dock.
“There’s the sloop!” cried Bob.
 “And there’s our boat!” exclaimed Ned. “We have found her
again!”
“Easy!” whispered Jerry as he sank down, pulling his companions
to the earth. “There are the men!”
As he spoke three roughly dressed men came from a small shack
near the dock, and walked to where the sloop was moored. They
were carrying boxes and bales aboard.
“Looks like the stuff we picked up from the broken-backed
steamer,” whispered Ned.
“I guess they’re wreckers, who gather stuff that floats ashore,”
came from Jerry.
“And I’ll bet they’re the men I heard plotting about the
lighthouse,” said Ned. “We are on their track!”
“But how did they get their boat in here, and how did they float
the Dartaway in?” asked Jerry. “This looks like a little lake.”
“Isn’t that a sort of creek over there?” asked Bob, pointing to an
opening in the midst of the rushes that surrounded the cove.
“So it is. That’s what he meant about high tide. They can only get
in the cove when the tide is up, and makes an entrance by way of
the creek.”
“And, for the same reason, we can’t get our boat out until high
tide, and that will not be until late to-night,” said Bob. “We’ll have to
wait until then.”
“Lucky we have the chance,” came from Ned. “I hope the coast
will be clear.”
“We’ll get our boat, anyhow!” exclaimed Jerry. “I’d like to see
those men keep her.”
“We’ll wait until there’s water enough in the creek to float her out,
and then we’ll sneak down there, get in and start off before they
know what’s happened,” spoke Ned. “I hope she’s in running order.”
 CHAPTER XXV
AT THE LIGHTHOUSE

The boys remained concealed in the high grass for some time.
They watched the men moving about on the sloop and near the hut,
but the thieves seemed to pay little attention to the motor boat.
“I wonder if they’re getting ready for a trip?” said Bob. “That will
make it easier. If they leave we can go down there and get
something to eat.”
“Oh, Chunky!” exclaimed Ned. “You—” but he could think of
nothing appropriate to say, and so stopped short.
“They’re all boarding the sloop,” Jerry remarked, as he saw four
men come from the shack and go on the sailing vessel. “Going out
of the cove maybe.”
“Can’t, with the water as low as it is.”
“I only hope they go to sleep in the shack,” Ned remarked. “It will
be easier for us then.”
Through the long afternoon the boys waited. The little camp on
the shore of the hidden cove seemed deserted. None of the men
was to be seen. Toward evening there arose a thin column of smoke
from the galley of the sloop.
“They’re getting supper,” remarked Bob, with a sorrowful note in
his voice.
“Never mind, Chunky, you’ll get yours sooner or later,” said Jerry
as comfortingly as he could.
As it grew darker the boys noticed that the water in the cove was
agitated. The sloop, and the motor boat rocked at their anchorages.
 “The tide’s coming in,” said Jerry. “It will soon be time to act. I
hope we can get to the Dartaway without being seen.”
“We’ve got to,” spoke Ned. “If they see us it means we’ll have a
lot of trouble. We must crawl along until we get close to her. Then
we’ll get in. I’ll crank up, you can steer, and Bob can use a boat-
hook to fend us out from the shore.”
“Lucky she’s headed the right way to get out of the cove,” Jerry
remarked. “It will save time by not having to turn her.”
Thus it was arranged, and the boys, tired and hungry, remained
hidden in the grass until it was dark enough to put their plan in
operation.
They watched the sloop closely. After their supper aboard, the
men came on deck and stood conversing a while. The boys could
just make out their forms in the dusk. One seemed to be doing the
most talking, and he frequently motioned off toward the sea.
“Acts as if he was trying to get them to go somewhere,” spoke
Bob softly.
But in the end the men went ashore, and after looking to the
fastening of the motor boat and a small rowing craft tied near it,
they went into the shack. Presently lights shone from it, and Jerry
said:
“I guess we can sneak down now. Go easy, everybody.”
Cautiously the boys left their hiding places and began to descend
the slope that led from the bluff to the shore of the cove. Every now
and then they paused to listen. They could hear the men laughing
and talking in the hut.
Foot by foot they crept nearer. There was a path leading from the
top of the sand dune to the hut, but the boys did not take this,
fearing they would be seen. Instead they crawled on their hands and
knees through the grass. The process was a painful and slow one,
for their arms and legs came in contact with sand burrs, while
innumerable insects attacked them. But they suffered in silence.
 “Easy now, we’re almost there,” came from Jerry.
At that moment the door of the hut opened, and a man looked
out. The boys, with wildly beating hearts, crouched down. They
feared they had been discovered.
“See anything?” called some one from inside the hut.
“No,” was the answer, “I thought I heard some one at the boats,
but I guess it was the tide swinging the sloop. Looks like a storm.
Hope we’ll get one by to-morrow night. It’ll be just what we need,”
and the man re-entering the hut, closed the door.
For a few seconds after this the boys remained silent in the grass.
“Lucky escape, that,” murmured Bob. “Five seconds more and he’d
caught us.”
Cautiously they resumed the progress toward the boat. Nearer
and nearer they came until Jerry, who was in the lead, was able to
step over the side into it. Ned and Bob followed. The latter grasped
a boat-hook and stood ready to fend off when the start was made.
Ned and Jerry cut the bow and stern lines with which the Dartaway
was made fast to the little dock. They worked quickly and silently.
Jerry turned on the gasolene, and waited a few seconds to allow it
to fill the carburettor, as the boat had not been run in several hours.
Then he switched on the spark.
“Turn her over!” he whispered to Ned, who was in the engine
cockpit.
The big flywheel went around under the impulse of Ned’s sturdy
arm. There was a sort of cough from the engine. Then came a chug,
followed by a splutter, and the motor got into action.
“Fend her off! She’s headed into the bank, and I can’t steer her
out quick enough!” cried Jerry to Bob.
Chunky pushed with all his strength, on the pole, against the
bank. Slowly the nose of the boat came out from the shore. The
screw was churning the water into foam. Jerry spun the wheel
around, and headed the craft for the channel, the opening of which
he could just make out.
At that instant the door of the hut flew open, and in the light
which streamed forth several men could be seen running toward the
shore.
“Hi there! Stop! Bring that boat back!” they called.
“Guess not! She’s ours!” Ned called back.
“We’re off!” exclaimed Jerry in a low tone. “She’s running like a
charm. They’ll never catch us!”
There was the sound of feet on the dock. Then came a squeaking
of a pulley block, the creak of ropes and the rattle of the boom on
the mast.
“What’s the use going after them in the sloop?” they heard some
one cry. “There’s no wind. Take the rowboat!”
The thud of men jumping into the small craft tied near the sail
boat could be heard. There was the rattle of oars, and then the
splash of them in the water.
“They’ll never get out of the channel,” the boys heard one of the
men say. “We’ll catch ’em before they strike open water.”
“You will, eh?” thought Jerry. “We’ll see about that.”
The engine was speeded up. Jerry was beginning to distinguish
things better as his eyes became accustomed to the darkness on the
water. The channel was a narrow and winding one, but the incoming
tide had made it plenty deep enough.
The boys could hear the men frantically rowing after them, but it
was a hopeless race. The Dartaway was speeding ahead. It kept
Jerry busy steering to avoid running into the bank, but presently the
channel widened and he had no more difficulty. On sped the craft
until the little creek emerged into a small bay, which, in turn, opened
into the ocean.
 “We’re safe now!” cried Jerry. “Let’s light the lamps, and put for
home.”
The men in pursuit had been left far behind. While Jerry held the
boat on her course up the beach Ned and Bob kindled the red and
green side lights and the search lantern. In about two hours the
Dartaway was safe at her dock, and the boys were telling their story
to a number of their friends.
“We must notify the police and get after those thieves,” said
Captain Jenkinson. “They’re dangerous men to have around. It’s a
good thing you discovered that cove. They probably have been
hiding there a long time.”
But the primitive police system of the shore summer resort could
not be gotten in readiness for a raid that night, and when some
constables did go to the cove the next morning they found the sloop
gone and the hut seemingly deserted.
The boys found their boat had suffered little damage at the hands
of the thieves. Some tools had been removed as had a few of the
cooking utensils, but these were easily replaced.
“Now I guess we’d better make a trip to the lighthouse,” remarked
Ned, the next afternoon, when the Dartaway had been put in shape.
“We ought to warn Mr. Hardack.”
“And, incidentally, I suppose, Jessica,” added Bob.
“I think they’ll give the whole plan up, now they see we are after
them,” Jerry added. “I believe they’ve cleared out for good.”
“It’ll do no harm to go over and see Mr. Hardack,” Ned insisted. “If
we find out there’s no likelihood of the thing coming off, we needn’t
say anything.”
They got to the lighthouse about five o’clock. Mr. Hardack greeted
them warmly.
“Come right in,” he said. “Sorry Jessica is not home. She was just
wishing some visitors would come, and about an hour ago that
Nixon chap came along in his boat and took her for a ride.”
Ned seemed less happy than when the start had been made.
“But come in,” the lighthouse keeper went on. “I’ve got some
fresh milk and Jessica baked some cookies this morning.”
Bob was the only one who looked pleased.
As the boys were getting out of their boat they saw a man coming
down toward where the oil lamps were usually filled. At first they
thought it was Bill Berry, but a second look showed them it was not.
“Got a new helper?” asked Jerry, trying to speak calmly.
“Yes, my other one skipped off yesterday. This chap came along
and I hired him. Had to have some one in a hurry.”
 CHAPTER XXVI
HELD PRISONERS

The boys glanced at each other. This was something they had not
counted on. Evidently Bill’s companions had told him what had
happened, the night the motor boat was stolen, and he had fled, for
some reason. It looked as if the scheme of the plotters had fallen
through.
“Did Bill—er—did your other helper say where he was going?”
asked Ned.
“Not a word. He was filling the lamps—let’s see—it was yesterday
morning—come to think of it. A boat pulled up at my dock, and a
man got out and spoke to Bill. I had to go up in the tower, then.
When I came down Bill was gone and so was the man in the boat.”
“Rather strange,” commented Jerry.
“So it struck me,” Mr. Hardack went on. “But then you know these
chaps are sort of tramps. They’re here to-day and gone to-morrow.
Always roving around. Of course in the winter I have a regular
assistant the government provides, but in the summer time, just as
at the life saving stations, they take things a bit easier. However, this
other man came along, and he seems a lot nicer than Bill Cherry or
whatever his name was.”
The keeper led the way up the steps to the house, the boys
following.
“Guess it’s just as well not to say anything,” spoke Ned in a low
voice. “They’ve given up the plot. We’d only be laughed at if we
mentioned it.”
 His companions agreed with him, glad enough to feel there was
going to be no attempt to wreck a ship by means of false lights. The
keeper set out a big pitcher of cool milk and a plate of cookies,
which, as Bob said, were the best he ever ate, but then Bob was apt
to say that about anything in the culinary line.
“Yes,” Mr. Hardack was saying, “Jessica would have been glad to
see you. Poor girl, she has quite a trouble on her mind. I’ve been
hoping things would straighten out, but they don’t seem to. Her
father, he—”
“Ting-a-ling-ling-ling!” rang the telephone bell. The keeper sprang
to answer it. The boys listened idly to the one-sided conversation.
“Yes, this is Mr. Hardack.”
“What’s that? Kate sick?”
“Come over? Yes—er—that is—Yes, I can come. I forgot I had a
new helper. I’ll be right over. Anything serious?”
“Can’t tell, eh? Well I’ll come as fast as I can,” and he hung up the
receiver.
“Any trouble?” inquired Jerry.
“Looks like it,” the keeper said. “My sister is quite sick. Taken
suddenly. They want me.”
“Where does she live?”
“It’s about six miles back in the country. I guess I can make it and
get back here by nine or ten o’clock. I wish I knew whether it would
be safe to leave the new man in charge.”
“Don’t the regulations provide for it?” asked Ned.
“Oh, yes, it’s my day and night off, and I have a right to go. But I
sort of hate to leave the light with him. He knows all about it,
however, and he’s got a government civil service certificate. He
knows just what to do, for he’s been in lighthouses before. I wish I
knew what to do.”
 “Let us stay and help him,” suggested Ned.
“Will you?” asked Mr. Hardack eagerly.
“Sure,” chorused Jerry and Bob.
“Then I’ll do it. I want to see my sister. Her health is not very
good, and the doctor said she might die in one of her spells. I’d feel
safe to go if I knew you boys would stay here and help the new man
if necessary.”
“We’ll see to things,” exclaimed Jerry. “It will be jolly fun to be
partly in charge of the lighthouse.”
“Whatever happens, don’t forget two things,” cautioned Mr.
Hardack.
“What are they?”
“The light must be lit at sunset, and it must be kept burning all
night. It must revolve regularly, even if it has to be done by hand,
and there must be a white flash and two red ones, at proper
intervals. But, you needn’t worry about that. The machinery is in
perfect order. The man will light the lamp, and start it going. It only
has to be trimmed once in a while. I’ll be back before ten o’clock.
When Jessica comes, she’ll get supper for you.”
Ned said nothing, but he looked as if that would be the best part
of it all, while Chunky’s eyes lighted up at the mention of another
meal.
Mr. Hardack was soon ready to go. He had to walk the entire
distance, as there was no conveyance handy, but he said he did not
mind that.
“I’ll introduce you to the new man,” he said, calling his helper
from where he was still busy filling the lamps. “His name is John
Elkwood.”
The assistant did not seem a very good natured chap. He only
nodded to the boys, when Mr. Hardack introduced them, and, as he
went back to his work, Jerry heard him muttering to himself.
 “Well, I guess I’ll get under way,” said the keeper as he started off.
“I say,” called Elkwood after him.
“What is it?”
“I don’t need those boys here. I can get along without ’em. They’ll
be in the way.”
“I want ’em to stay,” was Mr. Hardack’s answer, at which the boys
heard the new man muttering again.
“Not very friendly,” commented Jerry. “Still we can get along I
guess.”
The boys spent an hour going over the lighthouse, with which
they were now rather familiar. In the meanwhile Elkwood was busy
filling lamps, there being a number used in the big tower. He
attended to the light in the big glass lantern and spent some time
oiling the machinery.
“I wonder what time Jessica is coming back?” said Bob, as they
sat down in the sitting room.
“Was that one thought for her and two for the supper?” inquired
Jerry.
“It’s about time she should be back, I think,” came from Ned.
“He’s only thinking of her, you see, Chunky,” Jerry went on.
“No, but it seems to be getting foggy,” added Ned, “and Noddy
isn’t any too good a hand at managing a boat. I wish she hadn’t
gone out with him.”
“Oh, she’ll be all right,” commented Bob. “Tell you what’s let’s do.
We’ll get supper and have it all ready when she comes. I guess we
can find the things to eat.”
“Trust Chunky for that even if he doesn’t have any dishes on the
table,” Jerry remarked. “Well, we’ll get the meal and invite Noddy to
it.”
 “Not a bit of it!” exclaimed Ned. “When he sees us here he’ll go
back where he came from, fast enough.”
The boys found a well-stocked pantry, and, because of their
camping experiences had little difficulty in getting a meal ready. By
this time it was nearly seven o’clock. Ned kept rather anxious watch
of the hours.
“Let’s go down to the dock and see if we can get sight of her,” he
suggested.
“Who?” asked Bob.
“Why Jessica. It’s time she was back.”
Though he did not say so, Jerry was also a little anxious. The
weather looked anything but promising, and he had small respect for
Noddy’s ability to manage a motor boat in a calm, let alone a storm.
Still there seemed to be no cause for alarm.
The craft might have been stalled, but he did not believe Noddy
would venture far from shore, and, in the event of a breakdown, he
could signal to other boats, as there were several about the harbor.
It was still quite light, and would not be dark for another hour. It
was no use worrying, Jerry thought, until there was something to
get excited over.
They all went down to the dock, however, and scanned the sea for
a sight of the boat containing the girl and Noddy. Though there were
several craft in sight the boys did not notice Noddy’s, which they had
come to know from seeing it several times. It was one with a blue
hull, distinguishable for some distance.
“I vote we eat,” said Bob, as he turned to go back to the house.
“It wouldn’t be polite,” suggested Ned. “We’re only visitors.”
As they walked up the stone steps leading to the house, the boys
were met by Elkwood. The man had a scowl on his face.
 “It’s time you chaps were moving,” he said in surly tones. “I don’t
want you hanging around here.”
“Why, Mr. Hardack asked us to stay,” put in Jerry.
“I don’t care whether he did or not. I’m in charge here. This is
government property and I’m the boss. I tell you to go, and don’t
lose any time over it, either.”
“I guess we’ll stay,” said Jerry coolly. “We told Mr. Hardack we
would, and we’re going to.”
“And I say you’re not. I order you off. It’s against the regulations
for you to be here after dark.”
“It isn’t dark yet,” spoke Ned.
“None of your lip!” exclaimed Elkwood. “Are you going to leave?”
“Not until Mr. Hardack comes back!”
“Then you can take the consequences!”
Elkwood put his fingers to his lips and blew a shrill whistle. At the
signal three men sprang out from behind some rocks that bordered
the stairway. They rushed at the boys, who were too surprised to
stir. One of the men was Bill Berry.
“We’ve got you this time!” their old enemy cried.
The next instant the boys were struggling with the men, who
endeavored to throw bags over their heads.
 CHAPTER XXVII
TRYING TO ESCAPE

The struggle was a sharp but short one. The boys were no match
for the husky men, and though the lads kicked and punched with all
their might, they could not save themselves. In a few minutes they
were securely bound, and with the bags tight over their heads, were
picked up by the men.
“Where you going to put ’em?” they heard Elkwood ask.
“The storehouse is a good place,” Bill Berry replied. “They can yell
there all night and never be heard. Take ’em to the storeroom!”
The boys felt themselves being carried up the steps. Then they
could tell, by the muffled footfalls, that they were being taken into
some dungeon-like place.
“Shall we leave the bags on?” one of the men asked.
“No, I don’t want to smother ’em,” Bill replied. “They can’t make
themselves heard in here, no matter how they yell. Besides, there’s
nobody around. We’ve got Hardack out of the way and he’ll not be
back until morning.”
“You forget the girl. She may be back any minute.”
“I guess not. Noddy has charge of her. He’ll detain her some way
or other. Those motor boats have a habit of breaking down, you
know.”
Then the bags were taken from the boys’ heads, but their bonds
were not removed, and they were laid down on the cold stone floor
of the storeroom. With sinking hearts they heard the men withdraw
and lock the door, leaving them prisoners in total darkness.
 For a few seconds none of the boys spoke. They were so surprised
and shocked at the suddenness of it all they did not know what to
say. At length Jerry’s voice broke the silence:
“Are either of you hurt?”
“Only scratched and bruised,” replied Ned.
“My wrists are cut by the rope, and my legs hurt,” said Bob. “I’m
hun—”
“Let up on that!” exclaimed Jerry with a violence he seldom used.
“This is no time to think of eating. Boys, it’s a mighty serious matter.
These men are going to wreck the ship!”
“Do you think so?” inquired Bob.
“Of course; what else is it? They have carried their plot into effect,
but they did it differently than I expected. Bill Berry’s going away
was only a blind, and it fooled us. This new man, of course, is in the
game. He came along as soon as Bill left, so no one else would be
hired for the place.”
“Do you think they got Mr. Hardack away by a false message?”
asked Ned.
“Of course they did. It was all in the game. Noddy is helping
them.”
“If I ever get hold of him I’ll make him wish he’d never had a
hand in it,” and Ned spoke so sincerely that his companions knew he
would keep his word. They thought of Jessica out alone with the
bully, who, possibly had purposely disabled the engine to keep her
from getting back to the lighthouse.
“Oh, if we could only do something,” exclaimed Ned.
“We’ve got to!” cried Jerry. “We can’t let the ship be wrecked by
them changing the light.”
“But how we going to stop ’em?” asked Ned.
 “We must try and get loose,” Jerry replied. “They tied us in such a
hurry maybe some of the knots will slip. That’s our only plan. There’s
no use calling for help. It’s just as Bill said, no one would hear us.
Try and work your hands free.”
They all tried but to little purpose. The ropes were firmly tied.
Strain as they did they could not loosen the fastenings, and at last
they had to stop, as the cords cut into their flesh.
“Well, they certainly got us into a trap!” exclaimed Jerry as, once
more, he tugged at his bonds.
Suddenly Bob uttered an exclamation.
“Are you hurt?” cried Ned.
“Something cut my wrist!”
“What is it?”
“A piece of glass, I think.”
“Glass! Good!” came from Jerry. “Can you get it in your hands?”
“I have it.”
“Roll over towards me, and bring it with you.”
Bob did so. He came close to where Jerry was still tugging away at
the ropes.
“How did you find it, Bob?”
“I was trying to get the knot loose and something sharp touched
my wrist. I felt around until I found the glass.”
“What’s your plan, Jerry?” sung out Ned.
“I’m going to get Bob to hold the glass and I’m going to saw
through the ropes on my hands. Then I’ll set you all free!”
“Can you do it?”
“I’m going to make a big try.”
 Then in the darkness they began their efforts to escape. Bob
stretched out on his face, holding the jagged piece of glass from a
broken bottle between his bound hands. By careful feeling Jerry
edged his way over to him, until he could bring his wrists close to
Bob’s. Then both boys turned on their side, back to back, and Jerry
began sawing at the cords that bound him.
It was hard work, and more than once they had to stop because
their arms ached. Several times Jerry’s hands slipped and the glass
cut him, but he did not mind. Back and forth he drew the rope over
the keen edge until he could feel the strands giving way.
“It’s almost loose,” he said.
In another minute he gave a triumphant cry.
“I’m free!”
“Now to loosen us!” called Ned.
Jerry reached into his pocket for his knife. Luckily the men had not
searched them, or taken anything away from the boys. With his
hands free Jerry soon had the ropes from his legs. Then he cut the
bonds of Ned and Bob. Their limbs were stiff, from being tied so
long, but vigorous rubbing soon restored the circulation.
“Now to escape!” exclaimed Jerry. “We must find a way out of this,
and stop the rascals from setting the false lights!”
They stumbled about in the darkness. The storehouse was filled
with boxes and barrels, over which they fell as they felt around,
seeking for some door or window. At last Ned cried out:
“Here’s a door!”
The other boys made their way toward the sound of his voice.
“It’s locked!” said Jerry, as he pushed against the portal.
“Can’t we batter it down with a box or a barrel!” Bob said.
They searched around in the gloom for something to use, but
could find nothing. Everything was too heavy.
 “Maybe we can cut around the lock with our knives,” suggested
Ned.
In the darkness and silence they toiled. They could hear nothing
from the men they knew must be in the lighthouse, working to
cause the destruction of the steamer. They felt as if they were
imprisoned in a vault.
“I wonder if we can get out and be in time?” said Ned. “It must be
quite late.”
“Don’t talk! Work!” came from Jerry.
They redoubled their efforts to cut around the lock. But the door
to the storeroom was thick and strong, and the lock was a heavy
one.
“It’s no use,” declared Bob after an hour’s hacking away at the
tough wood. “We’ll have to stay here until they let us out.”
“Don’t give up,” Ned spoke.
“Hark! What’s that?” asked Jerry.
The others listened.
“They’ve started the machinery!” cried Bob. “The lenses are
turning.”
“Yes, and they are the wrong ones! They will get the ship on the
rocks!” cried Jerry. “We must escape!”
Terror struck to the boys’ hearts. They had tried every means and
failed. The plotters had outwitted them. They could do nothing. They
beat upon the door with their fists as though by their feeble efforts
they could break it down.
Ned stumbled aimlessly in the darkness, seeking for something
with which to batter down the door. As he passed by a pile of boxes
and barrels he uttered a cry.
“Have you found anything?” asked Jerry.
“Something, yes! A window in the wall! An open window!”
 Bob and Jerry hurried to where they heard Ned’s voice. As they
did so he had climbed up on a box. He pressed his face close against
the wall. A cool wind fanned his cheek.
“There is an opening!” he exclaimed. “But it is too small for us to
get out of. It’s only a ventilating window. But wait! Someone is
coming!”
The boys almost held their breaths. Then Ned called in a loud
whisper:
“Jess! Jess! Here we are! Let us out! Some bad men are in charge
of the place and are going to change the lights! They are going to
wreck a steamer!”
 CHAPTER XXVIII
JESS TO THE RESCUE

“Who are you talking to?” asked Jerry.

“Jess, of course,” replied Ned, greatly excited. “She’s outside. Jess!
Jess!” he called again. “We are locked in the storeroom!”
The boys waited anxiously. Then, from without, came a whisper
that sounded loudly through the darkened room.
“What has happened? Where is my uncle? Who are you?”
“It’s me; Ned,” was the reply, whispered from the prison. “They
captured us! Have you a key? Can you let us out? How did you get
away from Noddy?”
“Oh, this is terrible!” cried Jess. “How did it happen?”
She was standing under the small slit in the masonry that served
to let air into the storeroom. The light from a lamp in the kitchen of
the place streamed out from a window full on her, so Ned could see
the girl plainly, though of course she could not see him.
“Why you are all wet!” cried Ned. “Did you fall in the water?”
“No, I jumped,” came the tense whisper. “What shall I do to let
you out?”
“Can you get the key to this place?” asked Ned. “If you can, sneak
into the house, and open the door, let us out and we’ll call help, and
try to prevent the men from changing the light.”
“Where is my uncle?”
“He was called away, by a false telephone message, we believe, to
see his sick sister! The men put up a game to get him away! Quick
Jess, or it will be too late!”
Ned saw the girl step back out of the path of illumination and gaze
upward. As she did so she uttered a half suppressed scream.
“They are changing the light!” she uttered in a shrill whisper. “And
there’s a storm about to break! What shall I do?”
As she spoke there came a low rumble of thunder off to the west
and a flash of lightning.
“Let us out if possible!” whispered Ned. “They are so busy with
the light they may not notice you. Get the keys and let us out!”
“I will! I will!” exclaimed Jess. “If I can only succeed!”
Ned saw her dart around the corner of the house. Then she was
out of his line of vision. They could only wait developments now.
“Do you think she can do it?” asked Jerry.
“She will if it’s possible,” replied Ned. “Only there is not much
time. My! But it’s going to storm fierce!”
A loud crash of thunder sounded, making the stout lighthouse
vibrate. The flashes of lightning showed through the ventilating
window, illuminating the small apartment with a weird glow. The
wind was howling about the place.
“There’ll be a heavy sea on,” said Jerry. “The ship will get upon
the rocks and go to pieces. Then these scoundrels will go out and
pick up the cargo.”
“There may be many lives lost,” exclaimed Bob. “The life saving
station is short-handed. They all are in the summer time.”
“If Jess can only get the keys!” Ned murmured.
It seemed like an hour before there sounded a tapping on the
storeroom door. Ned sprang to answer it.
“Are you still there, boys?” they heard Jess ask.
“Yes! Yes!” whispered Jerry. “Have you the keys?”
 “No, the men must have taken them.”
“Then get an axe and see if you can’t break the lock.”
“It is too strong. Besides they might hear the blows.”
“Where are the men?”
“In the lantern tower,” the girl replied. “Wait a minute, let me think
of a plan.”
Outside the storm was raging. Locked in the storeroom the boys
felt like beating at the door with their fists to break it down, so they
might get out, change the light, and save the steamer.
“I have it!” Jess whispered through the big keyhole. “I will burn
the lock out.”
“How?” asked Ned.
“With a hot poker. I’ll heat it in the kitchen stove. I’ll burn a lot of
little holes all around the lock, and then I can knock the piece of
door out! The men can’t hear that!”
“Good!” cried Ned. “Hurry Jess!”
They could hear the girl moving about the kitchen. The rattle of
iron on iron came to their ears. Presently there was the smell of
burning wood. It grew stronger. Then a dull red point pierced the
door, and came through into the storeroom.
“That’s the first hole!” whispered Jess. “I’ll burn them as fast as I
can.”
To the boys it seemed as if there was half an hour between each
reappearance of the glowing point of the poker, but it was only a few
minutes. There were seven holes burned, when they heard Jess
hurry away.
Then resounded the tramp of feet in the lower part of the
lighthouse. A few seconds later the boys heard voices.
“Is it working all right?” a man asked.
 “You bet,” was the reply. “Now you and Bill had better put off in
the sloop. She’ll strike pretty soon, and you may pick up passengers
with a lot of valuables.”
“It’s blowing pretty hard to go out in the sloop,” one of the crowd
objected.
“Oh, don’t get chicken-hearted,” was the sneering response. “You
and Bill have got to go. Me and Jim will stay here and work the light.
We can tell when the rockets go up that she’s struck, and then we’ll
skip. We’ll meet at the cove.”
The voices died away, as though the men had left. The sound of
the storm increased. Anxiously the boys waited for Jess to come
back. It was several minutes before she did so. Then she whispered
through the keyhole:
“I had to run and hide when I heard the men coming from the
tower. Two of them have gone out, and the others have gone back
to the light. We must hurry!”
Once more came the smell of burning wood, and once more the
dull red point of the poker began to show. But it was slow work, for
the door was thick, and of hard material. Then too, the poker would
get cool carrying it from the stove to the portal.
But Jess worked like an Amazon. Back and forth she went with the
hot iron, burning herself several times when it slipped. But she gave
small heed to this. She wanted to save the ship and the honor of her
uncle, who might be blamed for losing control of the lighthouse.
Hole after hole was burned. Now Ned began trying to knock out
the piece of door containing the lock. He found a small stone and
hammered on the weakened wood. But it was still too strong for the
feeble instrument he had.
“Ten more holes and I think it will come out,” the girl whispered.
Out on the deep, struggling through the storm which had
suddenly broken, was a large steamer, laden with a rich cargo. There
were not many passengers, as it was from a South American port,
but these few, as well as the crew, had no warning of the danger
that threatened them.
In the bow stood the lookout, scanning the expanse of angry
water for a sight of lighthouses and headlands that would indicate
the channel up the dangerous coast. Suddenly off to his left there
shot out two brilliant red flashes.
“North light two points off the port bow!” he called to the pilot.
“Lookout?” called the pilot.
“Aye, aye, sir.”
“Are you sure that’s the North light?”
“Aye, aye, sir. The south light shows a white flash and two red
ones. These were only two red. There they are again, sir.”
“Yes, I see them,” as once more the false lights flashed across the
sea. “We must have passed the South light while the weather was
thicker. I’ll have to put her in a bit.”
Then the pilot, deceived by the light, steered the vessel over
toward the ledge of dangerous rocks, instead of keeping out, as he
would have done, had the two red flashes been preceded by a white
one.
But in the lighthouse three brave boys and as brave a girl, were
striving to aid the ill-fated steamer. Would they be in time?
Jess made hole after hole, though her arms ached, her eyes
smarted with the smoke, and her hands were burned in a number of
places.
Again and again Ned beat with his stone on the wood around the
lock. The circle of holes was complete at last.
“It’s giving away! It’s loosening!” cried the boy. He struck with all
his force. The stone flew from his hand, and fell through the opening
that suddenly appeared. The lock had been burned away, and the
heavy door swung inward. The boys were free.
 “Now to change the lights!” cried Jerry, as, followed by his chums
he dashed toward the winding stairs that led to where the big
lantern lenses revolved.
At that instant the door of the kitchen flew open and Mr. Hardack
entered, wild and disheveled, dripping water from the storm which
was now raging at its height.
 CHAPTER XXIX
THE RIGHT LIGHTS

“What has happened!” cried the keeper. “The light is flashing

wrong! There is a steamer outside the bar! It will be wrecked! Who
did it? Where is my assistant? There’s been foul work here! I was
waylaid on my way back when I found my sister was not ill. I just
managed to get away from the men. Speak, some of you! Quick!”
The keeper was panting from his exertions and from the
excitement. His face was drawn and pale, and his eyes were wild,
while his hair, matted by the rain, for he had lost his hat, straggled
about his forehead.
“The scoundrels are in possession of the tower!” cried Jerry. “We
must attack them and set the right light!”
“Come on!” cried the keeper, seizing the poker Jess had used to
burn the door. “Come on! I’ll give ’em battle!”
His eyes glared, in the fierceness of his righteous anger, at those
who would do so dastardly a deed.
“Come on!” cried Ned, seizing a heavy billet of wood.
“I’ll call the police on the telephone!” exclaimed Bob, springing for
the instrument. “We’ll need help!”
“I’ll not wait for the police!” fairly shouted the keeper. “I’ll tackle
’em single handed if need be!”
Bob rang up central, and, not waiting to be connected with the
distant police station, told the operator what the trouble was,
imploring that aid be sent promptly. Then he ran to join his
companions. Jess was crying in one corner of the room.
 Mr. Hardack led the way to the stairs which extended up inside the
tower to the lantern. He fairly ran up the stone steps, followed by
the boys. He was shouting challenges to the men as he ran.
“Let me get at you!” he yelled. “I’ll show you how an old man can
fight!”
Suddenly from above them a door slammed shut. There was the
clicking of a lock. Then, as they came to the heavy portal, which
gave access to the room where the lantern was, a voice cried:
“You’re too late this time, old man!”
Too late! The men had shut themselves up in the top of the tower,
and could control the working of the light to suit their evil purposes.
The keeper could not get in.
Mr. Hardack beat upon the door with the poker. Ned hammered it
with the block of wood.
“Let me in!” cried the aged man. “Let me in! Do you want to send
the ship to the bottom?”
“That’s just what we do!” was the mocking response.
“Get an axe and chop the door down!” cried Jerry.
“It would take too long,” replied the keeper, in a strangely calm
voice. “It is bound with iron, and is double thick. There is no help for
it. The steamer will be lost!”
Footsteps were heard coming up the stairs.
“Maybe help is at hand,” said the keeper hopefully.
Then Jess came into view. In her hand she held something which
she extended to Mr. Hardack.
“Here is your old horse pistol, uncle!” she exclaimed. “It is loaded
with a heavy charge. Fire it through the lock and shatter it! I heard
you pounding on the door and knew they had locked it!”
“Hurrah for you, Jess!” called Ned, and the girl blushed through
her tears.
Welcome to our website – the perfect destination for book lovers and
knowledge seekers. We believe that every book holds a new world,
offering opportunities for learning, discovery, and personal growth.
That’s why we are dedicated to bringing you a diverse collection of
books, ranging from classic literature and specialized publications to
self-development guides and children's books.

More than just a book-buying platform, we strive to be a bridge

connecting you with timeless cultural and intellectual values. With an
elegant, user-friendly interface and a smart search system, you can
quickly find the books that best suit your interests. Additionally,
our special promotions and home delivery services help you save time
and fully enjoy the joy of reading.

Join us on a journey of knowledge exploration, passion nurturing, and

personal growth every day!

ebookbell.com